A checkout request takes 4 seconds. Each of the nine services it touches reports a p99 of 200 milliseconds, and every dashboard is green. Logs tell you what each service did but not that they were the same request, and the arithmetic does not close because the time is being spent in places no single service can see. Tracing exists to close that arithmetic, and it only works if the context survives every hop.
01What a trace actually is
A trace is a tree of spans sharing one trace ID. Each span records an operation with a start time, a duration, a parent span ID and a set of attributes. The parent-child relationships are what turn a pile of timings into a shape you can read - you see that service D was called serially four times when it could have been called once in parallel, which no per-service metric would ever reveal.
The critical insight is that a trace shows you gaps. If the root span is 4 seconds and its children account for 1.2 seconds, the missing 2.8 seconds is real and it is happening somewhere you have not instrumented: queue wait, connection pool acquisition, TLS handshakes, DNS, or garbage collection. That gap is usually the answer, and it is invisible without the tree.
02Propagation is the whole thing, and it breaks silently
Context travels in headers - the W3C `traceparent` header carries the trace ID, the parent span ID and the sampling decision. HTTP frameworks with an OpenTelemetry instrumentation library handle this automatically, which is exactly why the failures are surprising: everything works until a hop that is not plain HTTP.
Propagation dies at boundaries the auto-instrumentation does not know about. Message queues need the trace context serialised into message headers and re-extracted by the consumer, otherwise the consumer starts a fresh root trace and the causal link is gone. Background jobs, scheduled tasks, thread pools where context lives in thread-local storage, native async runtimes, and any place you hand work to a different executor all lose it unless you carry it explicitly.
- Queue producers must serialise `traceparent` into message metadata; consumers must extract it and link the span.
- Thread pools and executors lose thread-local context - propagate explicitly at the submission point.
- Third-party SDKs frequently strip unknown headers; verify rather than assume.
- Alert on orphan root spans - a sudden rise means a propagation path broke in a deploy.
03Sampling: head-based is cheap and misses the point
Tracing every request at full detail is expensive in network, storage and CPU, so you sample. Head-based sampling decides at the root span - keep 1% of traces, discard the rest - and it is trivially cheap because the decision propagates in the header and every service honours it consistently. Its fatal weakness is that it decides before anything interesting has happened, so it keeps 1% of the boring requests and 1% of the failures.
Tail-based sampling buffers all spans for a trace at a collector, waits for the trace to complete, then applies rules: keep everything that errored, everything slower than a threshold, a small percentage of the rest. You get every pathological trace and a representative baseline. The cost is that the collector must hold all spans for the trace duration and must see all of them, which constrains how you scale collectors - spans for one trace must route to the same collector instance.
| Approach | Decision point | Keeps errors? | Cost |
|---|---|---|---|
| Head-based, fixed rate | At the root span | Only by luck | Very low - minimal network and collector load |
| Head-based, per-route rates | At the root span | Only by luck | Low - lets you oversample rare endpoints |
| Tail-based | After trace completes | Yes, by rule | High - collector buffers every span, needs trace-aware routing |
| Hybrid | Head-sample high volume, tail-sample the rest | Mostly | Moderate - the usual production answer |
04Span attributes decide whether a trace is searchable
A trace you cannot find is worth nothing. The complaint you get is 'checkout was slow for this customer around 14:30', and you need to go from that to the trace in under a minute. That requires attributes on the root span: tenant ID, user ID, route template, and the business identifier that customer support actually quotes - the order number, not the internal UUID.
Use the route template rather than the concrete path. Recording `/orders/8a3f-91c2/items` as the span name produces unbounded cardinality and makes aggregation useless; `/orders/{id}/items` lets you ask which route regressed. Never put secrets, tokens, full request bodies or personal data in attributes - traces are typically retained in a third-party system with looser access controls than your database, and this is a routine source of accidental data exposure.
05Reading a trace: the four shapes worth recognising
Most latency problems present as one of a small number of visual patterns, and learning to recognise them turns trace reading from archaeology into diagnosis. The staircase - many short spans in sequence to the same service - is an N+1 call pattern, fixed by batching. The wide bar with narrow children is time spent in the service itself, not downstream, so profile that process.
The long span with no children at all is the one people misread most often. It usually is not compute; it is waiting - connection pool acquisition, a lock, or a queue. Instrument pool checkout explicitly as its own span, because 'waiting 800ms for a database connection' and 'the query took 800ms' look identical in an uninstrumented trace and have completely different fixes.
- Staircase of identical spans - N+1 calls; batch them.
- Wide parent, thin children - the time is local; go and profile.
- Long childless span - almost always waiting; instrument the wait.
- Child span extending past its parent - clock skew across hosts, not a real anomaly.
06Correlating traces with logs and metrics
The three signals are only useful together. Put the trace ID into every log line as a structured field, and the exception stack trace you find in logs becomes one click from the trace that produced it. Attach exemplars to your latency histograms so that clicking the p99 bucket takes you to an actual slow trace rather than leaving you to search by timestamp.
The workflow that pays for the whole investment is: alert fires on a latency SLO, exemplar leads to a representative slow trace, trace shows which hop owns the time, trace ID pulls the exact logs from that hop. Without correlation IDs joining them, you have three tools and a manual timestamp search, which is what everyone had before tracing and is why the investment fails to land in some teams.
07Rolling it out without a two-quarter project
Instrument the edge first - the API gateway or front door - so every request gets a trace ID immediately. Then instrument the two or three services on your most painful path rather than all nine. A partial trace covering the slow path is useful on the first day; a complete trace covering everything arrives too late to help with the incident you are having now.
Use auto-instrumentation for frameworks, clients and drivers, and add manual spans only where the automatic ones leave gaps you care about: business logic boundaries, cache lookups, connection acquisition, and any external vendor call. Resist the urge to span every function - a trace with four hundred spans is as unreadable as one with none, and it costs considerably more.
Topics
Priya Iyer
Staff Engineer · SyncTrix
Writes about the engineering decisions behind production systems - architecture, delivery and the trade-offs that only show up at scale.
Building something like this?
SyncTrix engineers AI, SaaS, platform and cloud systems for enterprises and high-growth teams. Tell us what you're shipping and we'll scope it with you.