Parents, Children and Links
Nesting is a claim about causality and containment: a child span asserts its parent was waiting for it. Get that wrong — most often by making a queued job a child of the request that enqueued it — and the waterfall stops describing anything real.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
What nesting actually asserts
A child span makes two claims: this work was *caused by* the parent, and the parent was *still running* while it happened. Both halves matter. Causality alone is not enough to justify nesting, and this is precisely where async work breaks the model — an email sent forty seconds after checkout was certainly caused by the checkout, but the checkout span ended long before, so nesting it produces a parent that appears to run for forty seconds.
When both claims hold — a synchronous call out to a database, a cache, another service — parent/child is correct and the arithmetic works: parent duration should be at least the span of its children, and the difference is the parent's own work. That invariant is what makes self-time analysis possible, and violating it makes every duration in the trace suspect.
When causality holds but containment does not, the correct primitive is a link: a reference from the new span to the causing span, without nesting. The queued job gets its own trace, rooted at the consumer, carrying a link back to the producer. You keep the ability to navigate between them; you do not pretend one contains the other.
The waterfall that lies
The most common way a trace becomes misleading is a background task nested under the request that scheduled it. The request returns in 80 ms and the user is happy; the trace shows a 41-second root span. Every dashboard built on root-span duration now reports catastrophic latency for an endpoint that is fine, and — worse — the actually slow thing (a 900 ms SMTP call) is buried inside a span so large that nobody scrolls to it.
The mirror-image bug is under-nesting: work that genuinely blocks the parent recorded as a sibling or a separate trace, usually because a library created its own root. Now the parent has a large unexplained self time and the real cause sits in a trace nobody will join. This is where you get the confident-but-wrong conclusion "the service itself is slow, we need to profile it", when the service was waiting the whole time.
Both bugs are detectable mechanically, which is worth building: assert that every child span starts at or after its parent and ends at or before it, and flag traces where the root span exceeds the endpoint's p99 by an order of magnitude. Broken nesting is much easier to find with a query than by eye.
Choosing the relationship
The decision reduces to one question: was the parent blocked? If yes, nest. If no, link. Everything else — whether it is the same process, the same service, the same machine — is irrelevant. A synchronous in-process function call worth timing is a child span (INTERNAL kind); an async job in the same process is not.
Fan-out deserves a note. When a request makes five concurrent calls, all five are children and their spans overlap on the timeline. That is correct and it is exactly how the waterfall reveals parallelism: overlapping children mean concurrency, stacked children mean serialization. This is the shape Sequential or Parallel: Same Work, Different Latency teaches you to look for, and it only reads correctly if nesting is honest.
Batch consumers are the genuinely hard case: one worker pulls fifty messages that came from fifty different traces. There is no single parent. The honest model is a new trace for the batch with fifty links, which most backends will render as "this batch relates to these fifty traces" — imperfect, but not a lie. Nesting fifty unrelated request traces under one worker span would be.
| Situation | Relationship | Why |
|---|---|---|
| HTTP call to another service, awaited | Child (CLIENT → their SERVER) | Caused by, and the caller waits — both claims hold |
| Query to a database or cache | Child (CLIENT) | Same: synchronous and containing |
| Concurrent fan-out to five services | Five children, overlapping on the timeline | All awaited; overlap is what shows the parallelism |
| Enqueue a job | Child for the *enqueue*, link for the *job* | Enqueueing blocks; the job does not |
| Worker consuming a batch of 50 messages | New trace, 50 links | No single parent exists; nesting would invent one |
| Fire-and-forget notification | Link | Caused by the request, but the request never waited |
Key points
- A child span asserts both causation and containment: the parent caused it *and* was waiting for it. Async work satisfies only the first.
- Nesting a queued job under its enqueuing request produces a root span orders of magnitude longer than the endpoint's real latency, poisoning every derived dashboard.
- Under-nesting is the mirror bug: blocking work recorded in a separate trace shows up as unexplained parent self time and sends you profiling a service that was idle.
- Links express "caused by, but not contained in" — the right primitive for queued jobs, fire-and-forget work and batch consumers.
- Overlapping children mean concurrency and stacked children mean serialization, so honest nesting is a prerequisite for reading parallelism at all.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Instrumentation → span: the job scheduler passes the live request context into an async task, so the SDK makes the job a child.
- 2Job → trace: the job runs 40 s later and the parent span cannot end until it does, so the root span records 41 s.
- 3Trace → dashboard: latency panels built from root spans report a 41 s p99 for an endpoint whose server-side histogram says 80 ms.
- 4On-call → investigation: time is spent reconciling two contradictory latency numbers instead of on the 900 ms SMTP call that was the actual finding.
- • "Root span duration is our endpoint latency." Only if nesting is correct. When it is not, the server-side histogram is right and the trace is wrong.
- • "The job is caused by the request, so it belongs under it." Causality is necessary but not sufficient — containment is the second, missing half.
- • "Overlapping spans mean the trace is broken." Overlapping *siblings* are the correct rendering of concurrency. Overlapping parent/child boundaries are the bug.
- • "The service has huge self time, so its code is slow." It may be waiting on an uninstrumented or wrongly-parented call. Confirm with a profile before believing it (When the Trace Runs Out of Answers).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Query for traces where root-span duration exceeds the endpoint p99 by 10× — the classic nested-async-job signature.
- • Assert the containment invariant across a sample of traces: every child's start ≥ parent start and end ≤ parent end.
- • Compare root-span p99 against the endpoint's server-side latency histogram; a large divergence means the tree is wrong, not the service.
- • Look for services with large unexplained self time (parent duration minus children) as a hint of under-nesting or a library that started its own root.
- • Detach the context when scheduling async work: start a new trace in the consumer and add a link to the producing span.
- • Model the enqueue itself as a `PRODUCER` child (it is fast and synchronous) so the request trace still records that work was deferred.
- • For batch consumers, create one span per batch with links to each source trace rather than nesting unrelated traces.
- • Wrap libraries that create their own roots so their spans attach to the ambient context instead of floating free.
- • Root-span p99 should converge with the server-side latency histogram for the same endpoint after the change.
- • The containment assertion should pass across a fresh sample of traces.
- • The 900 ms SMTP call should now be the top span of its own short trace, i.e. visible instead of buried.
- • Confirm the link is navigable in the backend: opening the job trace should offer the originating request.
- • Links are less convenient than nesting: two traces to open instead of one, and backend support for link navigation varies.
- • Detaching context correctly is fiddly in runtimes with implicit context propagation, and easy to get wrong in both directions.
- • One span per batch loses per-message timing unless you add child spans, trading detail for a tree that is not a lie.
- • Enforcing containment assertions costs a small amount of ongoing telemetry processing.
- • Add the containment assertion to a periodic job over sampled traces and alert when violations appear.
- • Alert on divergence between root-span p99 and server-side p99 per endpoint — it catches this class of bug generically.
- • Code-review rule: any call that schedules work must either await it (child) or explicitly detach (link). No third option.
- • Cover it in the instrumentation test suite: enqueue a job in a test and assert the consumer starts a new trace with a link.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 41-second trace is constructed; the shape is common in codebases that pass request context into background schedulers.
- RUNTIME-SPECIFICHow easy it is to detach context depends on the runtime — implicit propagation (async-local storage, thread locals, Go contexts) makes accidental nesting much more likely than explicit passing.