The question this answers
One request in ten thousand fails. How do I find out why, instead of guessing?
The ladder guarantees a bounded search, not an answer. Each rung either eliminates a dimension or fails to, and a rung you cannot answer is not a dead end — it is the finding, because it names exactly which instrumentation is missing.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
Each service knows what it did with the requests it saw. No service knows what the request looked like before it arrived or after it left, and none knows which of its peers handled the same logical request. The whole difficulty of distributed debugging is that the object you want to reason about — one user’s request — exists nowhere as a whole. It has to be reassembled from fragments, and the only thing that makes reassembly possible is an identifier every fragment carried.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Six questions, in order
The ladder narrows a failure from "the system is broken" to a specific line of code, one dimension at a time. It is ordered so that each rung reduces the search space for the next.
Which request? Get one concrete failing request id. Not a rate, not a graph — one identifier you can follow. Almost all failed debugging sessions failed here: the team reasoned about aggregates for an hour, and aggregates cannot tell you *why*. Which service? In the trace for that request, which span carries the error, and is it the first error or an error propagated from below? Look for the deepest failing span, not the top one. Which region or zone? If failures cluster in one, the cause is environmental, not logical, and half the hypotheses die immediately. Which version? If failures appear only on instances running a particular build, you are done — this rung ends more incidents than any other, and it is answerable in seconds if version is a dimension on your telemetry.
Which dependency? Within the failing service, which downstream call failed, timed out, or returned something unexpected? Which state transition? The last and hardest: what state was the entity in when the failure occurred, and what transition was attempted? Distributed bugs concentrate here — a workflow that received a step-3 event while in step 1, a record updated by two writers, a retry that arrived after a compensation.
The rungs are cheap when the data exists and impossible when it does not. Which is the real content of this lesson: the ladder is a specification for what your telemetry must carry. Request id propagated end to end. Service and span identity. Region, zone and version as dimensions on every signal. Dependency calls as child spans. State transitions logged with before and after.
Why aggregates cannot answer "why"
A 0.1% error rate is a fact about ten thousand requests and tells you nothing about any of them. The distribution hides the mechanism: those hundred failures may be one bad instance, one poisoned key, one tenant with an unusual payload, one region, one version, or a hundred unrelated causes. Every one of those produces the same number on the dashboard.
This is why the first rung is "get one request". A single complete trace of a single failing request contains more causal information than a month of aggregate metrics, because it preserves the ordering and the identity that aggregation destroys. Metrics tell you *that* and *how much*; only a trace or a correlated log set tells you *why*.
The corollary is a sampling problem. Traces are usually sampled, and uniform sampling keeps mostly successful requests — the ones you do not need. Tail-based or error-biased sampling, which decides after the request completes, is what makes rung one answerable for rare failures. If your sampler is uniform at 1%, a 0.1% failure mode is invisible one time in a thousand, and rung one fails for the exact failures you care about.
# aggregate view — true, and useless checkout_error_rate 0.0011 (rising since 14:20) # ladder view — 4 minutes of work, if the dimensions exist 1. which request? req_8f3a21 (from an error-biased trace sample) 2. which service? deepest failing span = inventory-svc, not checkout 3. which region? 97% of failures in eu-west-1; other regions clean 4. which version? 100% on build 4471; build 4470 instances clean 5. which dependency? inventory-svc -> stock-cache, connection refused 6. which transition? reserve() attempted on an entity already in RESERVED finding: build 4471 changed the cache client's default port; only eu-west-1 had finished rolling. Rung 4 alone would have ended it.
Heisenbugs: failures that vanish under observation
Some distributed bugs are timing-sensitive, and the act of observing them changes the timing enough to hide them. Attach a debugger and the race resolves the other way. Add a log line in the hot path and the window closes. Enable full tracing and the added latency reorders two messages that used to overlap. Reproduce it in staging with one instance and the interleaving that caused it cannot occur at all. These are heisenbugs, and distributed systems produce them abundantly because the timing is set by a network nobody controls.
The distributed variety has a particular shape: the failure needs two events to land in a specific order at a specific separation — a message overtaking another, a retry arriving during a compensation, a lease expiring between check and use, a leader change landing mid-write. The window may be milliseconds wide, and anything that widens or narrows it makes the bug appear or disappear. Concurrency owns this phenomenon as heisenbugs; what is different here is that you have no shared memory to inspect and no single scheduler to control, so the techniques that work inside one process do not transfer.
What does work: stop trying to reproduce and start recording. Always-on, low-overhead, structured records that carry the identifiers and the ordering information — request id, causal parent, sequence numbers, state before and after — so that the *next* occurrence is fully described without you being present. Sample by error rather than uniformly. Keep exemplars. Record the state transition, not just the error, because for this class of bug the error is far downstream of the cause.
And the second technique: reason about ordering with causality rather than time. If two events are related, record the relationship explicitly — a parent span id, a causal token, a version vector — because [[causal-ordering]] survives observation while wall-clock timestamps do not.
- Symptom: fails reliably in production, never under a debugger, never in staging.
- Cause shape: two events must land in a specific order within a narrow window.
- Trap: adding instrumentation to catch it changes the timing that produces it.
- Technique: record continuously with causal identifiers rather than trying to reproduce.
- Technique: bias sampling toward errors, and retain exemplars of the failing case.
- Technique: log state transitions with before and after, not just the resulting error.
The rung you cannot answer is the deliverable
When a rung is unanswerable, the incident produces a better artefact than the fix would have. "We could not determine which version" is a concrete, cheap, permanent piece of work: add version as a dimension to error metrics and spans. "We could not follow the request past the queue" means the request id is not propagated through the message envelope. "We could not tell what state the order was in" means transitions are not logged with before and after.
Each of those turns "improve observability" — an action item that never gets done because nobody knows when it is finished — into a specific change with a clear completion test: run the ladder again on the next incident and see whether the rung answers.
A useful discipline for postmortems: record which rungs answered and which did not. Over a handful of incidents the same rung fails repeatedly, and that rung is where the observability investment belongs. This is more reliable than a wish list, because it is driven by what actually blocked real diagnosis.
Key points
- Six questions in order: request, service, region, version, dependency, state transition.
- Rung one requires one concrete failing request id — aggregates cannot tell you why.
- "Which version" ends more incidents than any other rung, and costs one telemetry dimension.
- A rung you cannot answer names the missing instrumentation, which is a better output than a guess.
- Heisenbugs vanish under observation: stop reproducing, start recording with causal identifiers.
- Uniform trace sampling keeps the requests you do not need; bias toward errors.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Obtain one failing request identifier, from an error-biased trace sample or an error log carrying the id.
- • Retrieve the full trace and find the deepest span carrying an error, not the topmost.
- • Filter the failure population by region and zone to separate environmental causes from logical ones.
- • Filter by build version to test whether the failure is a deploy.
- • Within the failing service, identify which downstream call failed and how.
- • Retrieve the entity’s state and the attempted transition at the moment of failure, plus any concurrent writers.
- • Record which rungs could not be answered, and treat each as a specific instrumentation task.
- • The request id is not propagated across an async boundary, so the trace ends at the queue.
- • Sampling dropped every example of the failing request class.
- • Clocks differ between services, so log ordering is misleading — see
[[correlating-distributed-logs]]. - • Version, region and zone are not dimensions on error signals, so rungs three and four cannot be answered at all.
- • The failing state transition is not logged, only its consequence, so rung six can only be inferred.
- • Adding the instrumentation needed to see the bug changes the timing and the bug disappears.
- • Debugging by aggregate: the operator spends an hour comparing dashboards and forms three hypotheses, because no single failing request id was ever obtained.
- • Trace ends at the boundary: the operator follows a request through four services and loses it at a queue, because the id was not carried in the message envelope.
- • Wrong service blamed: the operator sees the topmost span in error and pages that team, while the actual first failure is three spans deeper and belongs elsewhere.
- • Sampled away: the operator has an exact request id from a customer report and finds no trace for it, because sampling is uniform at 1%.
- • Heisenbug disappears: the operator adds debug logging to the suspected path, the failure rate drops to zero, and it returns the moment the logging is removed.
- • Version invisible: the operator cannot tell whether failures are confined to the new build, and the deploy is rolled back on suspicion — sometimes correctly, always without evidence.
- • The ladder itself requires no coordination between services at runtime, but it requires prior agreement on identifier propagation — every service must forward the id it received, including across queues.
- • That agreement is the coordination cost of debuggability, and it is paid once, at design time, in the request contract.
- • Rung two crosses team boundaries, so span ownership metadata matters: a trace that names services but not owners still leaves you asking in a chat channel.
- • Nothing about the ladder should require the failing service to be healthy — telemetry that must be scraped from a saturated process is unavailable exactly when needed.
- • Trace and log collection is usually best-effort: under load the telemetry pipeline sheds first, so the data is thinnest during the worst incidents.
- • Sampling decisions made at ingress cannot be revised later, so a request not sampled is permanently unavailable.
- • Identifiers survive failure well — an id in a message envelope outlives the process that created it — which is why they are the most robust debugging asset you have.
- • State-transition logs written to the same store that is failing will be missing exactly the transitions you need.
- • Detect: an error-biased sampler guarantees an exemplar exists for every failure class, which is what makes rung one answerable.
- • Contain: when rung four implicates a version, roll it back — that is containment, and it is available before you understand the bug.
- • Recover: fix the identified transition or dependency, not the symptom the top span reported.
- • Reconcile: entities that took the failing transition may be in an invalid state and need explicit repair; the error rate returning to zero does not fix them.
- • Verify: re-run the ladder on a request of the same class and confirm each rung now answers.
- • Request id propagated end to end, including through queues, retries and scheduled work.
- • Region, zone, instance and build version as dimensions on error signals and spans.
- • Error-biased or tail-based trace sampling, with retained exemplars per failure class.
- • Dependency calls as child spans, with status and duration — not as free-text log lines.
- • State transitions logged with entity id, before state, after state, actor and causal parent.
- • Per-incident record of which rungs answered — the empirical guide to where observability spend belongs.
- • Rare failures in a call graph deeper than two hops, where no single service’s logs contain the answer.
- • Incidents where several plausible causes exist and the team is about to choose one by seniority rather than evidence.
- • Any system where deploys are frequent, since rung four resolves a large fraction of incidents almost immediately.
- • A single-service failure with a clear stack trace — walking six rungs is slower than reading the exception.
- • When applied as ceremony during an active outage where containment, not diagnosis, is the priority. Roll back first; run the ladder afterwards.
- • Systems too small to have the dimensions the ladder queries, where building them costs more than the incidents do.
- • Roll back first and diagnose later: when a deploy correlates with the failure, containment beats understanding, and the ladder runs afterwards on retained telemetry.
- • Bisect by deployment or by feature flag: coarser than the ladder but often faster, and it needs no per-request instrumentation.
- • Reproduce in a controlled environment when the bug is deterministic — far cheaper than production forensics, and useless for heisenbugs.
- • Deterministic simulation replay for protocol-level bugs, where the whole execution can be replayed with the same interleaving.
One request in ten thousand fails. Six questions, in order.
| # | Question | Needs | Divides by | Candidates left |
|---|---|---|---|---|
| 1 | ✕ Which request? | a request id generated at the edge and propagated across every boundary — including queue messages, retries and background jobs | — | 907.2M |
| 2 | ✓ Which service? | that id on every service's structured log lines | ÷ 14 | 64.8M |
| 3 | ✕ Which region? | region and zone stamped on every error signal | — | 64.8M |
| 4 | ✕ Which version? | build version stamped on every error signal | — | 64.8M |
| 5 | ✕ Which dependency? | a span per outbound call, carrying status and duration | — | 64.8M |
| 6 | ✕ Which state transition? | state transitions logged with the value before and the value after | — | 64.8M |
What people believe, and what is true
The dashboard shows the error rate, so we know what is happening.
You know how much. Aggregates discard the identity and the ordering that carry the cause; one complete trace beats a month of them for answering why.
The service reporting the error is the one that is broken.
It is usually the one propagating an error from below. Look for the deepest failing span, not the topmost.
The bug is not reproducible, so we cannot debug it.
You do not need to reproduce it. You need the next occurrence to be fully recorded — identifiers, causal parents, state before and after.
Adding more logging will find the race.
For a timing-sensitive bug, added logging often closes the window and hides it. Record continuously and cheaply instead, and reason about causal order rather than log order.
We sample traces at 1%, which is plenty.
Uniform sampling keeps mostly successes. For a 0.1% failure mode it retains one failure in a thousand, so the first rung fails exactly for the requests you needed.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Which request, which service, which region, which version, which dependency, which state transition. Without those you are guessing.
Practical
Propagate a request id across every boundary including queues. Put region, zone and version on error signals. Sample traces by error, not uniformly. Log state transitions with before and after. Then record, per incident, which rung failed — that is your observability roadmap.
Advanced
The ladder is a search over a product space of dimensions, ordered by how much each one divides it. Version is early because it is usually binary and almost free to check; state transition is last because it is the largest space and needs the others to have narrowed it first. Reordering is legitimate when your priors differ — but the discipline of picking a dimension and eliminating it, rather than accumulating hypotheses, is what makes the search terminate.
Apply it
- 🔧 Take your last incident and mark each rung of the ladder as answered or not. The first unanswered rung is your next observability task.
- 🔧 Pick a request that crosses an async boundary and verify by inspection that the request id survives the queue, the retry and the dead-letter path.
- ⚡ Failures appear only on one instance of twelve, only for one tenant, and only after 14:00. Which rungs narrow this fastest, and in what order?
- 💬 One request in ten thousand fails. Walk me from that fact to a line of code.
- 💬 Which single telemetry dimension resolves the most incidents, and why that one?
- 💬 A bug fails in production and never under a debugger. What do you do instead of reproducing it?
- 💬 Your trace ends at a queue. What is missing, and where does it have to be added?