Production Debugging
A method for narrowing from symptom to cause under time pressure, using six questions in a fixed order rather than intuition.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Production is broken, you have partial information and people are waiting — what do you actually do first?
Local debugging works by reproducing a fault and stepping through it. In production you usually cannot reproduce it, cannot step through it, cannot pause it, and are being asked for an ETA while you look.
Read the error, form a hypothesis about the code, then go looking in the code for confirmation. It is the method that works everywhere else, and it feels like the fastest route to an answer.
The error you can see is frequently downstream of the fault. A pool exhaustion error names the pool, not the slow query that drained it.
- The error you can see is frequently downstream of the fault. A pool exhaustion error names the pool, not the slow query that drained it.
- A hypothesis formed in the first thirty seconds becomes the thing you spend the next forty minutes confirming. Production gives you enough data to confirm almost anything if you only look where you expect.
- Code reading is unbounded work with no natural stopping point, and the code has usually been correct for months. What is new is rarely the source file you are staring at.
- While you diagnose, users are still failing. A method that reaches cause before it reaches mitigation optimises the wrong thing (Stop the Harm Before You Understand It).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Debugging in production is a search over a suspect space, not a proof. The goal of each step is to eliminate a large region of that space cheaply, not to be right.
- The suspect space has a shape you already know: your code, your configuration, your infrastructure, your dependencies, and your traffic. Every production fault lives in one of those five.
- Six questions cut that space fastest, roughly in this order — what changed, what is impacted, what is healthy, what is saturated, where is latency introduced, which dependency changed. Each one is answerable from a dashboard in under a minute if the system is observable at all.
- The order matters more than the questions. "What changed" is first because the prior probability is heavily on it (Deployment-Centric Debugging). "What is healthy" comes before "what is broken" because a working component eliminates everything behind it.
- What you are producing is not an explanation. It is a decision: roll back, fail over, shed load, scale, or keep looking. The explanation can arrive in the postmortem (Postmortems).
The six questions, in order
Each question is chosen for how much of the suspect space it removes per minute spent. That is the only ranking criterion — not how interesting the answer is.
Answer them from signals, not from memory or from people. "Nothing changed" said by a human is a hypothesis; a change feed showing no deploys is an answer.
- 11. What changed?
Lists deploys, config changes, infrastructure changes, dependency versions and traffic shifts in the incident window.
fails by No change feed exists, so "nothing changed" means "nobody remembers changing anything".
evidence A single view of changes across all four categories, timestamped against the symptom.
- 22. What is impacted?
Establishes blast radius: which users, regions, endpoints, tenants, operations.
fails by Metrics aggregated globally hide that only one region or one tenant is failing.
evidence Error rate broken down by region, endpoint and tenant, not just a single global line.
- 33. What is healthy?
Finds a working comparison — another region, another endpoint, reads while writes fail.
fails by Everything is measured only when it breaks, so there is no baseline to compare against.
evidence A component you can name as definitively fine, eliminating everything it depends on.
- 44. What is saturated?
Checks the finite resources: CPU, memory, connection pools, thread pools, queue depth, disk, file descriptors.
fails by Saturation panels exist for the host but not for the pools, which is where it usually is (The Connection Budget).
evidence Utilisation against a known limit for each pool, not just a host-level graph.
- 55. Where is latency introduced?
Locates the slow hop rather than inferring it from timeouts.
fails by Timeouts cascade, so every service reports being slow and none of them is the source.
evidence A trace waterfall showing which span grew (Reading the Waterfall owns the reading of it).
- 66. Which dependency changed?
Checks the things you did not change: managed services, third-party APIs, upstream providers, certificates, DNS.
fails by Dependency health is trusted rather than measured, so a provider degradation looks like your own bug.
evidence Per-dependency error rate and latency, measured from your side of the call.
The output of the sequence is a decision — roll back, fail over, shed, scale, keep looking — not an explanation.
What each answer eliminates
The reason to run the questions in order is that early answers make later ones unnecessary. A symptom confined to one region eliminates every explanation that is not region-specific, including almost all code changes on a uniformly deployed service.
This is the part that people skip. Elimination feels like it is not progress because nothing is fixed yet, but halving the suspect space twice is worth more than any single good guess.
| Observation | What it eliminates | What it promotes to prime suspect |
|---|---|---|
| Started exactly at a deploy timestamp | Almost everything unrelated to that release | The release, and the config that shipped with it |
| One region failing, others fine | Application code deployed identically everywhere | Regional infrastructure, a regional dependency, a partial rollout |
| One tenant failing, others fine | Shared code paths and shared infrastructure | Tenant data shape, tenant config, a hot key or noisy-neighbour effect |
| Reads fine, writes failing | The whole read path, most of routing and auth | The primary, a lock, a migration, disk or replication |
| Error rate flat but latency up | Anything that produces hard failures | Saturation, a slower dependency, a lost cache (Operating a Cache) |
| Gradual onset with no change | A discrete change as the trigger | Growth against a limit: disk, connections, memory, a table scan getting slower |
| Recovered on its own | Anything requiring intervention to clear | A transient dependency, a retry storm that drained, an expiring lock |
The ways the method itself fails
A method has failure modes like anything else in production. Each of these is common, and each has a cheap counter that has to be applied deliberately, usually by someone who is not the person deep in the investigation.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A plausible cause is found in the first minute | Every subsequent piece of evidence is read as confirming it; contradictions are explained away | Anchoring — the hypothesis is now being defended rather than tested | State the observation that would disprove it and go look for that specific thing |
| The loudest alert names a saturated resource | Long investigation of a pool or queue that is itself a consequence | Debugging the symptom nearest the alarm rather than the impact | Return to blast radius: what are users experiencing, and when did that start |
| Several people join the call | Three people on the same theory, none on the other branches, duplicated queries | No coordination of the search (Roles During an Incident) | Assign branches explicitly and have each report what they eliminated |
| Nothing is obviously wrong and pressure is rising | Several changes applied at once — restart, scale up, flag flip, config edit | Acting to feel productive rather than to test something | One change at a time, each with a stated expected effect and a way back |
| The system has no change feed | "Nothing changed" from three people, then a deploy is discovered an hour in | Change history lives in memory rather than in a queryable record | Mitigate on the blast radius instead, and fix the change feed afterwards (Deploys on the Same Timeline as the Symptom) |
| Symptoms stop mid-investigation | The incident is closed with no explanation and recurs the next day | Recovery mistaken for resolution | Keep the timeline open; an unexplained recovery is an unresolved incident |
How to do it properly
Most important first.
- Ask "what changed in the last few hours?" before opening any code. Deploys, config, infrastructure, dependencies, traffic (Change Correlation).
- Establish the blast radius next: all users or some, one region or all, one endpoint or the whole surface, one tenant or everyone. It tells you which layer can possibly be responsible.
- Find something healthy. If one region is fine and another is not, the shared code is exonerated. If reads are fine and writes are not, so is most of the request path.
- Check saturation before cleverness — CPU, memory, connection pools, thread pools, queue depth, disk (USE: Utilization, Saturation, Errors in Observability owns the method).
- Locate latency rather than guessing at it. A trace waterfall answers "which hop got slower" in one look; log reading does not.
- Say out loud what would prove you wrong, then go look for that. It is the cheapest defence against the first hypothesis.
- Mitigate the moment you have a plausible suspect and a safe reversal. You can be wrong about the cause and still be right to roll back.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains a slow diagnosis — every minute of it is a minute of user impact. What limits the damage is a mitigation you can apply before you understand the cause.
What can go wrong
- Anchoring: the first plausible story absorbs all subsequent evidence, and contradicting data gets explained away.
- Debugging the alert instead of the impact — spending twenty minutes on a saturated pool that is a symptom of the real fault.
- Two people investigating the same branch and nobody investigating the other four, because nobody is coordinating (Roles During an Incident).
- Changing several things at once to "see if it helps", which destroys your ability to attribute the recovery and leaves you with an unexplained fix.
- The method itself failing when the system is not observable: with no deploy log, no traces and no per-dependency error rates, all six questions are unanswerable and you are back to reading code.
- "The method replaces knowing the system." It does the opposite: it tells you where to point your knowledge, and someone who knows the system will answer the six questions much faster.
- "Find the root cause, then fix it." During an incident you are looking for a *lever*, not a cause. Stop the impact first; the contributing factors are postmortem work (Root Cause vs Contributing Factors).
- "The error message tells you what is wrong." It tells you where a failure surfaced, which is usually not where it started.
- "If we had better tools we would not need a method." Better tools make each question cheaper to answer; they do not tell you which question to ask.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can state the blast radius in one sentence — who is affected, where, and on which operations — within a few minutes of starting.
- Each step you took eliminated a named region of the suspect space, and you can say which.
- The mitigation you chose is one you can reverse if it turns out to be wrong.
- The timeline afterwards shows a decision, not a wander (The Debugging Timeline).
- The debugging itself is not a change, but the mitigations are: prefer the ones with a known reversal — rollback, flag flip, traffic shift — over the ones without, such as a manual data edit.
- If you must make an investigative change in production (raise a log level, enable a debug flag), write down how to undo it before you make it. Debug settings left on are their own future incident.
- Automate the answers, not the diagnosis: deploy annotations on every dashboard, a change feed, per-dependency error and latency panels, saturation panels for every pool (Dashboards an Operator Can Act On).
- Automate the correlation query — "what changed between 13:50 and 14:05" should be a link, not an archaeology exercise (The Audit Trail).
- Keep the hypothesis, the decision to mitigate, and the judgement of severity human. An automated rollback is fine; an automated conclusion about cause is not.
- The method is slower than a lucky guess and faster than an unlucky one. On a team that is usually lucky, it will feel like ceremony until the day it does not.
- It costs investment up front: the six questions are only cheap if the signals already exist. Building them is work that only pays during incidents.
- Mitigating before understanding sometimes destroys evidence — a rollback can remove the state you needed to diagnose the fault. Capture what you can first, but never at the cost of user impact.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe six questions hold for any system with real users, on any stack. Which dashboard answers them differs enormously; that they must be answerable does not.
- SIMPLIFIEDPresented as a fixed order for teachability. Experienced operators interleave and skip questions when the symptom is diagnostic on its face — a certificate expiry does not need a saturation check.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — why a timeout in one service is indistinguishable from a failure in the one behind it, and what that does to a diagnosis.