DebuggingGENERALRUNTIME-SPECIFICSCALE-SPECIFIC
Debugging a Backend in Production
Turning "the API got slow" into a named cause by narrowing the search space with evidence instead of guessing at fixes.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The question
An endpoint that answered in about 100 ms is now taking about 3 s at p99. How do you find the cause rather than guess at fixes?
The requirement
Checkout feels broken. Support is escalating, the latency graph is up and to the right, and two people have already suggested "restart it" and "add a cache". Someone has to find out what actually changed.
The obvious build
Open the slow endpoint, read the code, find something that looks expensive and optimise it. If that does not help, put a cache in front. If that does not help, add instances.
Why it breaks
Reading code finds *expensive-looking* code, not *changed* code. That handler has been the same for months; whatever moved is somewhere else.
How it breaks in production
- Reading code finds *expensive-looking* code, not *changed* code. That handler has been the same for months; whatever moved is somewhere else.
- A cache in front of a slow query hides the problem until the first mass eviction, which then arrives as a thundering herd against the same dependency (Cache Stampede).
- Adding instances multiplies connections into a database that is already the constraint, so the mitigation makes the symptom worse (Connection Pools).
- Three people change three things in twenty minutes, latency recovers, and nobody knows which change did it — so the incident repeats next week with the same confusion.
- The endpoint you are staring at is slow because *every* endpoint is slow. That is a process-level or dependency-level cause, and no amount of reading one handler will reveal it.
Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale
What is actually happening
- Latency is additive along the request path. Every millisecond is spent in exactly one place: waiting to be accepted, waiting for a runtime resource, running your code, waiting on the database, waiting on an external call, waiting for a pooled connection, or serializing the response.
- A jump of that size is almost never one code path becoming thirty times more expensive. It is a queue forming in front of something finite — a connection pool, a loop thread, a worker set, a lock (Queue Backlog).
- Something changed. Either you changed it (deploy, config flip, feature flag, migration) or the world changed it (traffic mix, data volume, a dependency, an expiring credential). Those are the only two categories, and the first has by far the higher base rate (Deploys Are the First Suspect).
- Narrowing is a search over the request path. Good evidence eliminates whole branches; a new hypothesis that eliminates nothing has not moved the investigation forward.
- The symptom names a layer long before it names a line. Whether one endpoint or all of them, one instance or all of them, sudden or ramped, erroring or merely slow — each answer removes roughly half the tree.
Establish the shape before forming a theory
The first ten minutes of an incident should produce no fixes at all. They should produce a shape: which requests, which instances, which moment, and whether errors accompany the slowness. Each of those answers is cheap to get from dashboards you already have, and each one eliminates a large part of the suspect list.
This matters because the suspect list is genuinely long — deploy, query plan, cache, dependency, CPU, event loop, pool, queue, network — and every one of them has an advocate on the call. The shape is what lets you say "it is not that" with evidence instead of opinion.
UserLLMAgentToolDataDecisionHumanGuardrail
| What you observe | Largely rules out | Points at |
|---|---|---|
| One endpoint slow, others normal | Process, host, runtime, network | That endpoint's query, its dependency, its payload |
| Every endpoint slow, same instance | A specific query or route | Event loop, GC, memory pressure, that host |
| Every endpoint slow, whole fleet | Anything instance-local | Shared database, shared cache, a dependency, a deploy |
| Step change at a precise minute | Gradual growth, data volume | Deploy, config flip, flag, migration, credential expiry |
| Slow ramp over hours or days | A single change event | Leak, unbounded table, cache degradation, queue backlog |
| Slow and erroring with timeouts | Pure CPU cost | Dependency down, pool exhausted, cascading failure |
| Slow but every request succeeds | Hard failure | Contention, queueing, a plan change, N+1 growth |
The suspect list, in prior-probability order
Suspects are not equally likely. Order them by base rate and by cost to check, and work down. In practice the top three account for most sudden latency changes, and all three are answerable in minutes without touching code.
Each row below is a hypothesis that predicts an observation. If the prediction does not hold, cross it off and move on — that is the whole method.
Check in this order
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| A deploy or config change landed near the onset | Step change aligned to a deploy marker across the fleet | New code, new dependency version, changed default, changed pool or timeout setting | Roll back first, diagnose second. Recovery on rollback is a strong signal (Rolling Deployments) |
| Database time grew | Query spans dominate the trace; slow-query log fills; DB CPU or IO up | Plan change after data growth, missing index, lock contention, or an N+1 that only now got wide (The N+1 Query Problem) | Read the plan for the actual slow statement, not for the statement you assume is slow |
| Connection acquire time grew, query time did not | Handler slow, database calm, pool waiters above zero | Pool too small for concurrency, or connections held across a slow external call (Connection Pool Exhaustion) | Shorten the hold, not just the pool size — raising the limit moves the queue to the database |
| Cache hit rate fell | Backend load spikes with no traffic increase; database suddenly busy | Eviction, restart, key change on deploy, TTL alignment causing synchronised expiry (Cache Stampede) | Confirm hit rate against the graph before assuming the database "just got slow" |
| An external dependency degraded | One span dominates; errors and timeouts alongside slowness | Their incident, your missing timeout, or a retry loop amplifying it (Circuit Breakers) | Check their status page and your own client timeouts (Timeouts) |
| CPU saturated | Host or container CPU near limit; throttling metrics rising | Heavier payloads, a hot serialization path, a regex, or a container CPU limit being hit | Take a sampling profile of the live process (What Serialization Costs) |
| Event loop or scheduler blocked | All endpoints slow together on a process with modest CPU | Synchronous CPU-heavy work on the request-serving thread (Blocking the Event Loop) | Measure loop lag or thread state; move the work off the request path (Worker Processes) |
| Queue backlog grew | Async effects late; consumers at full utilisation; queue age rising | Producers outran consumers, or a poison message stalled a partition (Queue Backlog) | Compare enqueue rate to completion rate before scaling workers (Worker Scaling) |
| Network or DNS degraded | Connect time up, transfer time up, errors sporadic and cross-service | Resolution failures, packet loss, cross-zone routing after a topology change | Separate connect time from response time in the client metric |
One hypothesis, one change, one measurement
The failure mode that turns a thirty-minute incident into a three-hour one is parallel guessing: several people change several things because each change "might help". Latency eventually recovers and the causal chain is unrecoverable.
The discipline that replaces it is small: state the hypothesis, state what it predicts, check the prediction, then make exactly one change and note the time. Mitigate early if users are suffering — but capture the evidence the mitigation will destroy before you apply it.
Two ways to spend the same twenty minutes
Guess and fix
10:02 "must be the DB" -> bump pool 20 -> 100 10:05 "maybe cache" -> deploy cache layer 10:09 "scale up" -> 4 -> 12 instances 10:14 latency recovers 10:15 which one worked? -> unknown 10:20 DB connections at max -> new incident
Predict and check
10:02 shape: all endpoints, whole fleet, step at 09:58
10:03 deploy marker at 09:57 <- suspect
10:05 hypothesis: new ORM version changed a plan
predicts: DB time up, pool wait flat
10:07 trace: DB span 2.6 s, pool wait 3 ms <- holds
10:09 EXPLAIN on that statement: seq scan
10:11 roll back release, note the time
10:14 latency recovers; cause named and written downThe second sequence produces a cause, so the fix is permanent and the runbook improves. The first produces a recovery with three candidate explanations, two new configuration changes nobody will revert, and a database now carrying triple the connections.
How to build it
Most important first.
- Establish the shape first. One endpoint or all? One instance or all? A step change at a moment, or a ramp? Errors alongside, or slow-but-successful? Four answers, and most of the suspect list is gone before you open an editor.
- Check what changed, in time order. Deploys, config changes, flag flips, migrations, dependency upgrades, certificate and secret expiry. Deploy markers on the latency graph are the cheapest diagnostic you will ever build (Deploys Are the First Suspect).
- Split the time. Proxy-observed latency versus handler-observed latency; inside the handler, database time versus external-call time versus CPU time. The layer that grew is the layer to investigate (Tracing From the Backend's Side).
- Follow saturation, not averages. Pool in-use and waiting counts, event-loop lag or thread-pool queue depth, worker idle ratio, queue age. A saturated finite resource explains rising latency better than any individual slow function.
- Form one hypothesis that predicts something you have not yet looked at, then go look. "The pool is exhausted" predicts long connection-acquire time and roughly unchanged query time — confirm that before you change anything.
- Change one thing and write down the time. A timestamped incident log is the only reliable way to attribute the recovery to a cause afterwards.
What can go wrong
Failure modes
- Fixing the symptom and losing the cause: a restart clears the leak, latency recovers, and the same thing happens on the same schedule next week (Memory Leaks in Backend Services).
- Anchoring — the first plausible story becomes the lens, and every subsequent graph is read as confirming it.
- Mitigations that amplify: more instances against a database bottleneck, more retries against an overloaded dependency (Retry Storms).
- Missing instrumentation exactly where you need it, so the invisible layer silently becomes the layer you assume is healthy.
- The diagnostic itself causing harm: an interactive debugger or a full heap dump can pause a live process long enough to fail health checks and remove the instance from rotation.
- Concluding "it fixed itself" when traffic simply fell. Recovery correlated with the daily trough is not evidence about your change.
What can race
- The investigation races the system: caches refill, queues drain and instances get replaced while you look, so the evidence you need may be gone by the time you think to collect it.
- Two responders mitigating simultaneously can produce a recovery neither of them can explain, and a regression when one of the changes is reverted.
Security
- Debugging touches production data. Query with the tenant filter applied and never paste rows into a chat channel with a wider audience than the data (Tenant Isolation).
- Heap dumps contain tokens, session identifiers and personal data in plaintext. Treat a dump as a credential-bearing artefact: restricted storage, short retention, deliberate deletion.
- Raising log level to debug in production can start writing full request bodies and authorization headers into a pipeline many people can read (Secrets in Logs).
- Incidents are when access control gets bypassed "just for now". Break-glass access should be time-boxed and audited, not granted permanently under pressure.
Misreads
- "The graph recovered, so we fixed it." Correlation with your action is weak evidence when traffic, a dependency and a cache were all changing at the same time.
- "CPU is at 30%, so it is not CPU." On a single-threaded event loop one saturated thread can pin latency while whole-host CPU looks idle (Blocking the Event Loop).
- "The database is fine, query time is flat." Flat query time with rising endpoint time is the *signature* of pool contention, not evidence that the data layer is uninvolved.
- "It only affects one customer, so it is not our problem." One tenant with an unusual data shape is one of the most common triggers of an N+1 or a missing index becoming visible (The N+1 Query Problem).
- "No deploy went out, so nothing changed." Config, flags, secrets, dependency versions and data volume all change without a deploy.
Operating it
How you see it in production
- Latency percentiles per endpoint and per instance. A p99 that moved while p50 did not is a tail or queueing problem; both moving is systemic.
- Deploy, migration and flag-change markers on the same time axis as the latency graph.
- Connection pool metrics: in-use, idle, waiting, and time spent waiting. Sustained waiters above zero is a diagnosis, not a hint (Connection Pool Exhaustion).
- The runtime saturation signal for your stack — event-loop lag, thread-pool queue depth, GIL-bound worker time. A request log cannot show any of these.
- One end-to-end trace of a genuinely slow request, with a span per database and external call. A single good trace routinely replaces an hour of argument (Correlation Ids That Survive Every Hop).
- Error rate next to latency. Slow-and-succeeding, slow-and-timing-out, and fast-and-failing are three different incidents.
What changes at 10x and 100x
- At larger fleets the same cause presents differently: a per-instance leak looks like random slowness across the fleet, because instances hit their limits at different times.
- More instances make shared dependencies the usual answer. When every instance is equally slow, look outside the instances.
- Sampling becomes mandatory — full tracing at high request rates costs more than the service. Sample by default and force-sample slow or erroring requests (Tracing From the Backend's Side).
- At small scale, one instance and one database, the search space is small enough that guessing sometimes works. That habit does not survive growth.
What this costs
- Disciplined narrowing is slower than a lucky guess for the first ten minutes and much faster for the next two hours. Under pressure that trade feels wrong and is not.
- Mitigating before diagnosing is often correct — users matter more than your curiosity — but every mitigation destroys evidence. Capture a trace, a dump or a pool snapshot *before* you restart.
- Deep instrumentation costs money and metric cardinality. Instrument boundaries everywhere and depth only where a boundary has pointed (The Metrics a Backend Must Emit).
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe narrowing procedure is stack-independent; every backend has a request path with a small number of places time can be spent.
- RUNTIME-SPECIFICThe saturation signal to check differs: Node exposes event-loop lag because one loop thread serves every in-flight request; a JVM or Go service shows a thread-pool or scheduler queue instead; CPython under a pre-fork server shows per-worker busy time, since each worker handles one request at a time.
- SCALE-SPECIFICWith one instance and one database, "restart and see" is a defensible first move. Across a fleet it destroys the only evidence and tells you nothing.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
Securitylogging-security
Performancesymptom-to-signalincident-debuggingmeasure-before-optimizinggolden-signalsdeployment-markers
Concurrencypool-saturation
Domains that do not exist yet
- — Testing & Reliability Engineering — incident command, blameless review and the runbook that turns one investigation into a repeatable procedure.