7 lessons

Incident Debugging

Evidence-based diagnosis under pressure: reading a timeline, separating correlation from causation, watching the bottleneck move after every fix, and the trade-offs that make a system faster but worse.

Symptom→Signal→Measurement→Hypothesis→Evidence→Root Cause→Change→Validation→Regression Check

Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.

Debugging an Incident in Progress

Mitigation and diagnosis are different jobs, and doing them in the wrong order costs users minutes they never get back. Stabilize first, then form a hypothesis you can disprove in two minutes instead of browsing dashboards hoping something looks odd.

Symptom · A page fires, or support escalates: requests are slow or failing, and a channel full of people is proposing causes faster than anyone can check them.
Correlation Is Not the Root Cause

Two charts moved together at 14:03. So did four others. Establishing that one caused another needs a mechanism you can state, or an intervention you can run — and during an incident you usually have time for exactly one of them.

Symptom · Everything on the dashboard turned red within the same two minutes, and each person in the channel is confident about a different one of them being the cause.
Reading a Timeline: Observation Order Is Not Causal Order

The timeline shows database latency rising at 12:08 and API p99 rising at 12:10. That ordering is a property of your alert thresholds and scrape intervals as much as of the system — and the first thing you observed is routinely not the first thing that happened.

Symptom · A reconstructed incident timeline where every entry looks like it could be the trigger, and the team argues about which one to fix.
"What Changed?" — Deploy Markers and the Invisible Deploys

The highest-yield first question in any performance incident is what changed, and it is only answerable in seconds if changes appear on the same time axis as the metrics. Code deploys are the easy part; config pushes, feature flags and someone else's release are the ones that leave no mark.

Symptom · A step change in a performance metric with no obvious trigger, and a team that spends twenty minutes asking around in chat to establish whether anything shipped.
The Bottleneck Moves After Every Fix
▶ lab

You removed the CPU bottleneck and the system is still slow — because the constraint moved to the database, where it had been hiding behind the CPU limit all along. This is what success looks like, and predicting the next constraint is what separates a plan from a sequence of surprises.

Symptom · A carefully validated optimization lands, the targeted resource is now comfortably below its limit, and end-to-end latency improved far less than the arithmetic promised.
Every Optimization Buys Something and Sells Something

Caching buys database load and sells freshness. Compression buys bandwidth and sells CPU. Batching buys throughput and sells latency. There is no move that is purely faster — and the ones that appear to be are usually selling reliability quietly.

Symptom · A performance win that produces a new class of problem weeks later: stale data, higher bills, occasional data loss, worse tail latency, or an outage that the "optimization" made possible.
Performance and Observability Anti-Patterns

Every one of these is a plausible move that a competent engineer makes under pressure, and every one shares a single property: no measurement before, or no measurement after. That is the tell, and it is the only thing they have in common.

Symptom · Optimization work that ships regularly and produces no durable improvement in user-facing latency, alongside monitoring that grows continuously and answers fewer questions each quarter.