Correlation Is Not the Root Cause
Two charts moved together at 14:03. So did four others. Establishing that one caused another needs a mechanism you can state, or an intervention you can run — and during an incident you usually have time for exactly one of them.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The third cause explains most coincidences
Traffic rises 40%. CPU rises. Latency rises. Queue depth rises. Cache hit rate falls. Five correlated metrics, and it is tempting to build a causal chain out of any pair of them: "CPU is high, that is why we are slow" or "the cache degraded, that is why CPU rose". Both are stories built on a coincidence that a single upstream change — more traffic — fully explains.
This matters because the fix follows the causal claim. If you believe high CPU caused the latency, you scale up. If the real driver was a traffic spike that a bigger cache would have absorbed, you scaled the wrong tier and the next spike does the same thing. Getting the arrow backwards is not an academic error; it produces changes that cost money and fix nothing (Performance and Observability Anti-Patterns).
The structural fix is to look for the *common ancestor* before accepting any A→B story. When several metrics move together, ask what could move all of them at once — traffic, a deploy, a config change, a dependency, a scheduled job, a data-volume threshold crossing. Most incident "mysteries" are one change with five downstream shadows, and naming the change collapses all five explanations into one.
A causal claim names a mechanism
The difference between a lead and a root cause is that a root cause survives the question "by what path?". "The deploy caused the latency" is a lead. "The deploy added a LIKE '%term%' filter to the orders query, which cannot use the existing index, so the planner switched to a sequential scan and per-query time went from 4ms to 340ms" is a mechanism — and it is falsifiable at three separate points, any of which can kill it in under a minute.
Mechanisms are also what make a fix predictable. If you know the path, you can predict the size of the improvement before you ship: removing that filter should return DB time per request to roughly its old value, and if p99 only drops halfway, your mechanism was incomplete and something else is also contributing. A fix that improves things by an amount you did not predict is a fix you do not understand yet.
When you cannot construct a mechanism, the honest position is "correlated, cause unknown" — and that is a perfectly respectable incident state. It tells you the next action is evidence gathering, not shipping a change. The failure is upgrading a correlation to a cause silently, because then the team stops investigating and ships a fix aimed at a shadow.
1observation: GC pause count rose at 14:002observation: p99 rose at 14:003conclusion: "GC is causing the latency — let us tune the heap"4 5unanswered:6 · how much of the p99 rise do the pauses account for?7 · are the slow requests the ones that hit a pause?8 · what made GC pressure rise at 14:00 in the first place?9 10result: two days of heap tuning, p99 unchanged.1mechanism:2 deploy 14:00 added per-item DTO allocation in the list path3 → allocation rate 40MB/s → 610MB/s [check: alloc profile]4 → young-gen collections 3/min → 44/min [check: GC metrics]5 → ~18ms pause each, ~1 in 12 requests [check: pause histogram]6 → predicted p99 contribution: ~+18ms on affected routes7 8observed p99 rise: +1.9s9 10verdict: mechanism is REAL but accounts for 1% of the rise.11 GC is a symptom of the same deploy, not the cause.12 keep looking — what else did that deploy change?The second engineer confirmed a real causal chain and then measured whether it was big enough to matter. That arithmetic — does the mechanism account for the size of the effect? — is what separates a root cause from a true-but-irrelevant finding, and it is the step almost everyone skips.
Testing a causal claim while the clock is running
Outside an incident, you establish causation by intervention: change one thing, hold everything else, measure. Inside an incident, you rarely get a clean experiment — but you often get a *natural* one. Canary deploys, a change that reached one region first, one shard that was not migrated, a client version that has not rolled out: each is a population that differs in the suspected variable and nothing else.
Natural experiments are enormously undervalued during incidents. "The two hosts that did not get the config push are fine" is close to a controlled experiment and takes thirty seconds to check. So is "the tenant on the old schema is unaffected". Before proposing a change, ask whether some part of the fleet has already run the experiment for you.
When no natural experiment exists, the deliberate one is to revert a single change and watch. This is why single-variable changes matter so much during incidents: shipping three fixes at once and watching recovery tells you that *something* worked, and leaves you with three superstitions and no knowledge. The discipline is identical to the one benchmarking demands (Benchmark Fallacies: Confident Numbers That Are Wrong) — one variable at a time, or the result means nothing.
| Evidence | What it establishes | Cost during an incident | How it fools you |
|---|---|---|---|
| Temporal coincidence | A and B moved together. Nothing more. | Seconds | Deploys and traffic peaks are frequent, so coincidences are cheap and abundant |
| Blast-radius match | The affected population matches the suspected variable exactly | Under a minute | Two changes often ship to the same population, so the match can be shared |
| Stated mechanism | A path from cause to symptom that predicts specific readings | Minutes | A real mechanism can still be far too small to explain the effect — check the size |
| Natural experiment | A population differing in this variable behaves differently | Minutes, if one exists | The unaffected population may differ in other ways too (older hardware, less traffic) |
| Deliberate revert | Removing the variable removes the effect | Minutes to hours; risky | Only valid if you reverted exactly one thing, and if nothing else changed meanwhile |
| Reproduction in staging | The mechanism is sufficient to produce the symptom | Hours | Staging load and data volume rarely match production, so absence of the symptom proves nothing |
Key points
- When many metrics move together, look for the common ancestor first — most incident mysteries are one change with several downstream shadows.
- A root cause names a mechanism: the path from cause to symptom, stated precisely enough to predict specific readings.
- Check the size, not just the direction: a mechanism that is real but accounts for 1% of the effect is a distraction wearing a root cause's clothes.
- Natural experiments — canaries, unmigrated shards, partial rollouts, lagging client versions — are usually already available and take seconds to check.
- Revert one variable at a time; shipping three fixes and watching recovery teaches you nothing about which one mattered.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Change → several subsystems: one upstream event (traffic, deploy, config, data growth) perturbs CPU, cache, queue and latency within the same scrape interval.
- 2Dashboard → responder: the metrics arrive as a wall of simultaneous red, with no ordering information at the resolution being displayed.
- 3Responder → narrative: the eye pairs two of the moving metrics and constructs an arrow between them, usually in the direction that matches the responder's expertise.
- 4Narrative → fix: work is directed at a shadow of the real change, so the metric being optimized improves slightly and the symptom does not.
- 5Fix → recurrence: the actual driver is untouched, so the next occurrence of the upstream change reproduces the whole pattern.
- • "They moved at the same time, so one caused the other" — a shared cause produces exactly the same picture, and shared causes are more common than direct ones during incidents.
- • "The mechanism checks out, so we found it" — verify the magnitude. A confirmed mechanism explaining 1% of the effect leaves 99% unexplained and the investigation open.
- • "We fixed it and it recovered" — if you changed several things, or if a queue drained on its own, recovery is not attribution. Multi-change recoveries produce team superstitions that survive for years.
- • "Staging cannot reproduce it, so that is not the cause" — production data volume, concurrency and cache state routinely make staging incapable of showing a real production mechanism.
- • "The deploy is always the cause" — often true, which is exactly why it deserves a check rather than an assumption; the times it is wrong are the incidents that run long.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Overlay the suspected cause and the symptom on one time axis at fine resolution — a 1-minute-resolution chart cannot tell you which of two events at 14:03 came first.
- • Split both metrics by the same dimension (region, version, shard, tenant): a real causal pair moves together *within* every slice, a shared-cause pair often does not.
- • Quantify the mechanism's contribution: if GC pauses are the claim, multiply pause duration by pause frequency and compare against the actual latency delta.
- • Look for an unaffected population that differs only in the suspected variable, and check it before proposing any change.
- • Check whether the symptom actually started before the suspected cause — the single most common way causal claims die.
- • State the suspected mechanism in one sentence with an arrow chain before acting on it, and name the reading that would falsify it.
- • Compute whether the mechanism is large enough to explain the observed effect; if it is not, keep the finding and keep looking.
- • Find a natural experiment (canary, region, shard, client version) before designing a deliberate one.
- • When reverting to test a hypothesis, revert exactly one thing, and record the time so the timeline stays interpretable.
- • Write "correlated, cause unknown" explicitly when that is the state — it directs the next hour at evidence rather than at a plausible-sounding change.
- • The fix moves the metric the mechanism predicted, by roughly the amount it predicted — a surprise-sized improvement means the model is wrong even when the direction is right.
- • The correlated-but-innocent metrics return to baseline *without* being touched, which confirms they were downstream of the same cause.
- • Removing the mitigation does not bring the symptom back once the real cause is fixed.
- • A natural-experiment population that was previously unaffected shows no change from your fix — if it improves too, you did not fix what you thought.
- • Demanding a mechanism slows the first response; in a severe incident, mitigate on the correlation and require the mechanism before shipping the permanent fix.
- • Fine-resolution metrics make ordering visible but cost storage and cardinality — you cannot keep every metric at 1-second resolution ([[cardinality]]).
- • Single-variable reverts are slower than reverting everything, and in a severe outage restoring service beats learning why.
- • Natural experiments are cheap but confounded; the unaffected region may differ in traffic, hardware or data volume as well as in the variable you care about.
- • Record the mechanism in the incident review, not just the fix — the next responder pattern-matches on mechanisms, and a fix without a mechanism teaches nothing (Reading a Timeline: Observation Order Is Not Causal Order).
- • Alert on the mechanism's earliest link rather than the final symptom, so the next occurrence pages before users feel it.
- • Keep deploy, config and flag markers on every performance dashboard so temporal claims are checkable in seconds ("What Changed?" — Deploy Markers and the Invisible Deploys).
- • Where a mechanism is subtle, add a load-test scenario or CI benchmark that reproduces it, so the arrow is re-established automatically (Regression or Tuesday? Telling a Real Change from Noise).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe GC arithmetic and the traffic-spike fan-out are constructed teaching examples. Real allocation rates, pause durations and their latency contribution vary enormously by runtime, heap size and collector.
- WORKLOAD-SPECIFICWhether a given metric pair is causally linked or shares an ancestor depends on your architecture. The reasoning transfers; the specific arrows in the diagram are one system's shape.