Root Cause vs Contributing Factors
Why complex systems rarely have one cause, and why "human error" is a question rather than an answer.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why does asking for *the* root cause usually produce a worse account of the incident than asking what had to be true for it to happen?
Incident processes ask for a root cause, singular. The systems being analysed produce failure through several conditions holding at once, and the request forces an honest analysis into a dishonest shape.
Every failure has a root cause. Keep asking why until you reach it, write it down, fix it, and the incident cannot recur.
The search terminates on whichever answer feels sufficient, and "sufficient" is decided by who is in the room. Two competent teams analysing the same incident routinely name different root causes.
- The search terminates on whichever answer feels sufficient, and "sufficient" is decided by who is in the room. Two competent teams analysing the same incident routinely name different root causes.
- Naming one cause discards the others. The four other conditions that had to hold remain in place, so the same latent weakness produces a different-looking incident next quarter.
- The last change before the failure is over-selected, because it is the most visible. Often it is the trigger and not the interesting part — the interesting part is why the system had no defence against it.
- When the chain reaches a person, the search stops there, because a person is a satisfying-looking terminal node. "Human error" ends the analysis at precisely the point where the system questions begin.
- Single-cause thinking produces single fixes, which cluster in the cheapest category — a warning, a note in the docs, a reminder — because a single cause makes the incident look like a one-off (Action Items That Change the System).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Production failures are conjunctive. A change is deployed and the test suite had no coverage for that path and the canary window was too short to catch it and the alert threshold sat above the error rate it produced and the rollback needed a manual approval that took eleven minutes. Remove any one and the incident is smaller or absent.
- That structure means there is no privileged cause. There is a set of conditions, each of which is independently removable, and each removal shortens or shrinks the next incident of this class.
- A trigger is what made it happen now — usually a change, sometimes a traffic shift, sometimes a dependency. A contributing factor is what made it possible, undetected, or slow to fix. Triggers are nearly always visible; contributing factors are nearly always the ones worth money.
- Some incidents genuinely do have a dominant cause: a null dereference on a new code path, a certificate that expired. Even then, the honest account includes why it was not caught, because the missing detection is a separate and often more durable finding (Renewal: Automating the Thing That Expires).
- Hindsight bias is the mechanism that makes single causes feel right. After the fact, the path to the failure is the only path you can see, so the decision that led down it looks obviously wrong. At the time it was one of several reasonable options, chosen with less information than you have now.
One incident, five conditions
This is the checkout incident from Postmortems, drawn as what it actually was. The deploy is the trigger — the thing that made it happen at 14:02 rather than some other Tuesday. Everything else is a condition that was already true and would have been true for the next incident too.
The useful question for each node is the counterfactual: if only this had been different, what would have changed? The deploy alone would still have caused errors, but a tighter alert threshold would have paged at 14:08 instead of a customer calling at 14:25. Canary analysis would have caught it at 1% of traffic. Neither prevents the bug; both shrink it by an order of magnitude.
- Trigger — the deploy. Fixed by a code change; prevents this bug and no other.
- Prevention gap — no coverage of the non-US address path. Fixed by a test; prevents a family of bugs.
- Containment gap — full rollout with no canary window. Fixed once, helps every future deploy (Canary Analysis: Compared Against What?).
- Detection gap — a page threshold above the impact this failure produced. Fixed by alerting on the symptom users experience rather than on a round number (Alert on Symptoms, Not on Causes).
- Diagnosis gap — no per-provider error breakdown, so eleven minutes went to a wrong hypothesis. Fixed by a dashboard panel.
What to ask when the chain reaches a person
Every serious incident analysis eventually arrives at a human action: someone ran a command, approved a change, silenced an alert, chose an option. This is the point where the analysis either becomes useful or stops.
The reframe is a single question, and §81 states it plainly: do not ask why the person did it — ask why one human action was able to create that impact. A system where a single unverified action can affect everyone has a design property, and that property is what the postmortem is for.
Note that this is not softer than blame. It is considerably more demanding: instead of one reminder to be careful, it produces work for the people who own the tooling, the permissions and the rollout path.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Ran a command against the wrong environment | Production change nobody intended | Environment selected by ambient state — a shell variable, a last-used profile — and never displayed before execution | Require the target explicitly, print it, confirm destructive operations (Manual Production Changes) |
| Approved a change without reading it | A defect reaches production through review | Change too large to review meaningfully; approval is the only available response | Reduce change size; make review a real gate rather than a ritual (Review as a Gate) |
| Silenced a paging alert | A real failure goes unnoticed | That alert has been noisy for months and silencing it is the rational local choice | Fix or delete the noisy alert; treat silence rate as a signal (Alert Fatigue) |
| Skipped a step in a runbook | Mitigation does not work, or makes it worse | Runbook lists steps without reasons, so which are safe to skip is unknowable | Encode why each step exists (Runbook Anti-Patterns) |
| Deployed on Friday afternoon | Impact extends through a weekend at low staffing | Rollback requires people; nothing about the rollout was self-limiting | Make rollback fast and boring rather than making the calendar the control (Rollback: Only Useful If It Is Actually Safe) |
| Changed a config value directly in the console | Drift; the change is lost at the next apply, or persists invisibly | The reviewed path is slow enough that the console is faster under pressure | Make the safe path the fast path; reconcile console changes back into code (Configuration Drift) |
Trigger, permission, detection, recovery
Sorting factors into four buckets does one specific job: it stops every action item landing in the prevention bucket. Prevention is where teams instinctively go, and it is the bucket with the worst returns, because you are trying to enumerate the bugs you have not written yet.
Detection and recovery fixes generalise. A tighter symptom-based alert catches every failure that produces that symptom, whatever caused it. A rollback that takes two minutes instead of twenty shortens every future incident. Neither requires knowing what will break next.
| Bucket | Question it answers | Typical fix | How well it generalises |
|---|---|---|---|
| Trigger | What made it happen now? | Revert or fix the specific change | Not at all — one bug, one fix |
| Permission | Why was it possible to reach production? | Test coverage, validation, review, reduced blast radius | Within a family of similar mistakes |
| Detection | Why did we not know sooner? | Alert on the user-visible symptom; add the missing signal | Across every failure with that symptom |
| Recovery | Why did impact last as long as it did? | Faster rollback, kill switch, tested runbook, flag | Across every incident, regardless of cause |
How to do it properly
Most important first.
- Ask "what had to be true for this to happen?" and list every answer, rather than "what was the root cause?".
- Sort the list into trigger, conditions that permitted it, conditions that delayed detection, and conditions that slowed recovery. Four buckets, four different classes of fix.
- For each contributing factor, ask the counterfactual explicitly: if only this one had been different, would the incident have been avoided, smaller, or shorter? A factor that changes nothing is context, not a factor.
- When the chain reaches a human action, keep going. The next questions are: what information did they have, what did the tooling show them, what did the interface make easy, what would have caught the mistake, and why was the impact unbounded (Least Privilege in Production).
- Write the analysis so that someone who disagrees can point at a specific factor and argue about it. An account nobody can disagree with is not an analysis.
- Accept multiple durable fixes with different costs, and be honest that you will not do all of them now. Recording the ones you are deferring is more useful than pretending they were not found.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained by nothing in the moment; the cost is paid later, as a class of incident that keeps recurring with fresh triggers.
What can go wrong
- Contributing-factor analysis producing twenty factors, none prioritised, so nothing gets fixed and the process is judged useless.
- The reverse of blame: refusing to name any cause at all, producing a document that is diplomatically empty.
- Fixing only the trigger — reverting the change, adding one test — while every condition that let it reach production untouched.
- Treating a factor as fixed because a ticket exists for it. The system is unchanged until the change is merged and running.
- Confusing correlation with contribution. Something that happened during the incident and did not affect it is a distraction that consumes review time (Correlation Is Not the Root Cause).
- "There is never a root cause." Sometimes there is a dominant one. The claim is that the request for exactly one, always, distorts the analysis — not that causes do not exist.
- "Blameless means we cannot say a person made a mistake." You can and should say what happened, including what someone did. What you do not do is stop there, or treat the action as the explanation.
- "Contributing factors means everything is everyone's fault." It means the conditions are the target. Diffuse responsibility is a different failure and produces the same nothing that blame does.
- "We found the root cause, so we are safe from recurrence." You are safe from that exact trigger. The conditions that let it through are still there for the next one.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The incident document lists more than one factor, and each has a stated counterfactual.
- Action items span more than one category — prevention, detection and recovery, not three variations on prevention.
- A later incident with a different trigger is caught earlier because a detection gap named in an earlier postmortem was closed.
- The word "human error" does not appear as a cause anywhere in your incident records.
- Analysis has no rollback, but conclusions should be revisable. If new evidence arrives — a second occurrence, a log you could not reach at the time — amend the document rather than leaving a confidently wrong record for the next reader.
- If a fix derived from a mistaken causal claim makes things worse, revert it and say so in the record. The corrected causal claim is more valuable than the fix was.
- Automate correlation: surfacing what changed in the impact window — deploys, config changes, flag flips, infrastructure events — turns hours of archaeology into a list (Change Correlation).
- Automate factor tracking across incidents, so a condition named three times becomes visible as a pattern rather than as three unrelated notes.
- Do not automate causal attribution. Correlated change is a candidate, not a cause, and a tool that names one will be believed more than it deserves.
- Multi-factor analysis takes longer and ends without the satisfying closure of a single answer. Teams under pressure will feel the process is unfinished.
- It produces more action items than you can afford, which forces an explicit prioritisation conversation. That conversation is valuable and uncomfortable.
- Some stakeholders — executives, customers, occasionally regulators — want one sentence. Producing an honest summary sentence without collapsing the analysis is a real writing skill and takes time.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALConjunctive failure is a property of systems with redundancy and multiple layers, which is nearly all production systems. A single script on a single host can genuinely have one cause; a service behind a load balancer with a canary and an alerting stack essentially never does.
- ORG-SPECIFICWhether an incident record may say "several contributing factors" or must name one cause is set by process and sometimes by regulator. Where a single-cause field is mandatory, teams put the trigger in it and carry the real analysis in the body.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — coverage as one contributing factor among several, rather than as the thing that was supposed to prevent all of this.