Postmortems
The written reconstruction of an incident — impact, timeline, detection, contributing factors, what went well and what changes — done blamelessly and still accountably.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What does a postmortem actually produce, and why is a document the right output of an outage?
An incident is the most expensive information a system ever generates. Without a deliberate act of reconstruction, that information stays in the heads of the three people who were awake for it, and the same failure recurs with different names.
We fixed it, we talked about it in standup, everyone knows what happened. Writing it up is bureaucracy that takes an afternoon we do not have.
The people who were in the incident already know. The postmortem is not for them — it is for the engineer who joins in six months and makes the same assumption about the same subsystem.
- The people who were in the incident already know. The postmortem is not for them — it is for the engineer who joins in six months and makes the same assumption about the same subsystem.
- Memory of an incident degrades within days, and it degrades in a specific direction: towards the story that makes the ending feel inevitable. The timeline you can reconstruct on the day is not the one you can reconstruct on Friday.
- Without a written impact statement, severity becomes whatever the loudest participant remembers. Prioritising the follow-up work becomes an argument about feelings.
- Fixes agreed verbally do not have owners, and work with no owner and no ticket does not happen. The next incident report describes the same contributing factor.
- Repeat patterns are invisible without records. Three separate incidents caused by the same missing validation look like three unlucky weeks, not one systemic gap (Learning Across Incidents).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A postmortem converts an experience into evidence. The raw material is timestamps: deploy records, alert firings, dashboard state, chat messages, ticket transitions. The document is what you get when those are placed in order and interpreted.
- The timeline is the load-bearing part. Everything else — cause, detection quality, response quality — is read off it. A timeline reconstructed from logs and deploy annotations is checkable; a timeline reconstructed from recollection is a story (Reconstructing What Actually Happened).
- The gap between when it started and when we knew measures detection. The gap between when we knew and when impact stopped measures response. These are separate problems with separate fixes, and conflating them is how teams buy a faster pager for a monitoring gap.
- Blamelessness is an epistemic tool before it is a kindness. If people expect consequences for what they say, you get a sanitised timeline, and a sanitised timeline is a useless one. You cannot fix a system you are being lied to about — not maliciously, just carefully.
- Accountability lives in the action items, not in the narrative. The organisation is accountable for changing the system; a named owner is accountable for each change landing. Nobody is accountable for having been the person at the keyboard.
The eight parts, and what each one is for
A postmortem template is not a form to fill in; each heading answers a question that is expensive to answer later. The test for whether a section earns its place is whether omitting it would leave a specific question unanswerable.
Order matters. Impact first, because it calibrates how much analysis the rest deserves. Timeline before cause, because the cause is read off the timeline rather than asserted over it.
- 1Impact
States what users experienced, for how long, and how many.
fails by Adjectives instead of numbers, so severity becomes negotiable after the fact.
evidence Error counts, affected tenants and duration, taken from the same signals that alerted (Alert on Symptoms, Not on Causes).
- 2Timeline
Places changes, signals, actions and recovery in verifiable order.
fails by Reconstructed from memory, which compresses the confusing middle and sharpens the ending.
evidence Each row traceable to a deploy record, an alert, a log line or a chat message.
- 3Detection
Says how you found out and how long that took.
fails by Recorded as "we were alerted" when a customer actually told you.
evidence The gap between first user impact and first human awareness, in minutes.
- 4Contributing factors
Names the several conditions that had to hold at once.
fails by Collapsed into one cause, usually the last one found (Root Cause vs Contributing Factors).
evidence Each factor stated so that removing it alone would have changed the outcome or its severity.
- 5Root causes
Names the deeper conditions, where they are genuinely identifiable.
fails by Asserted with false confidence; or a person named as if they were a mechanism.
evidence A causal claim someone could disagree with on the facts.
- 6What went well
Records the mitigations, tooling and decisions that worked.
fails by Skipped as morale filler, losing the only record of what to invest in.
evidence Specific: the flag that killed impact in two minutes, the runbook that was correct.
- 7What went poorly
Records what slowed detection, diagnosis or recovery.
fails by Written about people rather than about the system that shaped their options.
evidence A missing signal, a stale runbook, an unclear ownership record — nameable and fixable.
- 8Action items
Commits to specific system changes with owners.
fails by "Be more careful", "add monitoring", "improve documentation".
evidence Merged changes, tracked to closure (Action Items That Change the System).
Two sections are optional in practice and disproportionately valuable: what surprised us — assumptions the incident falsified — and what we still do not know, which prevents the document from claiming more certainty than the evidence supports.
Blameless and accountable are not opposites
The confusion is understandable, because both words are used to describe the same moment: an engineer ran a command and production stopped serving. The blameless move is not to pretend the command did not happen. It is to treat the command as data about the system.
The reframe that does the work: an individual action that can take production down is a property of the system, not of the individual. Any engineer with that access, that tooling and that day would have had the same chance of causing it. The question is not why this person did it; it is why doing it was possible, easy, unverified and unrecoverable.
Accountability then has somewhere real to live. The organisation is accountable for changing the conditions. Named owners are accountable for the changes landing. What nobody is accountable for is having been the person holding the tool the system left unguarded.
This is not a claim that behaviour never matters. Repeatedly bypassing controls, or concealing what happened, is a management conversation — a separate one, held separately, and not conducted through the incident document. Mixing the two destroys the document's value as evidence, permanently and for everyone watching.
An engineer ran a migration against the production database without checking the target environment, dropping the `sessions` table. Cause: human error. Action: remind the team to double-check the environment before running migrations.
A migration intended for staging was applied to production. The CLI defaults to the last-used profile, does not print the target environment before executing, and the migration ran with a role permitted to drop tables. No confirmation step existed for destructive statements. Actions: (1) require an explicit `--env` argument with no default, owner A; (2) print target host and database and require confirmation for statements containing `DROP`, owner B; (3) remove `DROP` from the migration role and route destructive changes through a reviewed path, owner C ([[destructive-migrations]]).
The first version has one cause, one lesson, and no change to the system — the next person meets exactly the same trap. The second names four independent conditions, each removable, each with an owner. It is also the more accurate account of what happened.
A timeline you can check
This is a reconstruction of a routine outage in the shape they usually take: a change that looked safe, a signal that arrived late, a mitigation attempted before the cause was understood, and a recovery that came from reversing the change rather than from insight.
Read the two gaps. Nineteen minutes from first user impact to a human knowing — that is a detection problem, and no amount of faster response fixes it. Then twenty-three minutes from awareness to impact ending, of which eleven were spent on a hypothesis that turned out to be wrong. Those are two different action items with two different owners.
- 14:02changeDeploy of
checkoutv412 completes. Rolling update, six of six instances healthy, no readiness failures. - 14:06signalFirst failed checkout requests. Error rate at 0.4% — within the noise band of the existing alert threshold.
- 14:14signalError rate reaches 3.1%. Still below the 5% page threshold. Nobody is looking.
- 14:25signalSupport escalates three customer reports of failed payments.
- 14:25actionIncident declared. On-call engineer paged manually by support, not by an alert.
- 14:31actionDashboard shows errors concentrated in one region. Working hypothesis: regional dependency degradation.
- 14:42actionHypothesis discarded — the payment provider status page and its latency signals are clean.
- 14:44signalDeploy annotation for v412 spotted on the error-rate graph. Timing matches first impact.
- 14:46actionRollback to v411 initiated. Cause still unknown.
- 14:48recoveryError rate returns to baseline. User impact ends.
- 15:30recoveryCause identified offline: v412 sends a stricter address format that one payment provider rejects for a subset of countries.
The lesson in the last two rows: impact ended at 14:48 and the cause was understood at 15:30. Mitigating before understanding is correct, and a timeline that shows it makes the practice teachable rather than folkloric (Stop the Harm Before You Understand It).
How to do it properly
Most important first.
- Write it while the evidence is warm — within a day or two, before dashboards roll off retention and before the chat scrollback is hard to find.
- Reconstruct the timeline from artifacts first, then ask people to fill the gaps. Reversing that order anchors everyone on the first story told.
- State impact in user terms and in numbers you actually have: requests failed, tenants affected, duration of degradation. "Some users saw errors" is not an impact statement.
- Separate detection from response explicitly, with a timestamp for each. They fail independently and are fixed differently.
- List contributing factors, plural, and resist collapsing them into one (Root Cause vs Contributing Factors).
- Include what went well, honestly and specifically. It is not morale decoration: the mitigations that worked are the ones you want more of, and they are otherwise never written down.
- Give every action item an owner, a ticket and a size (Action Items That Change the System).
- Publish it where people who were not involved can read it, and make reading it cheap.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains a missing learning process. It surfaces as the same class of incident recurring across unrelated services, which is invisible until someone counts.
What can go wrong
- A postmortem that is really a status report: what happened, when it was fixed, no analysis of why the system permitted it.
- The template becoming the work. Twelve required headings produce documents nobody reads and a process people route around.
- Blamelessness misread as consequencelessness, so nothing changes and the process gets a reputation for being theatre.
- A blameless document written in an organisation that is not blameless. The written norm does not override the lived one, and people will correctly protect themselves.
- The review meeting scheduled three weeks out, by which point the participants have moved on and the action items have quietly aged out.
- Postmortems only for the largest severities, so the recurring medium-sized failures — the ones that predict the large one — are never examined (Severity: What It Should Reflect).
- "Blameless means nobody is responsible for anything." Blameless is about the narrative, not the follow-up. Action items have named owners and due dates, and the organisation is accountable for the system changing.
- "We need a postmortem for every incident." You need a record for every incident and analysis proportional to what you can learn. A five-minute self-recovering blip does not need a document; three of them in a month do.
- "The postmortem is done when the document is published." It is done when the action items have landed. A document is an intention.
- "It was a one-off, so there is nothing to learn." One-offs are exactly where the system's undefended edges are. If it happened once with these conditions, the conditions can recur.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- The timeline in the document can be checked against deploy records and alert history by someone who was not there.
- Action items from the last quarter's postmortems are merged changes, not open tickets.
- Someone outside the responding team has read it and can explain what happened.
- A later incident report cites an earlier one — the record is being used, not just filed.
- The document itself has no rollback, which is the point: it is a record. What can and should be reversed is process weight. If the template has grown to the point where people avoid writing postmortems at all, cut it back — a short honest document beats a thorough absent one.
- If a change made in response to an incident causes its own problems, that is a normal deploy with a normal rollback path, and it belongs in its own postmortem rather than being quietly reverted (Rollback: Only Useful If It Is Actually Safe).
- Automate evidence collection: an incident timeline seeded from deploy annotations, alert firings, and the incident channel is most of the reconstruction work and none of the judgement (Deploys on the Same Timeline as the Symptom).
- Automate the bookkeeping: action items filed as tickets with owners, tracked to closure, and reported on in aggregate.
- Do not automate the analysis. A generated summary of an incident channel produces a plausible narrative, and plausible narratives are exactly the failure mode this process exists to prevent.
- Do not automate severity or blame classification. Both are judgements with consequences for people.
- A good postmortem costs several engineer-hours from the people who just lost a night of sleep. That cost is real and lands on people who are already depleted; scheduling it badly is how the practice gets resented.
- Publishing widely improves organisational learning and raises the personal exposure of the people involved. In a low-trust organisation, that trade goes the wrong way, and the honest move is to fix the trust before widening the audience.
- Deep analysis of every incident is not affordable. Choosing which ones get the full treatment is itself a judgement that will sometimes be wrong.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICWhich incidents get a postmortem, who attends the review, how widely it is published and whether it is ever seen outside engineering are conventions set by company size, industry and regulator. A regulated payments company may owe an external report on a fixed clock; a ten-person startup writing a page in a shared doc is doing the same thing correctly.
- GENERALThe underlying structure — impact, timeline, detection, contributing factors, what worked, what did not, actions — is stable across organisations. Only its weight and audience vary.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — how much confidence a pre-production test suite can honestly provide, and why the residue is what postmortems are made of.