What Happens Between the Page and the Postmortem
Alert, acknowledge, triage, mitigate, recover, verify, learn — a defined sequence, so nobody has to invent one at 3am.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Something is broken in production and you have just been told. What happens now, in what order?
Under time pressure, with partial information and an audience, people improvise — and improvisation produces long incidents, duplicated work and decisions nobody can reconstruct afterwards.
Whoever notices starts debugging. When they work out what is wrong, they fix it and tell everyone. Process would just slow that down.
Debugging is unbounded and user impact is not. Time spent understanding the cause is time users spend failing, and the two activities compete for the same person.
- Debugging is unbounded and user impact is not. Time spent understanding the cause is time users spend failing, and the two activities compete for the same person.
- With no acknowledgement step, nobody knows whether the page was seen. Two people start working in parallel, both assuming the other has not, or nobody starts at all.
- Without a stated scope, the organisation has no idea how bad it is. Support answers customers with guesses; leadership asks the responder for updates, which slows the responder down (Telling People What Is Happening).
- Recovery gets declared when the graph looks better, not when users are actually served — so the incident is closed twice, and the second time with less patience.
- Nothing is written down while it happens, so the postmortem is built from recollection, which is confidently wrong about ordering (Reconstructing What Actually Happened).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- An incident is a period where the system is not delivering what users depend on and someone is actively working to change that. The process exists to make the sequence of moves predictable so cognition can be spent on the problem rather than on coordination.
- The stages are not bureaucracy; each one closes a specific failure. Acknowledge closes "did anyone see it". Triage closes "how bad and who else is needed". Mitigate closes "are users still being harmed". Verify closes "did our fix actually work". Learn closes "will this happen again".
- The stages are not strictly sequential. Communication runs continuously alongside all of them, and investigation continues after mitigation rather than preceding it.
- The single most important ordering property is that mitigation precedes diagnosis. You almost never need to know why something broke to stop it hurting people (Stop the Harm Before You Understand It).
- Every stage produces evidence. That is what makes the postmortem possible and what makes "we think it recovered" into "it recovered at 14:13, confirmed by the probe and the support queue".
The stages, and the failure each one closes
Read the middle column as the question the stage answers and the last as what tells you it actually happened. A stage with no evidence is a stage that gets skipped under pressure.
Investigation is deliberately not a stage. It runs from triage onward, alongside everything else, and it is never allowed to block mitigation.
- 1Detect
A signal says users are affected.
fails by Detection by customer report, because the alert set watches causes rather than symptoms (Alert on Symptoms, Not on Causes).
evidence Alert timestamp precedes the first support ticket.
- 2Acknowledge
A named person takes it, so nobody else has to wonder.
fails by Silent acknowledgement, or nobody responding and the escalation timer being the real detector.
evidence Acknowledgement time recorded against the page.
- 3Triage
Establishes scope and severity, and decides who else is needed.
fails by Guessing at impact because the dashboard cannot split by tenant or region.
evidence A written impact statement: what, for whom, since when.
- 4Mitigate
Reduces or stops user impact, cause not yet required.
fails by Debugging instead, or applying a mitigation with unexamined side effects.
evidence The symptom graph turns, and the turn lines up with the action (Deploys on the Same Timeline as the Symptom).
- 5Communicate
Keeps everyone else informed so they stop asking the responder.
fails by The responder is the channel and stops responding.
evidence Updates on a stated cadence, from someone who is not debugging (Telling People What Is Happening).
- 6Recover
Returns the system to normal — backlogs drained, capacity restored, data reconciled.
fails by Stopping at "errors are gone" while a queue is still hours behind (Operating Queues and Scheduled Work).
evidence Backlog age back to baseline; reconciliation completed.
- 7Verify
Confirms from outside that users are served.
fails by Confirming from the same internal metric that was misleading you.
evidence Synthetic probe passing and support queue quiet.
- 8Learn
Converts the event into a change to the system.
fails by A review that produces reminders to be careful (Action Items That Change the System).
evidence Action items with owners, and at least one that changes a default rather than a person's behaviour (Postmortems).
The order that matters most is Mitigate before understanding. Everything else can flex.
The first ten minutes
This is where incidents are won or lost, and it is almost entirely about not doing the interesting thing. The interesting thing is finding out why. The valuable thing is finding out how bad, and whether a known mitigation applies.
The two columns below are the same ten minutes spent two ways. Neither responder is incompetent; the left-hand one is doing what their training rewards.
00:00 page acknowledged 00:01 opens logs, greps for stack traces 00:04 finds a null pointer in the trace 00:07 reads the handler code 00:09 believes it is the tax module 00:10 starts writing a fix users failing throughout nobody else knows anything no impact statement exists
00:00 page acknowledged
00:01 dashboard: 4% of checkouts failing,
all regions, started 14:04
00:02 posts impact statement to channel
00:03 change feed: gateway config 14:03
00:05 reverts the config change
00:07 error rate returns to baseline
00:09 probe confirms; states mitigated
00:10 investigation continues, calmlyBoth responders end up understanding the incident. The right-hand one stops the harm at minute five and does the understanding without a clock running, and the left-hand one is still writing a fix while users fail. The mitigation was available before the diagnosis was — which is the usual case, not a lucky one.
Mitigations that do not require knowing the cause
The reason mitigation can precede diagnosis is that a small set of actions reduce impact for large classes of cause. You are not choosing the right one for the specific bug; you are choosing the one whose side effects you can accept.
Each has a cost, and each has a case where it makes things worse. That is what makes this a decision rather than a checklist.
Which lever reduces impact now, with side effects you can accept?
when A deploy, config or flag change lines up with the onset.
cost Loses whatever else shipped in it; unsafe if a migration has already contracted (Expand, Migrate, Contract).
when The affected path is behind a flag.
cost Nearly free and fast, which is exactly why the flag was worth having (Feature Flags: Deploy Is Not Release).
when The system is saturated and degrading everyone equally.
cost You are deliberately failing some requests to serve the rest (Load Shedding).
when The failure is scoped to one zone, region or dependency.
cost The target must have the capacity to serve everything, which is rarely tested (Region Failover).
when Capacity is the constraint and headroom is gone.
cost Takes minutes you may not have, and pushes the bottleneck downstream (How Autoscaling Fails).
when State-related degradation — leaks, stuck connections, exhausted descriptors.
cost Destroys the evidence that would have explained it, and can drop in-flight work (Graceful Shutdown).
when Impact is small and every available action is riskier than the symptom.
cost Requires the discipline to say so out loud and keep watching, rather than acting to look busy.
How to do it properly
Most important first.
- Acknowledge fast, even before you understand anything. Acknowledgement is a claim of attention, not a claim of competence.
- State scope in one sentence before doing anything else: what is failing, for whom, since when. It is also the first thing everyone else wants (Using Observability, Not Building It).
- Ask "what changed" immediately and in parallel with reading signals — it is the highest-prior-probability suspect (Deploys on the Same Timeline as the Symptom).
- Mitigate as soon as a mitigation exists, even a crude one: roll back, disable the flag, shed load, fail over, scale up (Rollback: Only Useful If It Is Actually Safe, Load Shedding).
- Escalate on a clock, not on a feeling. "If I have no working hypothesis in fifteen minutes, I pull in a second person" removes the hardest judgement from the worst moment.
- Keep a running log in the incident channel as you go — actions, times, what you observed. Writing it later costs more and is less accurate.
- Verify recovery from the user's side: the symptom alert clear, a synthetic probe passing, the support queue quiet. Not just the graph you were watching.
- Close the incident explicitly, and schedule the review while the details are still available (Postmortems).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The process itself is what contains blast radius — it is the mechanism by which impact is stopped before it is understood.
What can go wrong
- The mitigation makes it worse — a rollback that hits an incompatible schema, a restart that loses in-flight work, a failover into a region without the capacity to serve everything (Capacity During Failover).
- Two responders taking conflicting actions simultaneously because nobody owns the decision (Roles During an Incident).
- The responder becomes the communication channel and stops responding.
- Fixation on the first hypothesis, with contradicting evidence explained away rather than followed.
- Declaring recovery on a partial signal — one region healthy, one queue drained — while a backlog is still failing users.
- The incident quietly never ends: impact stops, nobody closes it, and no review is scheduled.
- The tooling needed to respond is behind the outage — the deploy system, the identity provider, the chat platform (Break-Glass Access).
- "Find the root cause, then fix it." That is the order for a bug in a test environment. In production the order is: stop the harm, then understand it (Stop the Harm Before You Understand It).
- "The incident is over when the error rate drops." It is over when users are served, which can lag by the length of a backlog.
- "Process slows us down." Improvisation slows you down; you just cannot see it, because there is no counterfactual in the room.
- "Only big incidents need this." The small ones are where the sequence gets practised cheaply.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Time of detection, acknowledgement, mitigation and recovery are all recorded, and came from systems rather than memory.
- The impact statement given at the start matched what the postmortem later established.
- Recovery was confirmed by a user-facing signal, not only by an internal metric.
- Someone outside the responding team could follow what happened from the channel log alone.
- Every mitigation must itself be reversible, and you should know how before applying it. A rollback that cannot be un-rolled-back is a second incident waiting behind the first (Roll Forward: When Going Back Is the Harder Option).
- If a mitigation does not improve the symptom within a stated window, undo it before trying the next one — stacked half-applied mitigations make the system unanalysable.
- Some actions cannot be undone: data deleted, messages dropped from a queue, a contracting migration applied. Those need explicit agreement before execution, not a fast decision (Destructive Migrations).
- Automate detection, paging, escalation timers, incident channel creation and the collection of recent changes into it.
- Automate the mechanical parts of common mitigations so they are one reviewed action rather than a sequence typed under stress — a rollback button, a flag kill switch, a traffic drain (The Agent Kill Switch).
- Automate timestamping: every action taken through tooling should appear in the log with its time, so the timeline builds itself.
- Do not automate the decision to mitigate, the choice of mitigation, or the declaration of recovery. Those are judgements with irreversible consequences (The Automation Trap).
- Process costs a small amount of time at the start of every incident, including the many small ones where it was not needed. It buys predictability in the few where it was.
- Mitigating before diagnosing sometimes destroys evidence — a restart clears the state that would have explained the bug. Capture what you can first, but not at the cost of prolonged impact.
- A heavyweight process applied to minor incidents trains people to skip it, which means it is also skipped on the major one (Severity: What It Should Reflect).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe sequence and its ordering property hold wherever software has users. What differs is the ceremony around it, not the moves.
- ORG-SPECIFICWho may declare an incident, who may roll back, whether a change advisory board must approve an emergency fix, and which stages are mandatory are all local policy. A regulated environment often requires a recorded approval before mitigation, which changes the timeline and must be designed for rather than discovered mid-incident (Change Management).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — rehearsing response through game days, so the first execution of this sequence is not during a real outage.