Action Items That Change the System
"Be more careful" is not an action item. "Add migration validation", "add a canary", "reduce the permission", "automate the verification" are.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What separates an action item that changes the system from one that only changes what people intend to do?
The analysis is good and the follow-up is a list of intentions. Six months later the same failure recurs, and the previous postmortem contains a resolved ticket titled "be careful with migrations".
The team now understands what went wrong. Write down what we will do differently, remind everyone, and move on.
Intentions decay. The people who formed them move teams; the new joiner never had them; the person who did have them is on hour nine of an incident and running on the fast, habitual path.
- Intentions decay. The people who formed them move teams; the new joiner never had them; the person who did have them is on hour nine of an incident and running on the fast, habitual path.
- A reminder is a control that depends on attention, and attention is the resource an incident consumes first. It fails exactly when you need it.
- Unowned items are unfunded items. A task with no owner, no ticket and no size competes with roadmap work and loses every sprint.
- "Add monitoring" is not actionable enough to start. What signal, on what dimension, alerting at what threshold, paging whom? The ticket sits because nobody can tell when it would be done.
- Items too large to schedule — "migrate to a new deployment system" — are real findings that will not happen. Recorded without being decomposed, they turn the list into a graveyard and discredit the process.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Controls form a hierarchy by how much they depend on a human being in a particular state. Eliminate the hazard, substitute something safer, guard so the mistake is impossible, detect so it is caught fast, procedure so the right steps are written down, remind so people intend to do it right. Durability falls steeply from left to right.
- Postmortems produce items from all six levels; the ones that get written are disproportionately from the right-hand end, because those are the cheap ones and the incident is over.
- The strongest items are usually structural and boring: a required argument with no default, a permission removed, a size bound restored, an automated verification step, a canary window. None of them require anyone to remember anything (Guardrails, Not Gates).
- Detection and recovery items generalise where prevention items do not. A tighter symptom alert catches every failure with that symptom; a faster rollback shortens every incident. Prevention only ever addresses the bug you already found.
- An action item is a change to the system. If it is finished when a document is updated and nothing about the running system differs, it was a note.
The same finding at six levels of durability
One finding — a migration was applied to the wrong environment — written as an action item at each level of the control hierarchy. Every row is a real thing a team could ship. They are ordered by how much they depend on somebody being alert at the moment it matters.
Note the cost column. The durable controls are not always expensive; the top three rows here are each roughly an afternoon. The reason teams land on the bottom rows is not cost, it is that the bottom rows can be written in the meeting and closed the same day.
| Level | Action item as written | Depends on a human? | Rough cost |
|---|---|---|---|
| Eliminate | Remove DROP and TRUNCATE from the migration role; destructive changes go through a separate reviewed path | No — the capability is gone | Hours, plus one workflow change |
| Substitute | Replace the ad-hoc migration CLI with the deploy pipeline's migration stage, which derives the target from the release | No — the target is not a human input | Days |
| Guard | Require an explicit --env with no default; print target host and database; confirm on statements matching DROP|TRUNCATE|ALTER .* DROP | Only to override deliberately | An afternoon |
| Detect | Alert on schema changes in production outside a deploy window, routed to the owning team | To respond, not to prevent | An afternoon |
| Procedure | Runbook step: verify target environment before applying, with the command that shows it | Yes — must be read and followed | An hour |
| Remind | Team reminder to double-check the environment | Entirely | Minutes, and it will not survive the quarter |
Rewriting the list
The left column is what postmortem lists usually contain. The right column is the same intent, specified to the point where somebody could start on Monday and you could tell when they had finished.
The test to apply to each one: could a reviewer approve this as a diff? If the answer is no, the item is a topic, not a task.
1. Be more careful with migrations. 2. Add monitoring for checkout. 3. Improve the runbook. 4. Review permissions. 5. Consider canary deployments.
1. Require explicit `--env` on the migration CLI; no default. Owner A, ~1 day. 2. Page on checkout success rate below 99.5% over 5 minutes, per provider. Owner B, ~1 day. 3. Add "payment provider rejects request" to the checkout runbook: symptom, the per-provider panel to check, mitigation (roll back), escalation (payments team), and how to verify recovery. Owner C, ~2 hours. 4. Remove `DROP` from the `migrator` role in production; destructive DDL via reviewed path only. Owner A, ~1 day. 5. Add a 10-minute canary stage at 5% traffic to the checkout pipeline, gated on error rate against baseline. Owner D, ~3 days.
Every item on the right names a specific artifact, has an owner and a size, and is finished when something about the running system differs. Items 2 and 5 also generalise: they catch failure modes nobody has thought of yet, which the items on the left never will.
How good items go bad afterwards
The second-order failures are worth knowing about, because each one starts as a correctly-specified item from a good postmortem. The pattern is that a control is added, its cost lands on people who were not in the incident, and nobody revisits it.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| An alert added per incident, for a year | Pages that nobody acts on; real alerts lost in the volume | Thresholds picked to catch the past incident rather than derived from user impact | Review alert value quarterly; delete alerts with no action taken (Alert Fatigue) |
| An approval step added after each severe incident | Delivery slows; changes batch up; risk per change rises | Gates accumulate and are never removed, because removing one requires arguing against an incident | Prefer automated guardrails to human gates; give each gate an expiry review (Guardrails, Not Gates) |
| Auto-remediation for a failure seen once | The automation fires on a different failure with the same symptom and makes it worse | A one-observation response encoded as a rule | Automate mitigation only for failure modes you have seen repeat and understand (The Automation Trap) |
| A guard on a common legitimate operation | People discover a workaround and use it routinely | Friction on the safe path exceeded friction on the unsafe one | Make the safe path fast; measure how often the override is used (Golden Paths) |
| An item assigned to another team without asking | Open for two quarters, then closed as stale | Ownership assumed rather than agreed | Agree the owner in the review, or record it as a deferred finding instead of a ticket |
| A runbook step added after every incident | A forty-step runbook nobody can follow at 3am | Additive maintenance with no pruning | Prune when adding; a runbook is read under time pressure (Runbook Anti-Patterns) |
How to do it properly
Most important first.
- Write each item as a change someone could review: what file, what config, what permission, what alert rule. If you cannot say what the diff looks like, it is not specified yet.
- Give it one named owner, a ticket in the normal backlog, and a rough size. Two owners is zero owners.
- Reach for the durable end of the hierarchy first. Ask "what would make this impossible?" before "what would remind us?".
- Deliberately include at least one detection item and one recovery item. Both generalise beyond this incident; all-prevention lists are a signal that the analysis stopped at the trigger (Root Cause vs Contributing Factors).
- Decompose the large ones. "Move to a reviewed migration path" becomes "remove DROP from the migration role" plus "require explicit
--env" plus "add a dry-run stage" — three shippable pieces, the first of which is an afternoon. - Cap the list at what you will actually do. Findings you are deferring belong in the document as findings, honestly labelled, not as tickets that will be closed unresolved in a quarterly tidy-up.
- Track completion in aggregate and report it. Unclosed action items are the leading indicator of the process becoming theatre (Learning Across Incidents).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains a weak follow-up. It surfaces as recurrence, typically one to two quarters later, by which time nobody connects the two incidents.
What can go wrong
- Items closed by adding a warning box to a document, which is a reminder with extra steps.
- Automation added in haste that becomes its own failure mode — an auto-rollback with no debounce that fights a healthy deploy (The Automation Trap).
- A new required approval step added after every incident, until the delivery path is slow enough that people route around it and the gates protect nothing (Change Management).
- An alert added per incident, with no thresholds derived from the impact, producing noise that raises the cost of every future page (Alert Fatigue).
- Action items that quietly transfer work to another team without that team agreeing, which is how they end up unowned in practice while looking owned on paper.
- "Training and documentation are never valid action items." They are valid and they are weak. Use them alongside a structural control, not instead of one.
- "Every incident needs action items." An incident whose analysis is "we handled this correctly and the system behaved as designed" needs a record, not homework.
- "More action items means we learned more." A long list is usually a sign that nothing was prioritised and therefore nothing will be done.
- "The item is done because the ticket is closed." The item is done when the system behaves differently. Those are frequently not the same event.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Items resolve as merged pull requests or applied configuration, and you can point at the diff.
- The change is verified in the way the incident would have been: the alert fires against a replay of the failure conditions, or the guard rejects the command that caused it.
- A later near-miss is caught by a control an earlier postmortem added, and the incident record says so.
- Aggregate closure rate for incident actions is tracked and is not falling.
- Action items are ordinary changes and get ordinary rollback paths. A guard that turns out to block legitimate work should be reverted quickly rather than tolerated — a control people resent gets circumvented, which is worse than not having it.
- If an added alert proves noisy, tune or delete it within a couple of weeks. Alerts added after incidents are politically hard to remove later, so the window for honest evaluation is short (An Alert Should Demand Action).
- Automate the verification an item promises: if the item is "check the target environment before running migrations", the shipped form is a program that refuses to run without an explicit target, not a step in a document.
- Automate tracking — action items linked to the incident, visible until closed, reported in aggregate.
- Do not automate the decision about which items are worth doing. Prioritising against roadmap work is a judgement with real cost, and it needs the people who own the trade.
- Do not automate a response you do not yet understand. An automatic remediation for a failure mode you have seen once will fire on the second, different failure that produces the same symptom.
- Durable fixes cost more than reminders, and the cost is paid now against a benefit that is a non-event later. That is the hardest trade in this domain to argue for.
- Guards add friction to legitimate work. A confirmation on destructive statements slows every destructive statement, including the intended ones.
- A small number of well-specified items means explicitly declining real findings. Being honest about that is better than a long list, and it feels worse.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe control hierarchy is not software-specific — it comes from occupational safety and transfers because the underlying claim is about human attention rather than about code.
- ORG-SPECIFICWhether incident actions can pre-empt roadmap work, and who arbitrates when they cannot, is set by the organisation. Where they cannot, teams need a standing allocation for this work or the list will never close.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — the regression test as the canonical durable action item, and its limits when the failure is not reproducible.