Learn and Improve
The stage that closes the loop: turning what production taught you into a merged change, rather than into a document and a resolution to be careful.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How does what production taught you become a change to the system rather than a memory?
Every incident, every slow rollout and every near-miss contains information about how the system actually behaves. Most of it is lost, because the stage that would have used it has no artifact, no owner and no deadline.
Write a postmortem after significant incidents, list some action items, and share it. The learning has happened because it has been written down.
Action items that say "be more careful" or "add monitoring" change nothing. They are descriptions of an intention, not changes to the system (Action Items That Change the System).
- Action items that say "be more careful" or "add monitoring" change nothing. They are descriptions of an intention, not changes to the system (Action Items That Change the System).
- Documents accumulate faster than anyone reads them. A repository of postmortems nobody reads is a record of learning, not learning.
- Only incidents get this treatment, so the much larger volume of information — near misses, slow deploys, repeated manual toil — never enters the loop at all.
- Action items with no owner and no date compete with feature work and lose, every time, silently.
- When metrics enter the picture they get optimised. A team measured on deployment frequency will deploy more often; whether that reflects healthier delivery is a separate question (CI/CD Anti-Patterns).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Learning becomes real when it changes something executable: a merged commit, a changed alert, a modified default, a deleted step, an updated runbook that someone then used.
- The unit of improvement is a specific change to a specific mechanism. "Improve migration safety" is a topic; "migration linting rejects a bare DROP COLUMN and requires an explicit override with a reason" is a change.
- Contributing factors, not a single root cause. Production failures are conjunctions — a change, a gap in verification, an alert that did not fire, an assumption about data. Picking one and calling it the cause discards the rest of the information (Root Cause vs Contributing Factors).
- Delivery metrics are useful as signals of where friction lives and destructive as targets. Lead time rising is a question worth asking; lead time as a quarterly goal produces smaller commits and no more safety (CI/CD Anti-Patterns).
- The feedback loop only closes if the improvement reaches the plan stage of the next change. That is a scheduling and prioritisation fact, not a documentation fact.
From finding to changed system
Each step here loses material. Being explicit about where it is lost is how you find which step your organisation is failing at, and they fail at different ones.
- 1Notice
Something is observed: an incident, a near miss, a rollout that nearly failed, recurring toil.
fails by Only incidents above a severity bar are noticed at all, so most information is discarded before entry.
evidence Near misses appear in the record, not only outages (Learning Across Incidents).
- 2Reconstruct
A timeline is assembled from evidence: deploys, alerts, actions, recovery.
fails by Reconstructed from memory days later, which reliably produces a tidier story than what happened (Reconstructing What Actually Happened).
evidence Timestamps sourced from systems, with gaps marked as gaps.
- 3Analyse
Contributing factors are identified — usually several, interacting.
fails by Collapsed into one root cause, discarding the factors that will recur (Root Cause vs Contributing Factors).
evidence The write-up names several factors and says which ones are being addressed and which are not.
- 4Decide
Each factor becomes a change, or an explicit decision not to change.
fails by Findings with no decision, which reappear identically after the next incident.
evidence Every factor has a disposition: a work item, or a recorded reason for accepting it.
- 5Implement
The change is made, reviewed and shipped like any other change.
fails by Action items deprioritised against feature work until they age out (Action Items That Change the System).
evidence Merged commits traceable to the incident.
- 6Verify the fix
The mechanism is exercised to confirm it does what was intended.
fails by A new alert or check that has never fired and is assumed to work.
evidence The guardrail was deliberately triggered once, in a controlled way.
Action items that change the system, and ones that do not
The difference is whether the item changes a mechanism or asks for more human vigilance. Vigilance decays; mechanisms persist. Both sides of this table are real items from the same class of incident.
| Finding | Item that changes nothing | Item that changes the system |
|---|---|---|
| Migration locked a large table | Be careful with migrations on big tables | Migration CI rejects statements without an explicit lock-timeout, with an override that records a reason |
| Rollback took 40 minutes | Improve rollback speed | Deployment manifests reference digests and the previous release is pinned; rollback is one command and is exercised monthly |
| Nobody noticed for two hours | Add more monitoring | Alert on the user-visible symptom that was actually failing, routed to the owning rotation (Alert on Symptoms, Not on Causes) |
| Config change broke startup | Review config changes more carefully | Config is schema-validated at startup and the deploy fails fast on an invalid value (Validate at Startup, Fail Clearly) |
| On-call did not know the procedure | Document the procedure | Runbook updated by the person who needed it, linked from the alert that fires (Runbooks) |
| Deploy went to everyone at once | Deploy more carefully | Progressive rollout is the default in the pipeline; all-at-once requires an explicit flag (Progressive Delivery: Exposure as a Dial) |
Signals, not targets
Delivery metrics are genuinely useful for finding where friction lives. They are destructive as goals, for the ordinary reason that any measure adopted as a target stops measuring what it measured.
Use them the way you would use a latency graph: to ask a question. Lead time doubled last quarter is worth investigating. Lead time must halve by Q3 produces smaller pull requests and no change in how safely anything ships.
Change failure rate has risen noticeably over two months. What is the right response?
when Always first. The number is a symptom; the question is which stage degraded.
cost Takes time and may find something uncomfortable, like a verification gate people have been overriding.
when Rarely useful on its own — the number can be improved by classifying fewer things as failures.
cost Optimises the measurement rather than the system, and damages the signal for future use.
when The investigation identified a specific defect class arriving unchecked.
cost Latency on every change, including the ones that were fine; needs an expiry review (Guardrails, Not Gates).
when Failures correlate with large batched changes, which is the common finding.
cost More deploys, more pipeline load, and a cultural shift that takes months (Change Size: Why Small Changes Are Safer, and When They Are Not).
when The rise reflects a deliberate, temporary trade — a migration period with known elevated risk.
cost Only defensible if it is stated, time-boxed and revisited; otherwise it is drift with a justification.
How to do it properly
Most important first.
- Turn every finding into a change with an owner and a due date, or explicitly decide not to and record why. An undecided finding is a finding you will rediscover during the next incident.
- Prefer changes that remove the possibility over changes that add vigilance: a guardrail beats a warning, and a warning beats a note in a document (Guardrails, Not Gates).
- Feed near misses into the same process. The rollout that nearly went wrong contains most of the information of the one that did, at a fraction of the cost (Learning Across Incidents).
- Track whether action items land. An action item completion rate below half is data about your process, not about your engineers.
- Use delivery signals — lead time, deploy frequency, change failure rate, time to restore — to find friction. Do not set them as targets (CI/CD Anti-Patterns).
- Write postmortems blamelessly and specifically. "The operator ran the command" is not a factor; "the command had no confirmation and no dry-run, and the runbook gave it without context" is (Postmortems).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains a loop that does not close. It shows up as the same class of incident recurring, which is the diagnostic signal for this stage failing.
What can go wrong
- The action item backlog: items created, never prioritised, closed in a quarterly cleanup without being done.
- A postmortem process so heavy that teams avoid declaring incidents to avoid triggering it — the process suppressing its own input.
- Improvements that add process where a mechanism would have worked, so the fix is a checklist item that erodes within months.
- Metrics adopted as goals, producing exactly the number requested and no change in reliability (CI/CD Anti-Patterns).
- Learning captured in a person rather than in the system: the engineer who now knows never to do that, and who leaves.
- "We do postmortems, so we learn." Postmortems are the input. The output is a merged change (Action Items That Change the System).
- "Five whys will find the root cause." It is one technique, it works for linear causal chains, and most production failures are not linear (Five Whys, and Its Limits).
- "Blameless means nobody is accountable." Blameless is about causes; accountability is about follow-through on the changes (Postmortems).
- "Improve the DORA metrics and reliability improves." The metrics are downstream signals. Optimising a signal directly is how you get the number without the property (CI/CD Anti-Patterns).
- "Small incidents are not worth analysing." Small incidents are the cheap way to find the factors that will combine into a large one.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can point at commits that exist because of a specific incident, and say which one.
- A class of failure that used to recur has stopped recurring, and you can name the mechanism that stopped it.
- Near misses are written up at all, which most organisations never reach.
- The runbook used during the last incident was updated afterwards by the person who used it (Runbooks).
- Action item completion is tracked and is not embarrassing.
- Improvements are changes and can be wrong. A new required check that produces mostly false positives should be reverted, not endured (Required Checks).
- Process added after an incident deserves an expiry review. Process is much harder to remove than to add, because removing it looks like removing safety.
- You cannot roll back a lesson learned, and you should not want to — but you can and should roll back a mechanism that turned out to cost more than it prevents.
- Automate the collection: incident timelines assembled from deploy annotations, alerts and chat, so reconstruction is not manual archaeology (Reconstructing What Actually Happened).
- Automate the tracking: action items as real work items in the same system as everything else, visible in the same planning.
- Keep human: the analysis. Determining contributing factors requires understanding intent and context, and is the part that must not be templated into a form-filling exercise (Five Whys, and Its Limits).
- Doing this properly costs engineering time that produces no feature. The return is real and diffuse, which makes it perpetually easy to defer.
- Every guardrail added after an incident constrains future changes, including the ones that would have been fine. Guardrails accumulate and are rarely audited.
- Blameless analysis takes longer than assigning fault and is the only version that produces usable information, because people describe what actually happened.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe principle — findings become executable changes — holds at any size. A two-person team does it in a conversation and a commit; the artifact-heavy version exists to make it survive turnover and scale.
- ORG-SPECIFICPostmortem format, severity thresholds for requiring one, and who owns action items are conventions that vary widely and are not facts about production engineering. What does not vary is that a finding without an owner does not get done.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — error budgets as the mechanism that turns reliability findings into a prioritisation decision rather than a preference.