The question this answers
The errors have stopped and the graphs are green. Is the system actually correct again?
None of the five steps guarantees correctness by itself. What the spine guarantees is a decomposition: each step has its own signal, its own owner and its own way of failing, so that "the incident is over" becomes a claim about five separate things rather than one vague feeling. Availability restored is a claim about step three; correctness restored is a claim about step four.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
During an incident every participant — human and process — holds a partial, stale view. A service knows its own error rate and the latency of its immediate dependencies. It does not know whether the fault is upstream or downstream of it, whether its peers see the same thing, or whether an operator has already begun a mitigation that will change its inputs. "The incident has ended" is never a local observation; it is an inference from several partial views, and it is the inference most often made too early.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Five steps, and the two nobody runs
An incident is not one activity. It is five, in order, and they answer different questions. Detect: does anyone know something is wrong? Contain: has the damage stopped spreading while we work? Recover: is the system serving correctly again? Reconcile: is the state that drifted while we were broken now repaired? Verify: can we show that, rather than assume it?
Almost every team executes detect, recover and a partial contain. The dashboards go green at the end of recover, the incident channel gets a thumbs-up, and the postmortem is scheduled. Reconcile is the step where somebody has to ask: *what state changed, or failed to change, during the window when we were degraded?* That question has no natural alert, because the symptom of skipping it is not an error — it is a row that is quietly wrong forever.
The ordering is not decorative. Recovering before containing is the single most common way a small incident becomes a large one: you restore capacity into a system that is still amplifying load, and the restored capacity is consumed instantly by the backlog that containment would have shed.
The reconcile gap is silent by construction
Consider a search index fed by a change stream. The indexer is down for forty minutes. During those forty minutes the database accepts eleven thousand writes, and the indexer’s consumer lag climbs. When the indexer comes back, one of two things happens: it replays from its last committed offset and catches up cleanly, or the broker’s retention already dropped the oldest events and it silently resumes from the earliest available offset.
In the second case, recovery succeeds. Lag returns to zero. Every graph is green. And several hundred documents in the index are now permanently stale, with nothing anywhere emitting an error about it. The only thing that ever reveals the gap is a comparison of the authoritative store against the derived one — which is exactly what [[reconciliation]] is for, and why it is a component of the system rather than a script somebody writes after the first time this happens.
The general shape: any state derived from another state can drift during an incident, and the drift produces no errors. Caches, search indexes, read models, denormalised counters, downstream partners’ copies of your data, and any workflow that was mid-flight when the failure hit.
# during recovery — everything a normal dashboard shows indexer_consumer_lag_seconds 0.4 OK indexer_error_rate 0.00 OK api_5xx_rate 0.001 OK p99_latency_ms 180 OK # what the reconcile job found four hours later documents_in_source_of_truth 4_812_339 documents_in_search_index 4_811_902 delta 437 <-- no alert exists for this line oldest_stale_document_age_hours 4.1
Each step has a different signal and a different failure
Treating the five steps as one blurred activity is why incident reviews produce action items like "improve monitoring". Split them and the action items become specific, because each step fails in its own recognisable way.
Note the asymmetry in the last column. Detect and recover fail loudly — you find out. Contain, reconcile and verify fail quietly, and you find out weeks later from a customer, a finance reconciliation, or a second incident whose root cause is the residue of the first.
| Question it answers | Signal that drives it | How the step itself fails | |
|---|---|---|---|
| Detecttypical | Does anyone know? | Symptom-level SLI breach, not a cause-level alert | Fires late, or fires on a cause that is not the one currently biting |
| Containtypical | Has the spread stopped? | Dependency error rates and retry-to-original ratio flattening | Skipped, because restoring capacity feels more urgent than shedding load |
| Recovertypical | Are we serving correctly? | Error rate and latency back inside the objective | Declared on the caller’s graph while a downstream is still catching up |
| Reconciletypical | Is drifted state repaired? | A delta between an authority and its derivations | Has no signal at all, so it is never started |
| Verifyassumption | Can we show it? | A re-run of the same comparison returning zero | Confirmed by the absence of errors, which proves nothing |
Contain before you recover
Containment is the step that buys time, and it is almost always the cheapest action available. Shed load. Disable the expensive feature. Drain the region. Stop the retry storm by cutting the retry budget to zero. Freeze the deploy pipeline so nobody adds a variable. None of these fix anything; all of them stop the blast radius growing while you work on the fix.
The reason it gets skipped is psychological rather than technical: containment actions are visibly harmful. Turning off recommendations, rejecting 20% of requests, or pausing a queue all look like *making the incident worse* on a dashboard. They are how you stop it becoming unrecoverable. This is the operational face of [[failure-containment]] and [[load-shedding]], and deciding these actions in advance — with thresholds and a named owner — is what makes them available at 03:00.
The test for whether containment exists in your system is not "could we do it", it is "is there a control that one on-call engineer can operate in under a minute without a deploy?" A containment action that requires a code change and a pipeline run is not a containment action.
- Load shed at the edge: reject the cheapest-to-reject traffic first, before capacity is exhausted.
- Feature kill switch: turn off the dependency that is failing, not the product that needs it.
- Retry budget to zero: stop callers converting one slow dependency into a self-sustaining overload.
- Drain a fault domain: remove the bad zone or region from rotation rather than debugging it live.
- Deploy freeze: stop adding new variables to a system you are trying to reason about.
Key points
- An incident has five steps, not three: detect, contain, recover, reconcile, verify.
- Recover restores availability. Reconcile restores correctness. They are different claims and need different evidence.
- The reconcile gap is silent: drifted derived state produces no errors, so no alert will ever start the step.
- Contain before recovering, or you restore capacity into a system that is still amplifying load.
- Verify means re-running the comparison and seeing zero, not observing that nothing is red.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A symptom-level signal crosses an objective and a human or automation is engaged — detect.
- • A pre-decided control is applied to stop the damage spreading: shed, disable, drain, freeze — contain.
- • The underlying fault is fixed or routed around, and the service resumes correct responses — recover.
- • Every store derived from an authority is compared against that authority for the incident window, and the delta is repaired — reconcile.
- • The same comparison is re-run and returns an empty delta, and the incident is closed against that evidence — verify.
- • The detection signal is itself served by the failing component, so the incident suppresses its own alert.
- • Containment actions exist only as documentation and cannot be executed without a deploy.
- • Recovery restores the service but replays a backlog that immediately re-saturates the dependency.
- • The reconcile comparison needs both stores healthy at once, and one of them is still catching up.
- • Verification runs against a cached or replicated read of the authority and confirms a state that is not the authoritative one.
- • Premature all-clear: the operator sees error rate at baseline and closes the incident, then a customer reports missing data four days later and there is no window in the logs to explain it.
- • Recovery-induced second peak: the operator sees the graphs recover and then spike higher two minutes later, because the queued backlog was released into a service still running at reduced capacity.
- • Silent derived drift: no operator observes anything at all until a scheduled comparison — or a finance reconciliation — reports a count mismatch with no corresponding error in any log.
- • Alert on the wrong layer: the operator is paged for high CPU on a node that is a symptom, while the failing dependency two hops away has no page attached to it and is found only by manual search.
- • Containment never applied: the operator sees dependency error rate rising monotonically for twenty minutes with no plateau, because every mitigation attempted was a fix attempt and none was a containment action.
- • Detection needs no coordination and should need none: a signal local to one service must be able to fire.
- • Containment usually does need coordination — shedding load at the edge changes what every downstream sees, so unilateral shedding by one team can mask the symptom another team is using to diagnose.
- • Reconcile requires both the authority and the derived store to be readable at the same time, which is a coordination point that can itself be unavailable exactly when you need it.
- • Verification is the one step that must not be coordinated with the recovery team’s beliefs: it should be a mechanical comparison, because a human who has just spent four hours fixing something is the worst available judge of whether it is fixed.
- • Durable state already committed before the failure remains committed; the spine does not risk it.
- • Invariants that span the failed component are unenforced for the whole window between detect and reconcile — not between detect and recover.
- • Any workflow that was mid-flight is in an indeterminate step, and will stay there until something drives it forward or compensates it.
- • Derived stores hold a snapshot of the moment the feed broke, and will serve it confidently until repaired.
- • Detect: alert on symptoms the user experiences, and make sure the alerting path does not depend on the component most likely to fail.
- • Contain: apply a pre-decided, one-minute, no-deploy control; accept that it looks like making things worse.
- • Recover: bring capacity back gradually rather than all at once, so the backlog does not immediately re-saturate.
- • Reconcile: run the authority-versus-derivation comparison bounded to the incident window, and repair the delta.
- • Verify: re-run the comparison; close the incident against a zero delta and record which stores you did *not* check.
- • Time from first user-visible symptom to first page — the honest measure of detect, and usually much worse than time-to-alert.
- • Whether a containment action was applied at all, recorded as a discrete event on the incident timeline.
- • The delta between each authoritative store and each store derived from it, computed continuously, not only after incidents.
- • Backlog depth and drain rate during recovery, so a second saturation is predicted rather than discovered.
- • Count of incidents closed with no reconcile step recorded — this is the metric that shows the habit, not the individual failure.
- • Any system where some state is derived from other state — which is nearly all of them once a cache or an index exists.
- • Incidents involving a queue, stream or replication feed, where the outage window maps directly onto a set of unprocessed changes.
- • Teams whose incident reviews keep producing "add monitoring" as the action item: splitting the spine turns that into five specific, differently owned gaps.
- • A stateless service with no derived state and no in-flight workflows: reconcile is genuinely empty and inventing work for it is ceremony.
- • Very short incidents where containment costs more user harm than the incident would have caused — a two-minute blip does not justify draining a region.
- • Running the full five steps for every minor alert trains the team to skip steps, which is worse than a shorter honest process.
- • For systems with no derived state, a three-step detect-contain-recover is honest and cheaper — but say explicitly that reconcile is empty, rather than forgetting it exists.
- • Continuous reconciliation instead of incident-triggered reconciliation: run the comparison always, and the incident step becomes "wait for the next pass" rather than "remember to do it".
- • Automated containment — an adaptive shedder or a breaker — replaces the contain step with a control loop, at the cost of a system that can now degrade itself for the wrong reason.
- • For workflows specifically, a durable orchestrator that resumes in-flight executions removes most of the reconcile surface by construction.
Detect, contain, recover, reconcile, verify — and where your team stops
# what the dashboard showed at the end of "recover" indexer_consumer_lag_seconds 0.4 OK indexer_error_rate 0.00 OK api_5xx_rate 0.001 OK # what a reconcile comparison finds writes_during_outage 10800 retention_window_min 30 events_past_retention 2700 <-- no alert exists for this line
What people believe, and what is true
The incident is over when the error rate returns to baseline.
That is the end of recover. The state that drifted while the error rate was elevated is still drifted, and it will never produce an error.
If reconciliation were needed, something would have alerted.
Nothing is measuring it. The characteristic property of derived-state drift is that every component involved reports success.
Containment wastes time we should spend fixing the problem.
Containment is what stops the problem growing faster than you can fix it. It is the cheapest action in the incident and the one most often skipped.
Verification means checking the dashboards once more.
Absence of errors is not evidence of correctness for a failure class that produces no errors. Verification is re-running the comparison.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Five steps: detect, contain, recover, reconcile, verify. Recover restores availability; reconcile restores correctness. Most incidents stop after recover.
Practical
Write down, per service, the containment control an on-call engineer can operate in one minute with no deploy, and the reconcile comparison for every store you derive from another. If either does not exist, that is the action item — not "improve monitoring".
Advanced
The spine is a decomposition by evidence type. Detect and recover are evidenced by symptom signals; reconcile and verify are evidenced by cross-store comparison; contain is evidenced by a discrete operator action on a timeline. Incidents blur because all five are argued from a single dashboard, and a dashboard can only ever supply the first kind of evidence.
Apply it
- 🔧 Take your last incident timeline and label each entry with one of the five steps. Report how many minutes were spent in contain, and how many entries fall under reconcile.
- ⚡ A cache warmer crashed overnight and restarted cleanly at 06:00. Error rate never left baseline. What, if anything, needs reconciling, and how would you bound the window?
- 💬 Your service was down for forty minutes and is now serving normally. What do you check before closing the incident?
- 💬 Name a containment action for your current system that an on-call engineer could take in under a minute with no deploy.
- 💬 A search index feeds off a change stream. The indexer was down past the broker’s retention window. What is now wrong, and what would tell you?