ReadinessGENERALORG-SPECIFIC

Runbooks

A document written for someone tired, under pressure, who did not build this: symptom, possible causes, checks, mitigation, escalation, and how to verify recovery.

What actually happens How to do it properly Evidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

What does an operator actually need in front of them at 3am, and how is that different from documentation?

The problem

The person responding to an alert is frequently not the person who built the service, is frequently not fully awake, and has minutes rather than hours. Architecture documentation is written for a reader with time and context. They have neither.

What teams do first

The service is documented — architecture, API, deployment. Anyone can read that and work out what to do.

How it breaks

Documentation is organised by structure and a runbook is needed by symptom. The operator has an error rate graph, not a component name; nothing in the architecture doc is indexed by what they can see.

How it breaks in production

Documentation is organised by structure and a runbook is needed by symptom. The operator has an error rate graph, not a component name; nothing in the architecture doc is indexed by what they can see.
Reading comprehension under sleep deprivation and time pressure is genuinely worse. Prose that is clear at 2pm is not clear at 3am, and the failure is not a character flaw.
Without a written mitigation, every incident is improvised, which makes response time a function of who happened to be on call.
Knowledge stays with whoever built it, so on-call is only really staffed by two people regardless of how many are on the rotation (Rotations People Can Sustain).
Nobody records how to tell it actually worked, so incidents get closed on the basis that the action was taken rather than that impact ended.

Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

A runbook is indexed by symptom, because a symptom is what the operator has. Everything else in the entry hangs off that: the symptom is the lookup key, and if the operator cannot find their symptom the rest of the document is inert.
Six parts do the work. Symptom — what you are seeing, phrased as it appears on a dashboard or in an alert. Possible causes — the small set of things that produce this, ordered by likelihood. Checks — the specific commands or panels that distinguish them. Mitigation — what stops user impact, with its own risks stated. Escalation — who to call, when, and what to tell them. Verification — the signal that confirms recovery.
Verification is the part that is skipped and the part that prevents the second incident. An action taken is not a recovery observed, and the signal that confirms it should be the same one that detected the problem (Alert on Symptoms, Not on Causes).
A runbook encodes understanding, not ritual. Each step should say what it is for, so an operator facing a variant nobody anticipated can reason from it rather than following it off a cliff (Runbook Anti-Patterns).
Runbooks decay in a specific way: the system changes and the document does not. The countermeasure is that they are exercised — read during real incidents and during rehearsals — because an exercised document reveals its own errors.

Six parts, in the order an operator needs them

The ordering is not stylistic. An operator arrives with a symptom and a clock. They need to know what stops the bleeding before they need to know why it is bleeding, and they need to know how to tell it stopped before they can close the incident.

The anatomy of one runbook entry

1
Symptom
Names what the operator is seeing, in the words the alert or dashboard uses.
fails by Titled by component instead of symptom, so it cannot be found from what you can see.
evidence The alert links straight to this entry.
2
Possible causes
Lists the small set of things that produce this symptom, most likely first.
fails by Exhaustive and unranked, so it reads as a list of everything that could ever go wrong.
evidence The causes match what has actually happened, drawn from incident records.
3
Checks
Gives the specific commands or panels that distinguish the causes, and says what each result means.
fails by Commands with no interpretation, leaving the operator with output they cannot read.
evidence Each check has a stated meaning for its likely outputs.
4
Mitigation
States what stops user impact, and what each option risks.
fails by A single unconditional action with no stated risk (Runbook Anti-Patterns).
evidence Mitigations that have actually been used, with their side effects written down.
5
Escalation
Says when to escalate, to whom, and what to bring.
fails by "Contact the team" — no threshold, no rotation, no information to hand over.
evidence A named rotation and a time or severity trigger (Roles During an Incident).
6
Verification
Names the signal that confirms impact has ended and the value that counts as recovered.
fails by Omitted, so incidents close on action taken rather than on impact ended.
evidence The same signal that detected the problem, back within its normal band.

One entry, written out

KUBERNETES-SPECIFICThe check commands assume Kubernetes. On a VM platform the equivalents are a process supervisor status and the service log; on a managed queue service they are console panels. The structure — distinguish the causes before choosing the mitigation — is what transfers.

This is a complete entry for a common failure. Notice how much of it is reasoning rather than instruction — what each check means, what each mitigation costs, when to stop and escalate. That is the difference between a runbook and a list of commands.

Notice also that the last section is not optional. Without it, "we restarted the consumers" gets recorded as a resolution, and the incident reopens forty minutes later.

runbooks/checkout-api.md — "Order queue consumer lag rising"

1SYMPTOM
2  Alert: checkout_queue_age_seconds > 300 for 5m
3  Users see: orders confirmed in the UI but no confirmation email,
4  and no fulfilment record. Impact grows with the backlog.
5 
6POSSIBLE CAUSES  (most likely first)
7  1. Consumers crash-looping        - most common, usually after a deploy
8  2. Downstream dependency slow     - fulfilment API latency
9  3. Genuine traffic spike          - backlog with healthy consumers
10  4. Poison message                 - one message failing repeatedly
11 
12CHECKS
13  Consumer health:
14    kubectl get pods -l app=order-consumer
15    -> CrashLoopBackOff or high RESTARTS means cause 1. Check the deploy
16       timeline before the logs: a restart loop that began within minutes
17       of a rollout is a bad release, not a runtime problem.
18  Consumer logs:
19    kubectl logs -l app=order-consumer --tail=100 --previous
20    -> The SAME message id failing repeatedly means cause 4, not cause 1.
21       A crash loop and a poison message look identical in pod status.
22  Dependency latency:
23    Dashboard: checkout-overview, panel "fulfilment API p99"
24    -> Above ~2s with healthy consumers means cause 2. The consumers are
25       working; they are blocked, so restarting them will not help.
26  Arrival rate:
27    Dashboard: checkout-overview, panel "queue arrival rate"
28    -> Well above the recent baseline with healthy consumers and normal
29       dependency latency means cause 3.
30 
31MITIGATION
32  Cause 1 (bad release):
33    Roll back the consumer deployment. Restarting without rolling back
34    reproduces the crash, and each restart drops in-flight messages.
35      kubectl rollout undo deployment/order-consumer
36    Risk: in-flight messages are redelivered. Handlers are idempotent by
37    message id, so redelivery is safe; verify no duplicate emails after.
38  Cause 2 (slow dependency):
39    Do NOT restart consumers - they are healthy and blocked. Reduce
40    concurrency to stop adding load, and page the fulfilment team.
41    Risk: lag grows while the dependency recovers. That is the correct
42    trade; the alternative amplifies their outage into ours.
43  Cause 3 (traffic spike):
44    Scale consumers up. Check the connection budget first - each consumer
45    holds 4 database connections and the pool ceiling is 200.
46    Risk: scaling past the pool exhausts connections for the API too.
47  Cause 4 (poison message):
48    Move the offending message to the dead-letter queue by id. Do not
49    purge the queue - that discards valid orders alongside the bad one.
50    Risk: the message is a real order and needs manual reconciliation.
51    Record the id in the incident channel.
52 
53ESCALATE
54  - Backlog still growing 15 minutes after mitigation -> payments-oncall
55  - Cause 2 confirmed                                 -> fulfilment-oncall now
56  - Any suspicion of data loss or duplicate charges   -> incident commander,
57    regardless of backlog size
58 
59VERIFY RECOVERY
60  1. checkout_queue_age_seconds back below 60 and falling.
61  2. Arrival rate and processing rate roughly equal - a falling age with
62     processing below arrival means you are still behind, just less so.
63  3. Spot-check three recent orders for a fulfilment record.
64  4. Only then close. Consumers running is not recovery; the backlog
65     being drained is.

The two most valuable lines are the ones telling you *not* to act: do not restart healthy-but-blocked consumers, and do not purge the queue. A runbook that only lists actions will get those wrong under pressure, because acting feels better than waiting.

Keeping it true

Runbooks fail by decay, and decay is silent: nothing breaks when a runbook becomes wrong, which is exactly the property that makes it dangerous. The defences are all forms of exercising it, because an exercised document reports its own errors.

How runbooks stop being true

Trigger	Symptom	Cause	Response
Service rearchitected	Commands reference components that no longer exist	Runbook lives apart from the code that invalidated it	Keep it in the service repository so the change that breaks it appears in the same review
Dashboard reorganised	Panel links 404 mid-incident	Links to mutable dashboard state	Link to stable dashboard ids; check links in the same job that checks the alerts (Dashboards an Operator Can Act On)
Never exercised	First real use reveals three wrong steps	Written from intention rather than from execution	Exercise in game days and drills; a step nobody has run is a hypothesis
Mitigation automated	The manual path in the runbook is stale when the automation fails	Automation replaced the practice, not just the toil	Keep the manual path documented and rehearsed — you meet it only on the worst day (The Automation Trap)
Grew by accretion	Forty steps; the operator cannot find the relevant one	Every incident added a step and none removed one	Prune when adding; a runbook is read under time pressure and length is a real cost (Action Items That Change the System)
Assumes unavailable access	Command refused mid-incident	Written by someone with broader permissions than the on-call rotation has	Write and test as the on-call role; if elevation is needed, the runbook says how to get it (Break-Glass Access)

How to do it properly

Most important first.

Write one entry per symptom, and make the alert link directly to its entry. The lookup should cost zero seconds.
Put the mitigation before the explanation. Stopping impact comes first; the operator can read the background afterwards (Stop the Harm Before You Understand It).
State what each check is for and what each result means, not just the command. kubectl get events is a command; "check for OOMKilled events — repeated kills mean the limit is too low for current traffic, not that the process is unhealthy" is a check (OOMKilled: Over the Memory Limit).
State the risk of each mitigation. "Restarting drops in-flight requests" is the sentence that lets an operator decide rather than guess.
Give escalation a trigger and a script: when to escalate, to whom, and what information to bring. An escalation with no threshold happens too late.
End with verification: the specific graph, query or endpoint that shows impact has ended, and what value counts as recovered.
Update it from incidents. The moment a runbook is found wrong during a real incident is the moment it is most cheaply fixed and most likely to be believed.
Test entries in rehearsals — game days, failover drills, restore drills — because untested runbooks are wrong at a rate nobody expects (Restore Drills).

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongOne tenant

One testEveryone

What contains it

Contained to the service, but the effect is on duration rather than scope: a missing runbook makes every incident on that service longer.

What can go wrong

Failure modes, including of the mitigation

A runbook written once at launch and never revisited, describing a service that has since been rearchitected.
Steps with no reasons, which produce ritual: the operator performs them in a situation where they do not apply, because nothing told them what the step was for.
A wall of prose with the mitigation buried in paragraph four.
Commands that no longer work — a renamed flag, a moved dashboard, a decommissioned host — discovered during an incident.
Every possible failure documented at equal weight, so the common ones are hard to find among the exotic ones.
A runbook that assumes access the on-call engineer does not have, discovered at the moment the command is refused (Break-Glass Access).
Escalation described as "contact the team" with no path, no threshold and no rotation name.

Misreads this invites

"A runbook is documentation." Documentation explains the system to a reader with time. A runbook is an operational tool for a reader without it. Different audience, different structure, different index.
"Every possible failure needs an entry." Cover what actually happens plus the catastrophic-but-plausible. Exhaustive runbooks are unnavigable, and navigability is the whole value.
"If it is automated we do not need the runbook." You need it more — the operator now meets the failure only when the automation did not handle it, which is the hardest version.
"Runbooks make on-call mechanical." They make the known parts fast so attention is available for the unknown parts, which is where it is needed.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked

An engineer who has never operated this service can resolve a common failure using only the runbook.
Alerts link to the runbook entry for their symptom, and the link works.
Runbook entries are edited after incidents — the edit history is the proof they are used.
A game day or drill exercised the runbook and produced corrections.
Time to mitigate for documented failure modes is shorter than for undocumented ones, and you can see that in the incident records.

How you get back

If a runbook step turns out to be wrong or harmful, correct it during the incident, in the document, not afterwards. The correction costs a minute then and is usually never made later.
If a runbook has grown to the point where nobody can navigate it under pressure, cut it back to the three or four symptoms that actually occur. Deleting stale entries is maintenance, not loss.

What to automate, and what stays human

Automate the mitigations that are unconditional and well understood — draining a bad instance, restarting a stuck consumer with a bounded retry, scaling out a queue consumer. If the answer is always the same regardless of context, a human reading it is toil (Toil).
Automate the checks into a single command or dashboard so the operator gathers state in one step instead of eight.
Do not automate mitigations that depend on judgement: whether to roll back or forward, whether to fail over a region, whether to shed load and from whom.
Do not automate a response you have seen once. A rule derived from one observation will fire on the second, different failure that shares its symptom (The Automation Trap).

What this costs

Runbooks cost continuous maintenance, and the maintenance is invisible until the day it was skipped. A wrong runbook is worse than no runbook, because it is trusted.
Detail helps a stranger and slows an expert. Structure — symptom first, mitigation early, background last — is what serves both, and it takes more effort to write than prose.
Automating a mitigation removes toil and removes the operator's exposure to how the system fails. After a year of automation, nobody on the rotation has seen the manual path.

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

GENERALThe six-part structure holds across platforms and organisations. What varies is the checks: the same "consumer lagging" entry is a kubectl command in one environment, a managed-service console panel in another, and an SSH session in a third.
ORG-SPECIFICWhere runbooks live and how formally they are reviewed is a convention. Keeping them in the service repository means they are reviewed with the code that invalidates them, which is the strongest available defence against decay — but only where the on-call engineer can reach the repository quickly.

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Backenddead-letter-queues job-idempotency connection-pools

Cloudinfrastructure-observability

Observability & Performancequeue-age queue-backlog dashboard-design

Domains that do not exist yet

— Testing & Reliability Engineering — game days as the mechanism that turns a runbook from a hypothesis into a tested procedure.