On-Call Is Production Ownership
Someone has to be reachable when production breaks. Done well it is the shortest feedback loop a team has; done badly it is the fastest way to lose people.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why does someone need to be reachable at 3am, and what is that person actually expected to do?
Production fails outside working hours, and a failure nobody responds to lasts until morning — but the arrangement that fixes that puts a real cost on people's lives.
Put the operations team on call. They run production, so they should be the ones woken when it breaks.
The people woken cannot fix most of what wakes them. They can restart things and escalate, which means the actual fix waits for the team that wrote it anyway.
- The people woken cannot fix most of what wakes them. They can restart things and escalate, which means the actual fix waits for the team that wrote it anyway.
- The feedback loop is severed: the engineers whose decisions produced the pages never experience them, so nothing about the code changes (What DevOps Actually Means).
- Operational quality has no advocate. Nobody in the room when the design is chosen is the person who will be woken by it.
- The rotation becomes a hardship posting, which selects for who will tolerate it rather than for who can respond well.
- Escalation is the normal path rather than the exception, which adds a hop to every incident.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- On-call exists because production has users at times when nobody is at a desk. That is the whole justification, and it is enough.
- The person on call is not there to fix everything. They are there to detect, triage, mitigate, communicate, escalate and recover — six things, and only two of them require deep knowledge of the failing component (Stop the Harm Before You Understand It).
- That framing is what makes it teachable. A responder needs to know how to establish scope, how to reach the mitigation levers, how to communicate, and how to get help. Deep expertise in every service is neither achievable nor required.
- It is production ownership, not punishment. The team that builds a service is the team best placed to operate it, and being on call for your own work is the feedback loop that makes operability a design consideration rather than an afterthought (Shared Ownership).
- That only holds if ownership is real. Being paged for a system you cannot change, or having no authority to fix what wakes you, is not ownership — it is being the alarm for someone else's system, and it is the arrangement that makes on-call feel punitive.
- The rotation is part of the production system. Its size, its load and its supporting material determine response quality as directly as the monitoring does (Rotations People Can Sustain).
The six things a responder does
This is the job description, and it is deliberately modest. Only two of the six require knowing the failing component well, which is what makes it possible for a team to cover services no single person has memorised.
The evidence column is what a shift should be able to demonstrate afterwards — and what a runbook and a dashboard should make possible.
- 1Detect
Receive and acknowledge the page; confirm it is real.
fails by Alert noise makes acknowledgement automatic and unread (Alert Fatigue).
evidence Acknowledged within the stated target, with the alert actually read.
- 2Triage
Establish scope and severity: what, for whom, since when.
fails by The dashboard cannot answer scope, so it is guessed (Severity: What It Should Reflect).
evidence A written impact statement within minutes.
- 3Mitigate
Apply a known lever to stop user impact.
fails by No lever exists, or the responder lacks access to pull it (Production Access).
evidence The user-facing signal turns, attributable to the action.
- 4Communicate
Tell the people who need to know, on a cadence.
fails by The responder becomes the channel and stops responding (Telling People What Is Happening).
evidence Stakeholders stopped asking directly.
- 5Escalate
Bring in the people who can go further.
fails by Treated as an admission, so it happens two hours late.
evidence Escalation happened on a clock rather than at a breaking point.
- 6Recover
Return to normal — backlogs, reconciliation, capacity — and confirm it.
fails by Stopping when the error rate drops (Reconstructing What Actually Happened).
evidence Confirmed from a user-facing signal, not an internal one.
Understanding the root cause is not on this list. It happens afterwards, in daylight, with the people who know the component (Stop the Harm Before You Understand It).
Who carries the pager
There is no universally right arrangement here, and the wrong ones are usually chosen for reasons that made sense at the time. What matters is being honest about what each shape costs.
The one constant across all of them: whoever is paged must be able to act. A rotation without authority is the arrangement that turns on-call into something people leave over.
Who should be reachable when this service fails at night?
when The team can deploy, roll back and change the service themselves.
cost Operational load on people who also have delivery work; needs to be planned into capacity, not absorbed quietly.
when Individual teams are too small to sustain their own, and services are similar enough to be operable from runbooks.
cost Responders operate systems they did not build; weakens the feedback loop and demands much better runbooks (Runbooks).
when Regulatory separation of duties, or 24/7 coverage that engineering teams cannot staff.
cost Severs feedback from author to consequence; needs a deliberate replacement path so operability still gets designed in (What DevOps Actually Means).
when You genuinely have engineers in enough time zones.
cost Nobody is woken, and every handover is a place state gets lost; the handover discipline becomes critical.
when The service has no meaningful overnight usage and nothing degrades unattended.
cost An honest, legitimate choice — as long as it is a choice, with the failure mode understood, rather than an omission.
What a shift needs before it starts
Most bad on-call experiences are not caused by the incident. They are caused by something missing that should have been there before the shift began, and every item below is cheap to provide and expensive to lack at 3am.
These are a readiness checklist for the rotation itself, in the same sense that a service has one (Production Readiness Review).
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Alert with no runbook | Responder reads the alert and does not know what it means | The alert was added without the material to act on it | No runbook, no page — write it first (An Alert Should Demand Action) |
| Access expired or never granted | Ten minutes lost requesting permission | Time-bound access with no on-call path | Grant standing access to the rotation, or a fast break-glass route (Break-Glass Access) |
| Runbook is a list of commands | The situation differs slightly and the script does not apply | It encodes steps rather than understanding | Write what each step is for and how to tell if it worked (Runbook Anti-Patterns) |
| Alert routes to a team that does not own the service | Responder can only escalate | Ownership records are stale | Fix ownership; a page nobody can act on is not a page (The Ownership Record) |
| Deploy system is behind the outage | Cannot roll back the thing that broke | Response tooling shares a failure domain with production | Keep an out-of-band path and test it (Disaster Recovery as an Operation) |
| No handover from the previous shift | Responder is surprised by a known degradation | Handover is informal and gets skipped | A written handover as part of the rotation, not an optional courtesy |
How to do it properly
Most important first.
- Put the team that owns the service on its rotation, with the authority to change the service — ownership and pager together, never one without the other (The Ownership Record).
- Define what is expected in writing: acknowledge within a stated time, establish scope, mitigate if a known lever applies, communicate, escalate without hesitation if not.
- Make escalation explicitly encouraged and time-bound. "Escalate if you have no working hypothesis within fifteen minutes" removes the hardest judgement from the worst moment, and it removes the implication that asking for help is a failure.
- Give the responder what they need before the shift: runbooks that encode understanding rather than command lists, dashboards linked from alerts, and the access to act (Runbooks, Production Access).
- Onboard people onto the rotation deliberately — shadowing first, then primary with a named backup. Nobody's first shift should be alone.
- Hand over between shifts with the state that matters: ongoing issues, recent changes, anything degraded and being watched.
- Treat what happens on the shift as data about the system. Pages that required no action, missing runbooks and unclear alerts are defects to fix, not the price of the rotation (Alert Fatigue).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
The rotation is the containment: without a responder, every failure lasts until someone notices in the morning.
What can go wrong
- On-call without authority: paged for a service you cannot deploy, change or roll back.
- A rotation of one, or effectively one because only one person can actually handle the pages.
- Runbooks that are lists of commands with no explanation, which fail the moment reality differs from the script (Runbook Anti-Patterns).
- Access that has expired or was never granted, discovered at 3am (Break-Glass Access).
- The response tooling depending on the thing that is down.
- Escalation treated as a personal failure, so people struggle alone for hours.
- Handover skipped, so the incoming responder inherits a degraded system they do not know about.
- "On-call means you must fix anything that breaks." It means you detect, triage, mitigate, communicate and escalate. Mitigation rarely requires knowing the cause.
- "On-call is what junior engineers do until they are senior." It is production ownership. A team where seniority exempts you from the pager has an operability problem it cannot see.
- "Being paged is a punishment for shipping bugs." Being paged is how a team learns what production is actually like. Treating it as punishment is how a team stops reporting problems.
- "We have too few incidents to justify a rotation." The rotation is for the incident you have not had yet; the question is what happens at 3am on the night it arrives.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A responder who did not build the failing service was able to establish scope and mitigate using the runbook and dashboards alone.
- Escalation happened recently and was treated as routine.
- Every page in the last month is attributable to a service someone on that rotation owns and can change.
- New joiners have shadowed before carrying the pager alone.
- A shift can be handed over mid-incident when someone is exhausted or out of depth. This should be normal and low-friction; a rotation where handing over feels like giving up will produce worse decisions at hour six than at hour one.
- If a service is generating pages the rotation cannot act on, take it off the rotation and fix the ownership question rather than continuing to page people who cannot help (The Ownership Record).
- Automate scheduling, paging, escalation timers and override handling — the mechanics should be invisible.
- Automate the context delivered with the page: service, owner, dashboard, runbook, recent changes (Deploys on the Same Timeline as the Symptom).
- Automate the safe, well-understood remediations that would otherwise be a page — clearing a known-transient condition, restarting a process on a specific and unambiguous signal — while recording that it happened, so the underlying defect stays visible (How to Automate Something).
- Do not automate the judgement calls: whether to declare, whether to mitigate, whether to escalate, whether to wake someone else.
- Team-owned rotations are the shortest feedback loop and they distribute operational load onto people who also have delivery work. That cost is real and should be planned for, not absorbed silently.
- A specialist rotation concentrates skill and severs the feedback loop; a team rotation preserves it and spreads the load thinner.
- Follow-the-sun removes night pages and requires enough people in enough time zones, plus handovers that actually transfer state.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICRotation length, compensation, whether nights are covered at all, escalation policy, and whether on-call is voluntary are organisational and often contractual choices, and they differ by jurisdiction. What generalises is that someone must be reachable and empowered; how that is arranged and paid for does not.
- SCALE-SPECIFICA service used only in one country's business hours may need no night coverage at all, and adding it is cost without benefit. A team of four cannot sustain a 24/7 rotation without the shift frequency becoming harmful, which is a staffing question rather than a scheduling one (Rotations People Can Sustain).
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — game days as the way a rotation practises before the night it matters.