RTO and RPO
Two business objectives that only mean something when they are traced to an architecture, a runbook and a measured drill.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
How do recovery objectives stop being numbers in a document and start constraining the system?
Recovery objectives are usually agreed in a meeting and never connected to anything. The number exists, the architecture that would achieve it does not, and nobody discovers the gap until the event.
Write down an RTO and an RPO per service — an hour of downtime, fifteen minutes of data loss — get them signed off, and file the document.
Nothing in the system changed as a result. The backup interval, the replication topology and the failover procedure are whatever they were.
- Nothing in the system changed as a result. The backup interval, the replication topology and the failover procedure are whatever they were.
- The stated objective is compared against a hope rather than a measurement. Actual restore time is unknown until a drill measures it (Restore Drills).
- Objectives are set uniformly across services, so the cheapest system is over-engineered and the critical one is under-protected.
- Recovery *point* is confused with backup interval alone, ignoring replication lag, archive shipping delay and — the big one — how long it takes to notice the problem.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- RTO is how long the system may be unavailable: the interval from the start of impact to service restored. RPO is how much recent work may be lost: the interval between the last recoverable state and the moment of failure.
- They are not preferences. Each is a constraint that forces architecture. An RPO shorter than your backup interval is unachievable no matter how good the runbook is; an RTO shorter than the time to move the data is unachievable no matter how much you want it.
- Achievable RPO is bounded by the freshest usable recovery point: for periodic backups it is the interval; with continuous log archival it is the shipping delay; with synchronous replication it approaches zero for infrastructure loss and stays unbounded for logical corruption, because the corruption replicates too.
- Achievable RTO decomposes into detect, decide, execute and verify. Teams optimise "execute" and are undone by detect and decide, which are often the largest terms and are organisational rather than technical.
- The two objectives interact with cost monotonically: each step down in either costs more standing infrastructure, and the last increment before zero is the most expensive one.
- The objectives are per failure class. A system may fail over between regions quickly and still take far longer to recover from a dropped table, because those are different mechanisms (Disaster Recovery as an Operation).
From an objective to the thing that delivers it
This table is the whole lesson. An objective is only real if you can fill in the middle two columns, and only trustworthy if you can fill in the last one with a date.
| Objective | Mechanism that delivers it | Runbook step it implies | Evidence |
|---|---|---|---|
| Loss bounded to the backup interval | Periodic full backups, retention past detection lag | Choose the newest good copy; restore; verify | Drill record with per-phase timings |
| Loss bounded to log shipping delay | Continuous archival, monitored for chain gaps | Restore base copy; replay to chosen instant | A point-in-time drill, not only a full restore |
| Near-zero loss for infrastructure failure | Synchronous or low-lag replication with a promotable standby | Promote standby; repoint application; verify | A promotion rehearsal with lag observed at the time |
| Availability through region loss | Second region with data present and capacity ready | Shift traffic; confirm capacity, config, dependencies | A failover exercise (Region Failover) |
| Recovery from logical corruption | Retention past detection time; side restore and extraction | Freeze writes; restore aside; reconcile; backfill | A corruption drill on a copy (Partial and Logical Data Recovery) |
Where the recovery-time clock actually goes
Teams shorten the execute phase because it is the technical one, and are then surprised that recovery took much longer than the drill suggested. The drill usually starts at "we have decided to restore"; the incident starts when a user is affected.
The relationships hold even without numbers attached. Transfer scales with data volume; index rebuild scales with data volume and index count; detection scales with how good your alerting is; decision scales with how clearly authority was assigned.
- 1Detect
Something signals that a recovery-class event is happening.
fails by The first signal is a customer, or the alert exists but pages the wrong rotation.
evidence Time from first impact to first page, taken from the incident timeline (Reconstructing What Actually Happened).
- 2Decide
A named person declares and chooses the recovery path and the recovery point.
fails by Diffusion of responsibility; debate about whether this is really a disaster.
evidence A declaration timestamp and a named declarer in the incident record.
- 3Execute
Restore, promote or fail over, then rehydrate configuration and secrets.
fails by Missing permission, missing capacity, missing key, unexpected rebuild time.
evidence Per-phase timings from drills at production data volume.
- 4Verify
Confirm the application serves correct data before declaring recovery.
fails by Declaring recovery when the process is up, then discovering the data is wrong.
evidence The same application assertions the restore drill uses.
Detect and decide are organisational; execute and verify are technical. Teams invest almost entirely in the second half and are limited by the first.
Stated versus demonstrated
The difference between these two documents is the difference between a plan and a capability. Both are one page. Only one of them survives contact with an event.
Service: checkout RTO: 1 hour RPO: 15 minutes Approved by: leadership, last quarter
Service: checkout
RTO target 1h | measured, last drill: detect 0:07, decide 0:05,
execute 0:41, verify 0:09 -> 1:02, over target
RPO target 15m | mechanism: continuous WAL archival, chain
monitored; freshest recovery point alerted
when older than 15m
Gap: execute exceeds plan as data volume has grown; owner
assigned to evaluate a promotable standby
Logical corruption: NOT covered by the 15m figure; bounded
by retention and detection ([[data-recovery]])
Last exercised: dated, by an operator who did not write the runbookThe second version can be wrong and you would find out. It names a mechanism per objective, distinguishes failure classes, records a measurement rather than an intention, and turns the shortfall into work with an owner instead of a number that continues to look satisfied.
How to do it properly
Most important first.
- Set objectives per system and per failure class, in the language of business impact — what does an hour of this being down actually cost, and who feels it.
- For each objective, name the mechanism that delivers it and the drill that proves it. An objective with no mechanism is a wish; an objective with no drill is an untested claim.
- Define the clock explicitly: RTO starts at impact, not at declaration; it ends at verified service, not at "the instance came up".
- Measure the achieved figures during drills and during real incidents, and treat a gap between stated and measured as a defect with an owner.
- Include detection time in RPO. If corruption is typically noticed the next day, an hourly backup does not give an hourly recovery point for that class — retention has to cover the detection lag (Partial and Logical Data Recovery).
- Publish objectives where they change decisions: in the readiness review and the runbook, not in a strategy deck (Production Readiness Review).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
An unachievable objective contains nothing; it converts a technical event into an expectations failure on top of the outage.
What can go wrong
- Objectives are aspirational and the architecture cannot reach them, discovered during the event.
- RPO is stated against infrastructure loss only, and silently does not apply to the logical corruption case.
- Objectives never change after the system does — data grows tenfold, restore time grows with it, the number stays.
- Meeting the objective becomes a target to optimise rather than a signal, so effort goes into the measurement rather than the capability.
- "Our RPO is five minutes because we replicate synchronously." That is the RPO for infrastructure loss. For a bad DELETE it is however far back your backups let you go.
- "RTO is how long the restore command takes." It is detect plus decide plus execute plus verify, measured from user impact.
- "We agreed the objectives, so we have them." An objective is a claim about the system. Until a drill matches it, it is unverified.
- "Zero RPO is achievable with the right technology." Zero loss for infrastructure failure is approachable at high cost. Zero loss for a logical error is not a technology problem.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- For each system: stated RTO and RPO, the mechanism that provides each, and the date and result of the exercise that measured them.
- A recorded gap analysis where measured exceeds stated, with an owner rather than a footnote.
- Drill records that time detect, decide, execute and verify separately, so improvement can be aimed at the largest term.
- Relaxing an objective is a legitimate outcome — it is more honest than an unachievable number — but it is a business decision that has to be made explicitly with whoever depends on the system, not quietly by an engineer.
- Tightening an objective is a commitment to spend: on standing infrastructure, on drills, and on the operational complexity of a faster recovery shape.
- Automate measurement: instrument drills so phase timings are captured, not estimated afterwards.
- Keep human: setting the objectives, accepting the cost, and deciding which failure class matters most for a given system.
- Tighter objectives cost continuously and non-linearly; the last increment toward zero is the most expensive part of the curve (Cost Awareness).
- Per-system objectives are more accurate and harder to administer than one company-wide standard. The standard is cheaper and misallocates protection.
- Making objectives visible invites them to be treated as targets. State them as constraints with evidence, the same way readiness is reported without a score (The Readiness Scorecard).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- ORG-SPECIFICThe numbers themselves are a business decision, and in regulated sectors may be externally mandated with reporting obligations attached. Engineering owns the mapping from objective to mechanism and the honesty of the measurement, not the choice of number.
- SCALE-SPECIFICAt small data volumes, backup-and-restore can satisfy surprisingly tight objectives because transfer and rebuild are quick. The same architecture at large volume cannot, and the transition is gradual — which is why objectives need revisiting as data grows.
- DATABASE-SPECIFICAchievable RPO depends on what the engine offers: continuous log archival and synchronous or asynchronous replicas give very different floors, and a managed service may expose only some of them. Verify against your engine and tier rather than against the general claim.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — why synchronous replication trades write latency for recovery point, and what a partition does to that trade.