The question this answers
How do you know your backups work — and what is the only answer that is not a guess?
The recovery plan claims an RTO of four hours and an RPO of fifteen minutes. Someone has to be able to say, with evidence, that both are true. A configuration screenshot is not evidence.
Evidence. A measured restore duration, a verified data set, a list of the things that were broken and are now fixed, and a team that has performed the procedure before the day it matters.
A backup you have never restored is not proven recovery
Say it in reviews, put it in the runbook, use it to end arguments. A backup job that exits zero proves that a process ran. It does not prove the archive is complete, that it is readable, that the encryption key still exists, that the schema matches what the current application expects, that the restore finishes inside your RTO, or that anyone knows how to start it. Every one of those has failed in production somewhere, most of them repeatedly.
The reason this is not obvious is that backups are the only part of a system whose correctness is never exercised by normal operation. Deploys are exercised constantly. Scaling is exercised constantly. Health checks run every few seconds. Backups are written and never read, which means every defect in the path is silent by construction. A backup pipeline can be broken for eight months and every dashboard stays green the entire time.
The list below is not hypothetical. Each item is a common way a first restore fails, and each is trivially discoverable by a drill and undiscoverable any other way.
| Failure found by drilling | How it looked before the drill | Why only a restore reveals it |
|---|---|---|
| Archive is truncated or corrupt | Job succeeded; file size looked plausible | Nothing reads the file until a restore does |
| Encryption key is gone or inaccessible | Backups encrypted, policy documented | The key path is only exercised on decrypt |
| Schema mismatch with current application | Backup is current, application is current | The two are only combined during a restore |
| Restore takes 9 hours, documented RTO is 4 | RTO written in the plan | Restore duration is never measured otherwise, and it grows with data |
| Nobody knows how to start a restore | Runbook exists | A runbook is a hypothesis until someone follows it |
| Object storage, secrets and DNS were never backed up | Database backups green | Only a full recovery exposes what is missing from the inventory |
| Restore requires a permission nobody has | IAM policies reviewed and approved | The restore action is never invoked in normal operation |
| Application will not start against the restored data | Data restored, row counts correct | Only an end-to-end drill runs the application against it |
The drill, as an actual procedure
A drill is not "check that the backup file exists". It is a rehearsal of the recovery, into an isolated environment, timed, with verification, performed by someone who is not the person who built the backup system. That last constraint matters more than it sounds: if only the author can run it, you have a bus factor of one on the recovery path.
Isolation is essential. The restore target must be a separate environment with no ability to write to production — a restore that accidentally overwrites the live database during a drill has happened, and it is a memorably bad day. Give the drill its own account or project, its own network, and read-only access to the backup store.
Verification has to go beyond row counts. Check that the newest record present is as recent as your RPO claims. Check referential integrity across a few relationships. Start the actual application against the restored data and run a smoke test that exercises a real workflow. And record the wall-clock time of each stage — the measured RTO is the deliverable, and if it exceeds the documented one, the documented one was fiction and now you know.
- 1Pick a scenario10 min
Choose a specific disaster: "the primary database was dropped at 14:00 yesterday" or "the account is gone".
A vague scenario produces a vague drill. Specificity is what makes it useful.
- 2Start the clock—
Begin timing at the simulated moment of the disaster, not at the moment you type the restore command.
Timing only the restore hides the detection and decision time that dominates real RTO.
- 3Stand up an isolated target20–60 min
Create the recovery environment from infrastructure code, with no write path to production.
IaC that has only ever run incrementally often fails against an empty environment. Finding that here is the point.
- 4Restoremeasure it
Read the backup into the target, using only credentials that would be available during the real disaster.
Missing keys, missing permissions, retrieval delays from cold storage.
- 5Verify30–60 min
Row counts, newest record timestamp against the RPO claim, referential integrity, then start the application and run a real workflow.
A restore that completes is not a restore that is correct. Skipping this makes the drill worthless.
- 6Stop the clock and record10 min
Record measured RTO and measured RPO next to the documented targets.
Not writing it down means the next drill starts from opinion again.
- 7File and fix what brokedays
Every drill finds something. Turn each finding into a ticket with an owner and a due date.
Findings that are noted and not fixed make the next drill find the same things.
- 8Tear down15 min
Destroy the drill environment, confirming it left no production side effects and no lingering copy of production data.
An abandoned restore environment full of real customer data is a data-protection incident waiting to be found.
Making it happen, given that it never happens
Restore drills are universally agreed to be important and are routinely not done, because they are never the most urgent thing in a sprint. The only reliable fixes are structural: put the drill on a calendar with a named owner, make its outcome a metric someone reports, and reduce its cost until it is boring.
Automation is what makes the cadence sustainable. A pipeline that restores the latest backup into an ephemeral environment, runs an integrity check and a smoke test, records the duration and tears everything down turns a quarterly all-hands exercise into a nightly job. Once that exists, "when did we last verify we can restore?" is answered by a dashboard rather than by a search through chat history.
Keep the manual drill as well, at a lower frequency. Automation verifies the archive and the mechanism; it does not verify that a human can navigate the procedure under pressure, that the runbook is readable, or that the person on call has the permissions they need at two in the morning. Run the automated check nightly and the human drill quarterly, and gradually widen the scenario: database only, then database plus object storage, then a full environment rebuild from nothing.
One last practice worth adopting: restore into a *fresh* environment, never into the existing one. Restoring in place tests the happy path where the environment still exists, which is not the disaster you are preparing for.
RESTORE DRILL RECORD
---------------------------------------------------------------
date 2026-08-14 scenario primary DB lost
run by a.mueller (did not build the backup system)
target isolated account, built from IaC, no prod path
documented RTO 4h 00m measured 6h 12m FAIL
documented RPO 15m measured 11m PASS
stage timings
build target env 52m IaC failed twice on empty account
restore data 3h 41m cold-tier retrieval added ~40m
verify 1h 09m app start blocked by missing secret
------------------------------
total 6h 12m
findings
1 IaC assumes a pre-existing VPC -> CLOUD-4412 a.mueller
2 backups in cold tier; retrieval unbudgeted -> CLOUD-4413 ops
3 app secret not in backup inventory -> CLOUD-4414 platform
4 restore runbook missing the key-access step -> CLOUD-4415 a.mueller
verdict RTO target is not currently achievable. Either fix 1-3
or change the documented RTO to 7h. Do not leave it at 4h.
The drill cost half a day. The alternative was discovering all four
findings during an outage, in sequence, at night.Key points
- A backup you have never restored is not proven recovery. Say it out loud, in reviews.
- Backups are the only part of the system never exercised by normal operation, so every defect in the path is silent.
- A drill restores into an isolated environment, is timed from the simulated disaster, and ends with an application actually running.
- Have someone who did not build the backup system run it; otherwise the recovery path has a bus factor of one.
- Measured RTO is the deliverable. If it exceeds the documented one, the documented one was fiction.
- Automate a nightly restore check for the mechanism; keep a quarterly human drill for the procedure and the people.
- Restore into a fresh environment, not the existing one — the disaster you are preparing for is the one where the environment is gone.
- Every drill finds something. A drill that finds nothing was probably not thorough enough.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A scenario is chosen and the clock starts at the simulated moment of failure, not at the restore command.
- • An isolated target environment is created from infrastructure code, with no write path to production.
- • The backup is restored using only the credentials and keys that would be available during the real disaster.
- • Verification runs at three levels: structural (row counts, integrity), temporal (newest record versus the RPO claim), and functional (the application runs a real workflow).
- • Durations and findings are recorded; findings become tickets with owners.
- • The environment is destroyed, confirming no production side effects and no lingering copy of customer data.
- • Schedule drills and give them an owner. Unscheduled drills do not happen, in any organization, ever.
- • Automate a nightly restore-and-verify into an ephemeral environment so the mechanism is continuously proven.
- • Rotate who runs the manual drill, and require them to follow the runbook rather than their own knowledge.
- • Track measured RTO over time; it degrades silently as data grows, and the trend is the early warning.
- • Fix findings with due dates. A drill whose findings are not fixed converts into a ritual that proves nothing.
- • Widen the scenario over time: database, then storage and secrets, then a full environment rebuild from an empty account.
- • The drill is never scheduled, so the first restore in the organization's history happens during a disaster.
- • The drill only tests the happy path — restore in place, into an environment that still exists.
- • Verification stops at row counts, so a restore that is complete but functionally broken passes.
- • The drill environment is not isolated and a restore touches production.
- • Findings are recorded and never fixed, so the same four issues appear in every drill record.
- • Only the backup system's author can run it, so the capability disappears when they are on holiday or leave.
- • The drill environment is left running, full of production data, and becomes a data-protection finding of its own.
- • Restore duration grows with data volume, so a drill cadence must be frequent enough to catch the moment the RTO becomes unachievable.
- • Large data sets make full drills expensive; sample or partial restores can run nightly with a full drill less often.
- • More services means the drill must widen beyond the database, and coordinating a multi-service drill is itself a skill worth rehearsing.
- • Automated drills scale well and prove the mechanism; human drills scale poorly and prove the procedure. You need both.
- • The restore environment holds a full copy of production data with, typically, weaker controls. Give it production-grade access control and destroy it afterwards (Backups Are Sensitive Data Copies).
- • Drills must exercise the real credential path, including key access, or they validate a path that will not exist during the disaster.
- • A drill is an excellent test of break-glass access: does the on-call engineer actually have what they need, and is the use audited (Audit Trails)?
- • Restoring production data into a lower environment for convenience is a data-protection incident, not a shortcut. Isolate properly or use masked data for functional checks.
- • Drill infrastructure is temporary and cheap; the real cost is engineering hours, half a day at a time.
- • Cold-storage retrieval charges and delays appear during a drill — which is exactly where you want to discover them.
- • Automated nightly drills add a small continuous cost and remove the largest unknown in the recovery plan.
- • Compare against the cost of discovering these findings sequentially, at night, during an outage. The drill is the cheapest insurance in this domain.
- • Date of the last successful drill, per data set, on a dashboard. If it is not visible, it will not happen.
- • Measured RTO and RPO from each drill, plotted next to the documented targets.
- • Restore duration trend, which is the leading indicator that data growth has invalidated your RTO.
- • Open findings from previous drills and their age, which is a direct measure of whether drills are changing anything.
- • Automated nightly verification results: archive readable, integrity check passed, smoke test green.
- • The signal that lies: backup job success. It is the metric that stays green through every failure this lesson describes.
- • Automated verification only — restore into an ephemeral environment nightly with no human involved. Cheaper and continuous; it proves the archive and the mechanism, not the people or the runbook.
- • Partial restores of a representative subset, when full restores are prohibitively large. Proves readability and integrity, and does not measure a true RTO.
- • Provider-verified backups, where the platform validates its own snapshots. Better than nothing and it validates the platform's half only (Shared Responsibility).
- • For a small system, one thorough annual drill plus nightly automated verification is a proportionate programme. The point is that the number is not zero.
- • Drills cost engineering time and are the only way to convert a recovery plan into a capability.
- • Automated drills are cheap and continuous and do not test whether a human can follow the procedure under pressure.
- • Realistic drills — full rebuild, real credentials, no shortcuts — find the most and cost the most.
- • Restoring real production data into a drill environment is the most realistic test and creates a real data-protection obligation.
What people believe, and what is true
Our backups are verified — the job checks the checksum.
A checksum proves the bytes survived transfer. It says nothing about schema compatibility, key availability, restore duration or whether the application runs against the result.
We restored a table last month, so restores work.
Restoring one table into a working environment is not the disaster. The disaster is a fresh environment, missing dependencies, and a stopwatch.
We will work it out during the incident.
Under pressure, at night, with a missing key and an unfamiliar runbook, you will work out something. It will take multiples of your documented RTO.