The question this answers
When the environment itself is gone — a region, an account, a deletion nobody meant to run — what is the documented path back, and how long does it take?
A regional outage, a ransomware event, a mistaken terraform destroy against production, or an account suspension must all have an answer that is better than improvisation. The answer must be written down, owned, and rehearsed.
A defined recovery path with a known time and a known data-loss window, so a catastrophic event becomes a long, unpleasant, survivable procedure rather than an open-ended crisis.
Disaster recovery is not high availability
High availability handles component failure inside a working environment: a host dies, a zone fails, the load balancer routes around it, and nobody is paged for very long. Disaster recovery handles the loss of the environment itself. They are different problems with different mechanisms, and one does not substitute for the other.
The distinction matters most for the disasters people leave out of the plan. A regional outage is the one everybody names. The more likely ones are self-inflicted or malicious: a deletion executed against the wrong environment, a migration that corrupts a table and replicates the corruption to every replica within a second, ransomware that encrypts everything the compromised credential could reach, an account suspended over a billing dispute. Replication does not protect against any of these — it faithfully copies the damage. Only a backup in a separate failure domain, with separate credentials, does (Backup Strategy).
So the DR plan has to answer for a wider set of events than the HA design does, and it has to answer with a procedure rather than a mechanism. Its output is not a topology; it is a document with an owner, a rehearsal date and two numbers (RPO & RTO).
The four mechanisms, and which disaster each one answers
A DR plan is assembled from four mechanisms, and confusing them is the most common planning error. Backup creates an independent point-in-time copy — the only mechanism that survives deletion, corruption and compromise. Restore turns a backup into a running system, and is the step whose duration nobody knows. Failover switches to already-running infrastructure elsewhere, which is fast and only helps for infrastructure failures. Rebuild re-creates the environment from infrastructure code, which is what you need when the environment itself is gone rather than merely unhealthy.
Map them against the disasters and the gaps become obvious. Regional outage: failover, or restore into another region. Data corruption: restore from a point in time before the corruption; failover is useless because the replica is corrupt too. Ransomware or credential compromise: restore from an immutable backup the compromised credential could not reach, and rebuild the environment because you cannot trust the old one. Account loss: rebuild from code plus restore from a backup held outside the account.
Two components are almost always missing from the plan and always needed. The infrastructure definitions must live outside the environment they define (Infrastructure as Code) — a terraform state file stored only in the account you lost is not a recovery plan. And the runbook must be reachable when your systems are not: a wiki hosted in the region that is down is a document you cannot read on the day you need it.
| Disaster | Failover helps? | Replica helps? | What actually recovers you | Typical RTO |
|---|---|---|---|---|
| Zone failure | Yes — automatic | Yes | Multi-zone HA. Not a DR event at all (High Availability) | Seconds |
| Regional outage | Yes, if a standby exists | Yes | Cross-region failover, or restore into another region | Minutes to hours |
| Data corruption / bad migration | No | No — the corruption replicated | Point-in-time restore from before the event | Hours |
| Accidental deletion | No | No — the delete replicated | Restore from backup; soft-delete and versioning shorten it dramatically | Minutes to hours |
| Ransomware / credential compromise | No | No — the attacker had the same reach | Immutable backup in a separate account + full rebuild | Days |
| Account suspension or loss | No | No | Rebuild from IaC held elsewhere + restore from an external backup | Days |
The declaration, and the sequence
The step people forget to design is the first one: deciding that this is a disaster. It requires a named authority, stated criteria and a communication path, all agreed in advance. Without them the first forty minutes are spent establishing who may make the call — which is forty minutes added directly to your recovery time, every time.
The sequence after that is mechanical but full of ordering constraints. Restore the data before starting the application, or the application connects to an empty database and starts writing. Verify before cutting over, because a restore that completed is not the same as a restore that is correct. Communicate throughout, because the perception of the outage is shaped by what customers were told during it, not afterwards.
And plan the return. A DR environment is usually a reduced one: less capacity, fewer background jobs, older configuration. Living there is a temporary state, and going back involves reconciling everything written while in DR — the same fail-back hazard as Active-Passive Failover, for the same reason.
- 1Detectminutes to days
Monitoring or a human establishes that something beyond normal failure has happened.
Corruption and compromise can go undetected for hours or days, which quietly extends the required backup retention.
- 2Declare5–60 min
A named authority declares a disaster against stated criteria and notifies stakeholders.
No named authority means the first long delay is purely organizational.
- 3Assess and choose a recovery point15–90 min
Determine what was lost and pick the restore point — before the corruption, after the last good state.
Choosing badly restores the corruption. This step needs data, not guesses.
- 4Prepare the target30 min – hours
Stand up or confirm the environment to recover into, from infrastructure code.
IaC that only ever ran against the existing environment fails on a clean one — an untested code path.
- 5Restore the datahours — scales with data size
Backup data is written into the recovered data tier.
The step whose duration nobody knows, because nobody has timed it at full size (Restore Testing).
- 6Verify30–120 min
Check integrity, row counts, recent records, referential consistency, and run application smoke tests.
Skipping verification is how a corrupt restore becomes the new production.
- 7Cut over15–60 min
Point traffic at the recovered environment and resume service, possibly degraded.
Third-party allow-lists, DNS caching and webhook endpoints all still reference the old environment.
- 8Return to normaldays
Rebuild the primary, reconcile data written during DR, and fail back deliberately.
The most dangerous step. Failing back onto pre-disaster state discards everything from the incident.
Key points
- DR handles loss of the environment; HA handles failure inside a working one. Neither substitutes for the other.
- Replication copies deletion, corruption and ransomware faithfully. Only an independent backup survives them.
- Four mechanisms — backup, restore, failover, rebuild — and each answers a different disaster.
- Infrastructure definitions and the runbook must live outside the environment they recover, or you cannot read them on the day.
- Declaring the disaster is a designed step with a named authority; without one it is the first long delay.
- Restore duration scales with data size and is the number nobody has measured.
- Returning to normal is harder than leaving it, because data written during DR must be reconciled.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Backups are taken on a schedule and stored in a separate failure domain with independent credentials.
- • Replication maintains a warm copy for infrastructure failures; it is explicitly not a backup.
- • Infrastructure code, stored outside the environment, can re-create the environment from nothing.
- • A restore reads backup data into a target data tier — the duration is a function of data volume and restore throughput.
- • Verification confirms the restored state is correct before any traffic is pointed at it.
- • Cutover redirects traffic; fail-back later reverses it after reconciliation.
- • Maintain a DR runbook with an owner and a rehearsal date, stored somewhere reachable when your infrastructure is not.
- • Define the declaration authority and criteria in advance, and make sure the people named know they are named.
- • Rehearse at least annually — full restore, full rebuild, timed. Quarterly for anything with a tight RTO (Restore Testing).
- • Keep IaC applicable to an empty environment: test it by building a clean one, because that is what a disaster asks it to do.
- • Track the inventory of what must be recovered: databases, object storage, secrets, certificates, DNS, third-party configuration. The list is always longer than the database.
- • The backup exists and cannot be restored — wrong format, missing key, incomplete, or nobody has ever tried.
- • The runbook is unreachable because it lives in the environment that is down.
- • IaC fails against an empty account because it was only ever applied incrementally to an existing one.
- • The chosen restore point still contains the corruption, because assessment was rushed.
- • Recovery succeeds and integrations fail: partner allow-lists, webhook URLs and OAuth redirect URIs all still point at the old environment.
- • Fail-back discards everything written during the disaster.
- • The compromised credential could also delete the backups, so there is nothing to restore from at all.
- • Restore time grows with data volume and is frequently the binding constraint on RTO — parallel or incremental restore is the only real lever.
- • More services means a longer recovery inventory; the coordination cost grows faster than the number of components.
- • Retention requirements grow storage cost linearly and are set by how long a corruption might go undetected, not by convenience.
- • A large estate makes full rebuild impractical, which forces prioritization: tiered recovery, most critical services first.
- • Backups must be outside the blast radius of a production compromise: separate account, separate credentials, immutable retention (Backup Strategy).
- • Backups contain everything sensitive that production contains. Encrypt them, and manage the keys so that they are available during a disaster but not to an attacker (Key Management and Encryption at Rest).
- • Break-glass access used during recovery must be time-bound, pre-provisioned and audited (Audit Trails).
- • A rebuilt environment must not inherit the compromise: rotate every credential, and rebuild from source rather than from an image that may itself be the entry point (The Infrastructure Supply Chain).
- • Backup storage is a continuous usage-shaped cost driven by data size, frequency and retention.
- • Standby infrastructure, if any, is fixed and idle — the trade between money and recovery time.
- • Rehearsals cost engineering time and temporary infrastructure, and are the only reason to believe any of the rest works.
- • The comparison worth making explicit: cost of the DR posture per year versus cost per hour of outage × expected hours. That is the conversation the business can actually have.
- • Backup success, age and size — with an alert on a backup that succeeded but shrank, which is how a silently broken backup presents.
- • Restore rehearsal date and measured duration, treated as a monitored fact rather than a memory.
- • Replication lag, which is the RPO for the failover path specifically.
- • Completeness of the recovery inventory, reviewed whenever a new stateful component is added.
- • The signal that lies: a green backup job. It reports that a job ran, not that the data inside it is restorable.
- • Backup and restore only, with an honest multi-hour or multi-day RTO. Cheapest and, for most systems, entirely adequate — provided the restore is actually tested.
- • Provider-managed point-in-time recovery, which covers corruption and deletion within its retention window without any infrastructure of your own. Usually the best first step.
- • Rebuild-from-code with no standby: if the infrastructure is fully described in code and the data restores in an hour, you may not need a standby environment at all.
- • Accepting the loss for a class of data. Derived data, caches and analytics that can be recomputed do not need a recovery plan — say so explicitly so the plan stays small enough to rehearse.
- • A shorter RTO always costs more, either as standing infrastructure or as engineering effort in automation.
- • Longer retention protects against slowly-discovered corruption and costs storage linearly.
- • Immutable backups protect against a compromised administrator and remove your ability to delete data on request — a real tension with data-deletion obligations.
- • A thorough plan covering every service is harder to keep current than a focused one covering the critical path; an out-of-date plan is worse than a short one.
What people believe, and what is true
We replicate to another region, so we have disaster recovery.
Replication covers infrastructure failure. It copies deletions, corruption and ransomware faithfully and instantly.
The managed database has automatic backups, so DR is handled.
Automatic backups cover one component within one retention window inside one account. They do not cover object storage, secrets, DNS, third-party configuration or the account itself.
DR is about regional outages.
Regional outages are the rarest item on the list. Deletion, corruption and compromise are far more likely and are exactly what replication does not help with.