The question this answers
If one region serves everything and another waits, what exactly happens at the moment the first one fails?
The business will tolerate perhaps fifteen minutes of downtime and a few seconds of lost data during a regional failure, but not six hours. It will not fund two fully live regions, and the team is six engineers.
A second region that can take over on a defined signal, with a data-loss window equal to replication lag and a recovery time equal to detect + decide + promote + reroute.
The design: one serving, one warm
Active-passive keeps all traffic in one region while a second region holds a replica of the data and enough infrastructure to take over. How much infrastructure defines the variant: cold means little more than backups and templates, with hours of recovery; pilot light means the data tier replicates and the compute is scaled to near zero, with tens of minutes; warm standby means a small live fleet ready to scale, with minutes.
The attraction is that it avoids the entire class of problems active-active creates. There is exactly one writer, so there are no write conflicts, no conflict resolution, no ambiguity about ordering. The application does not need to be aware of regions. Most teams that say "multi-region" and mean "we would like to survive a region failure" want this, not Active-Active.
The price is paid in three places. Idle capacity billed continuously. A recovery-time budget that is real, not zero. And a failover path that only ever executes during a disaster — which is precisely the problem the next section is about.
The failover path is the part nobody tests
Every step below is a place a real failover has stalled. The pattern is always the same: the infrastructure was correct and something adjacent was not. The IAM role that promotes the database existed only in the primary region. The runbook lived in a wiki hosted in the region that was down. The standby's TLS certificate expired eight months ago and nobody noticed because nothing was serving on it. The database was promoted successfully and the application could not reach it, because its connection string was in a secret that had never been replicated.
The deeper problem is that this path executes zero times a year in most organizations. Every other part of the system is exercised constantly — deploys, scaling, restarts — and is therefore self-correcting when it breaks. The failover path is exercised only during a disaster, so it decays silently and its first execution in production is also its first execution ever, performed by stressed people at an awkward hour.
The remedy is unglamorous and it is the whole point of this lesson: schedule the failover. Fail over deliberately, on a calendar, during business hours, and then fail back. Quarterly is a reasonable rhythm. Each drill converts an unknown into a known and, more importantly, converts the standby from a diagram into a thing that has served real traffic. Regularly failing over also has a pleasant side effect: the standby stops being special, so it stops rotting.
- 1Detect2–10 min
Health checks and monitoring establish that the active region is not serving.
Ambiguity: is it the region, or our monitoring, or a partial failure? Usually the longest step.
- 2Decide1–15 min
A human or an automated policy declares failover. Someone has to have the authority.
Nobody is sure who may declare it, so the decision waits for a manager who is asleep.
- 3Promote the data tier1–5 min
The replica becomes the primary and begins accepting writes.
The promoting identity does not exist in the standby region. Writes not yet replicated are lost — this is your RPO.
- 4Scale the standby3–10 min
The standby fleet grows from minimal to full capacity.
Provisioning delay, cold caches, and a quota in the standby region that was never raised.
- 5Reroute traffic1–15 min
Global routing withdraws the failed region and directs clients to the standby.
DNS caching by clients and resolvers that ignore your TTL; a long tail keeps hitting the dead region.
- 6Verify5–20 min
Real requests succeed end to end: writes commit, jobs run, third parties accept calls from the new source addresses.
Outbound IP allow-lists at partners still list the old region. This one surprises people every time.
- 7Fail backhours to days
The original region returns; data written in the standby must be reconciled before switching back.
The most dangerous step. Failing back onto stale data destroys everything written during the incident.
Automatic or human? Both answers are defensible
Automatic failover minimizes recovery time and can trigger on a partial signal — a network partition between your monitoring and the active region looks identical to a regional failure, and an automatic system will happily fail over during a false positive, then fail over again when the partition heals. That oscillation is worse than the original problem, and it is how split brain happens.
Human-triggered failover avoids false positives and adds the time it takes to reach a person, obtain a decision and execute the steps. In practice most teams choose human-triggered with heavy automation behind it: one command or one pipeline that performs every step, invoked by a person who has decided. That combination gets the correctness of human judgement and most of the speed of automation.
Whichever you pick, write down the criteria in advance. "Fail over if the active region's error rate exceeds X for Y minutes and the standby's replication lag is under Z" is a decision made calmly. Deciding it during an incident produces long meetings while the service is down.
| Variant | What is running | Recovery time | Cost | Main risk |
|---|---|---|---|---|
| Backup only (cold) | Backups in another region; nothing else | Hours to a day | Lowest | Restore has never been tested (Restore Testing) |
| Pilot light | Data replicating; compute defined but scaled to zero | 30–60 min | Low | Scaling from zero during an incident, with quotas nobody raised |
| Warm standby | Data replicating; a small live fleet | 5–20 min | Medium | Drift between regions, and the fleet must still scale up |
| Hot standby | Full capacity running, taking no traffic | 1–5 min | High — near double | Paying full price for idle capacity; failover still untested if never drilled |
| Active-active | Both regions serving | Near zero | Highest | A different, harder problem entirely (Active-Active) |
Key points
- Active-passive keeps one writer, which removes the entire class of conflict problems that active-active creates.
- The failover path executes only during disasters, so it decays silently and its first real run is its first run ever.
- Schedule failover drills — quarterly, in business hours, with a fail-back — and the standby stops being a diagram.
- Recovery time is detect + decide + promote + scale + reroute + verify. Detection and decision are usually the largest terms.
- Replication lag at the moment of promotion is your data loss. Monitor it as a live RPO.
- Fail-back is more dangerous than fail-over, because data written in the standby must be reconciled first.
- Everything the standby needs must already exist there: secrets, certificates, IAM roles, quotas, partner allow-lists.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The data tier replicates asynchronously from the active region to the standby; lag is monitored and is the data-loss window.
- • The standby holds provisioned but minimal compute, plus its own copy of secrets, certificates and configuration.
- • Global routing sends all traffic to the active region and is capable of withdrawing it on a health signal or a manual action.
- • Failover promotes the replica to primary, scales the standby fleet, and switches routing.
- • Fail-back reverses the direction of replication after reconciling any writes accepted during the incident — which is why it is the hard direction.
- • Drill the failover on a schedule. This is the single highest-value operational practice in this lesson and the one most often skipped.
- • Keep the regions defined by the same code with a region parameter, so drift is structurally hard rather than merely discouraged (Infrastructure as Code).
- • Verify the standby's certificates, quotas, secrets and IAM roles on a schedule — they rot precisely because nothing uses them.
- • Write the failover criteria and the authority to invoke them before the incident, and keep them somewhere reachable when a region is down.
- • Track replication lag as an SLI with an alert, because it is the number that tells you what a failover would cost right now.
- • The standby cannot serve when promoted: a missing secret, an expired certificate, an IAM role that only ever existed in the primary.
- • Data loss at promotion equal to the replication lag, which under a write burst may be far larger than the steady-state number you remember.
- • Split brain: the original region is not actually dead, both accept writes, and reconciliation becomes a manual project.
- • Routing does not follow: cached DNS keeps a long tail of clients pointed at the dead region well past the TTL.
- • Capacity shortfall: the standby scales too slowly or hits a quota nobody raised, so failover completes into an overloaded region.
- • Third-party rejection: partners allow-list the active region's egress addresses, so outbound integrations fail from the standby.
- • Fail-back onto stale data, silently discarding everything written during the outage.
- • The standby must be able to reach full capacity, so its quotas, address space and instance availability need to be sized as if it were live.
- • Replication throughput must keep up with peak write volume, or lag grows exactly when the risk of needing failover is highest.
- • Recovery time grows with how much scaling the standby must do — a pilot light is cheap and slow, a hot standby is expensive and fast.
- • More regions in a passive arrangement multiply the drift surface without proportionally improving resilience.
- • The standby needs its own copies of secrets, keys and certificates, which means the same material exists in more places and must be protected identically (Secrets in Infrastructure).
- • Break-glass credentials used during failover are high-privilege and must be time-bound and audited (Audit Trails).
- • A standby that is never exercised is also never patched in anger — its images and dependencies age quietly, and it may be the least secure part of your estate.
- • Replication traffic crosses a regional boundary and must be encrypted in transit, with keys available on both sides (Key Management and Encryption at Rest).
- • You pay continuously for capacity that serves nothing; the variant you choose is a direct trade of money against recovery time.
- • Replication transfer is a continuous usage-shaped meter proportional to write volume.
- • The standby's storage bills at full price — data is the same size whether or not anyone reads it.
- • Drills cost engineering time and are the only reason the rest of the spend is worth anything.
- • Replication lag, alerted. It is the live RPO and the number to check before declaring a failover.
- • Standby readiness as a monitored fact: certificate validity, secret presence, image freshness, quota headroom.
- • Date of the last successful failover drill, treated as a first-class metric. If it is over a quarter old, the standby is unproven.
- • Time-to-failover measured during drills, broken into the lifecycle stages so you know which one to shorten.
- • The signal that lies: the standby region's infrastructure dashboard, which is green because nothing is asking it to do anything.
- • Backup and restore into a second region, with a documented multi-hour RTO. Dramatically cheaper, and honest — and it is the right answer whenever the business can tolerate hours (Disaster Recovery).
- • Multi-zone within one region. Covers the far more common failure class for a fraction of the cost and complexity (Multi-Zone Deployment).
- • A managed database with built-in cross-region failover, buying a tested promotion path instead of building one.
- • Active-active, when the recovery time of any passive design is genuinely unacceptable — and only with the organizational capacity to run it (Active-Active).
- • Accepting the regional risk explicitly. For many businesses, a rare multi-hour outage is cheaper than a standby plus the discipline to keep it working.
- • Buys regional survivability with a single writer; costs idle capacity, a real recovery-time budget and a failover path that must be rehearsed to be real.
- • A warmer standby shortens recovery and costs more, continuously.
- • Automatic failover is fast and can trigger on false positives; human-triggered is safe and slow.
- • Asynchronous replication keeps write latency local and guarantees a data-loss window.
- • Every drill costs a maintenance window and is the only thing that makes the design trustworthy.
What people believe, and what is true
We have a standby region, so we are covered.
You are covered if the failover path works. Until it has been executed, it is an untested assumption with a monthly bill.
Failover is the hard part.
Fail-back is harder. Data written in the standby must be reconciled before returning, and getting this wrong destroys the writes from the incident.
Automatic failover is strictly better.
It cannot distinguish a regional failure from a partition between it and the region, and failing over on a false positive can create split brain.