CapacityGENERALCLOUD-SPECIFICSIMULATED
Capacity During Failover
If two regions each serve half the traffic, either one must be able to serve all of it — and most teams find that out during the failover.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
The production question
When a failure domain goes away, does the capacity that remains actually fit the traffic that remains?
The problem
Redundancy is designed for availability and reviewed for correctness, and the capacity question — can the survivor carry everything — is quietly assumed to have been answered by someone else.
What teams do first
We are active-active across two regions, each carrying half the traffic. If one fails, the other takes over. That is what active-active means, and the failover has been tested.
How it breaks
A failover test that shifts traffic during a quiet period proves routing works. It does not prove the survivor can carry peak traffic, which is the only load that matters.
How it breaks in production
- A failover test that shifts traffic during a quiet period proves routing works. It does not prove the survivor can carry peak traffic, which is the only load that matters.
- If each region normally serves half, the survivor must serve all of it — so genuine two-region active-active means each region is provisioned at roughly twice its steady-state load. That is arithmetic on the stated split, and it is the sentence teams find surprising.
- The survivor was already running at whatever utilisation you considered efficient. Doubling its load takes it past its limit immediately, before any scaling has begun (Headroom).
- Scaling the survivor takes minutes, and the traffic arrives in seconds. Meanwhile every failed request from the lost region is being retried into the survivor (How Autoscaling Fails).
- Capacity is not only compute. The survivor's database now takes all writes, its caches are cold for the shifted traffic, its connection pools must double, and its per-region provider quotas were sized for half the load.
- The failover itself is the load event. Reconnections, session re-establishment and cache fills all land at once, so the peak during a failover exceeds the steady-state traffic being moved.
Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Split traffic evenly across N failure domains and lose one, and each survivor carries 1/(N-1) of the total instead of 1/N. The fewer the domains, the more brutal the step: with two, the survivor's share doubles.
- The provisioning requirement follows directly. To survive the loss of one domain at peak, each domain must be able to serve total peak divided by (N-1), not total peak divided by N.
- This is why the cost of two-region active-active is not "the same fleet, spread out". It is closer to two fleets, each sized for everything — which is the thing that gets negotiated away in planning and rediscovered in an incident.
- More, smaller domains are cheaper per unit of resilience. Spreading across many zones means losing one costs a small fractional increase; spreading across two regions means losing one doubles the survivor's load.
- Shared and regional limits do not move with the traffic. Per-region quotas, per-account limits and database maximum connections are configured per domain and were sized for that domain's normal share.
- Warm state does not transfer. The survivor's caches hold its own working set, so the shifted traffic misses, and cache misses land on a database that is simultaneously taking twice the writes (Operating a Cache).
What each survivor has to carry
SIMULATEDArithmetic on the stated assumptions rather than a measurement. It transfers as a shape: fewer domains means a bigger step, and two is the worst non-trivial case. Uneven traffic, data gravity and sticky sessions all make the real requirement higher.
This is the whole lesson in one table. Assume traffic splits evenly across identical domains and one is lost. The middle column is what the survivors carry; the right-hand column is the provisioning requirement that follows.
The two-domain row is the one that surprises people. It is also the most common architecture, because two regions is where teams start.
| Domains | Normal share each | After losing one, each survivor | Each must be provisioned for |
|---|---|---|---|
| 2 | Half of total | All of total | Twice its steady-state load |
| 3 | A third of total | Half of total | One and a half times its steady-state load |
| 4 | A quarter of total | A third of total | One and a third times its steady-state load |
| N | 1/N of total | 1/(N-1) of total | N/(N-1) times its steady-state load |
| Active-passive | All in primary | All in passive | The passive side sized like the primary, or accept degradation |
Everything else that has to double
Compute is the part people remember, and it is the part that scales most easily. The rows below are the ones that turn a survivable failover into an incident, because most of them cannot be changed quickly and several cannot be changed at all during the event.
What the survivor actually runs out of
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Traffic shifts to the survivor | Database connection errors within seconds | Server-side connection maximum sized for one region's fleet | Budget connections for failover load; use a proxy or pooler (The Connection Budget) |
| Shifted traffic arrives | Cache hit rate collapses; database read load spikes far above double | The survivor's cache holds its own working set, not the transferred one | Expect and plan for a cold-cache period; pre-warm where feasible (Operating a Cache) |
| Outbound calls double in one region | Throttling from a managed service or third party | Quotas are commonly per-region or per-account and were sized for the normal share | Audit and raise quotas ahead of time — this cannot be done during the event |
| Autoscaler reacts to the step change | Capacity arrives minutes after saturation | Scaling lag against an instantaneous load step (How Autoscaling Fails) | Pre-scale before shifting; hold enough standing capacity to bridge the lag |
| Failed requests from the lost region retry | Offered load exceeds the traffic actually being moved | Retries plus reconnection storms on top of the shift | Shed and backoff during the transition; shift traffic in increments if possible |
| Writes concentrate in the survivor | Replication lag and write latency climb | The write path was sized for a fraction of the total | Size the write tier for failover load, or accept degraded write features (Operating a Production Database) |
| Failback returns traffic at once | A second incident in the recovered region | Cold caches and empty pools taking a full share instantly | Ramp traffic back gradually and watch hit rate before increasing |
A failover that fails on capacity
SIMULATEDA composite reconstruction produced for this lesson, not an incident report. The timings are illustrative of ordering only; what transfers is the sequence — shift, saturate, scale into the constraint, shed — and not the minutes.
The shape below is a reconstruction of the common case, not a report of a specific event. Notice that everything up to the traffic shift goes well: detection, decision and routing all work exactly as designed. The failure is entirely on the receiving side.
Routing worked; capacity did not
- T+0signalRegion A becomes unreachable; health checks fail
- T+2mactionFailover decision taken; DNS and global routing shift all traffic to region B
- T+2mchangeRegion B request rate steps to the full total, plus retries from the failed region
- T+3msignalRegion B latency rises sharply; connection pool wait time climbs
- T+3msignalDatabase rejects new connections — the fleet grew, the server maximum did not
- T+4mchangeAutoscaler adds instances; each new instance opens its own pool, worsening the connection shortage
- T+6msignalCache hit rate for shifted traffic is near zero; read load exceeds double the normal rate
- T+9mactionOperator caps the fleet size and enables shedding for bulk and background traffic
- T+12mrecoveryLatency stabilises at a degraded level; interactive traffic served, background deferred
- T+40mrecoveryCaches warm, load returns to a sustainable level for the single region
The two decisive moments are T+4m, where scaling made the binding constraint worse rather than better, and T+9m, where an operator chose degradation over collapse. Both were available before the incident: a connection budget and a shedding policy are things you write down in advance.
changesignalactionrecovery
How to do it properly
Most important first.
- State the provisioning requirement explicitly for each redundancy design: with N domains, each must carry total peak over N-1. Put it in the design document, not in someone's head.
- Fail over at peak, on purpose, on a schedule. A failover drill in a quiet hour tests routing; a drill at peak tests capacity (Disaster Recovery as an Operation).
- Pre-scale the survivor before shifting traffic, when the failover is planned. Scaling and shifting simultaneously means racing your own lag (Region Failover).
- Audit per-region quotas and connection limits against the failover load, not the steady-state load. This is the most commonly missed item and the cheapest to fix.
- Prefer more, smaller failure domains where the architecture allows. Three regions is materially cheaper than two for the same survivability (Operating in More Than One Region).
- Decide in advance what degrades. If carrying full load on one region is not affordable, say which features are shed during failover rather than discovering it live (Load Shedding).
- Include the reconnection and cache-fill surge in the requirement, because the failover peak is higher than the traffic being moved.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Blast radius if this is wrongEveryone
One testEveryone
What contains it
Nothing — this failure appears at the moment the containment mechanism is invoked, so the redundancy is the thing that fails. Only a drill at peak finds it beforehand.
What can go wrong
Failure modes, including of the mitigation
- Active-active provisioned for the steady-state split, so failover moves the outage rather than ending it.
- The survivor scales successfully and its database does not, converting a compute failover into a database incident (The Connection Budget).
- A per-region provider quota rejects the doubled call volume, and the failure looks like a dependency outage rather than a capacity mistake.
- Active-passive where the passive side is scaled down to save cost, so failover starts from a fleet that cannot serve and must scale under full load (Scale to Zero has the same shape at a smaller size).
- Failback done all at once, producing a second surge into a region whose caches are now cold.
- The mitigation failing: a failover drill that quietly excluded the database or the third-party integrations, so the rehearsed procedure is not the real one.
Misreads this invites
- "Active-active means we have twice the capacity." It means you have twice the capacity while both are healthy, and exactly enough while one is — if you paid for that.
- "We tested failover, so we are covered." Ask what the traffic level was during the test. If it was not peak, the capacity question is still open.
- "Autoscaling will handle the surge." Autoscaling reacts in minutes to a step change that lands in seconds, into a database and quota that do not scale with it.
- "We only need this for regions." The same arithmetic applies to zones, clusters, shards and node pools. It is milder with more domains, never absent.
- "The passive region is cheap because it is idle." It is cheap because it is small, and small is exactly the property that makes it unable to take over (Idle Capacity).
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
How you know it worked
- A failover drill executed at or near peak traffic, with latency and error rate recorded through the transition.
- A written capacity statement per domain showing the failover requirement alongside the steady-state one.
- Per-region quota and connection-limit values recorded, with the failover requirement next to each.
- Cache hit rate and database load recorded during a drill — the two signals that reveal whether the survivor was really carrying it.
How you get back
- Failing back is a second failover and deserves the same care: shift traffic gradually so the recovered region fills its caches and pools before it takes its full share (Region Failover).
- If the survivor is degraded rather than failing, rolling back the failover may be worse than continuing — decide on user impact, not on which region is nominally primary.
- Some failovers cannot be rolled back at all: once writes have been accepted in the survivor, going back is a data reconciliation problem rather than a routing change (Partial and Logical Data Recovery).
What to automate, and what stays human
- Automate the drill: a scheduled, announced, at-peak failover that runs whether or not anyone remembers it.
- Automate the pre-scale step so it cannot be forgotten in the order of operations under stress.
- Keep the decision to fail over human for anything ambiguous. Automatic failover on a false signal moves all traffic into a survivor for no reason, and that is a self-inflicted version of this exact problem.
What this costs
- Full survivability at peak with two domains means paying for roughly double the steady-state fleet. That is the honest price, and the alternative is not "cheaper resilience" — it is degraded service during failover, which is a legitimate choice made explicitly.
- Three or more domains lower the per-domain reserve but raise operational complexity, data placement difficulty and cross-domain traffic cost (Operating in More Than One Region).
- Active-passive with a small passive side is much cheaper and converts the capacity problem into a recovery-time problem — which may be exactly the right trade if the objectives allow it (RTO and RPO).
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe N to N-1 arithmetic holds for any redundancy scheme at any granularity — instances, zones, clusters, regions. Only the size of the step changes.
- CLOUD-SPECIFICPer-region quotas, per-account service limits and regional capacity availability are provider concepts, and providers differ in whether a limit is per-region, per-account or global. On-premises the equivalent constraint is physical: the survivor site has the hardware it has, and no amount of API calls adds more.
- SIMULATEDThe fractions in this lesson are produced by arithmetic on stated assumptions — even traffic split, identical domains, one domain lost — not measured from any deployment. They are relationships, and a real estate with uneven regions, sticky routing or data gravity needs more than they suggest.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
Databasereplication
Domains that do not exist yet
- — Distributed Systems — correlated failure, and why domains fail together more often than independence assumptions allow for.