The question this answers
Why did all three replicas fail at the same time?
Replication guarantees survival of *independent* failures only. Against a common cause — a shared dependency, a shared bug, a shared trigger, a shared saturation point — N replicas provide exactly the same availability as one, and cost N times as much.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A node knows it failed, or that a peer is unreachable. It cannot know whether the cause is local or shared, which is why every node in a correlated failure independently concludes that everyone else has failed — and why the resulting logs read like N separate incidents rather than one.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The arithmetic and the assumption underneath it
The calculation everyone has seen: if a component is available 99.9% of the time, three independent copies are all unavailable only 0.1%³ of the time, which is one in a billion. It is arithmetically correct and it depends entirely on the word *independent* — on the probability of the second failing being unaffected by the first having failed.
That assumption is almost never true for the failures that matter. Identical software means an input that crashes one replica crashes all of them, at the same moment, because they all received it. Identical configuration means a bad value is applied everywhere. Shared dependency means the config store, the certificate, the database, the DNS zone. Shared load means that when one replica dies its traffic redistributes onto the others, making their failure *more* likely rather than independent of the first. Shared trigger means a leap second, a certificate expiry, a scheduled job, a month rollover, a coordinated cache expiry.
The honest way to state it: replication protects extremely well against random hardware failure, which is genuinely close to independent, and protects poorly or not at all against everything else. Since random hardware failure is a small fraction of real outages, the availability of a replicated system is dominated by its correlated failure modes, and improving the number of replicas moves the number that does not matter.
| Independent? | Replication helps? | |
|---|---|---|
| Disk failure, power supply, single host faulttypical | Close to independent | Yes — this is what it is for |
| A request that crashes the processprotocol | No — all replicas receive it | No |
| Bad configuration pushprotocol | No — applied everywhere | No |
| Certificate expiryprotocol | No — same expiry everywhere | No |
| Shared dependency outageprotocol | No | No |
| Load redistribution after one failureassumption | Anti-correlated — the first failure causes the next | Makes it worse |
| Memory leak reaching the limittypical | No — same rate, similar start time | Briefly, if start times are staggered |
The load-redistribution trap
This one deserves its own treatment because it is not merely non-independent — it is anti-independent: the first failure actively increases the probability of the next. Ten instances each running at 70% of capacity lose one; the remaining nine now run at 78%. Lose another and the remaining eight are at 87%. Lose a third and the rest are past their limit, at which point they fail in quick succession and the service is gone. The failure rate accelerates rather than staying constant, which is why these incidents look like a cliff rather than a slope.
Retries make it sharper. When an instance fails, its in-flight requests are retried against the survivors, adding load precisely when they can least absorb it. The mechanism is the same one behind One Retry per Tier Is Not One Retry — It Multiplies and it turns a capacity event into a cascade — Performance owns the retry-storm treatment and it is worth reading, because the arithmetic of the amplification is what determines whether the system recovers on its own.
The defences are unglamorous and effective: keep enough headroom that losing a domain does not push the rest past their limit, cap retries with a budget rather than a count, and shed load rather than queueing it when saturated. The last one is counter-intuitive under pressure and is often what breaks the cycle — a system that refuses excess work stays up serving the work it accepted, while one that queues it stops serving anything.
Where correlation is deliberately reduced
Some correlation can be engineered away, and the techniques are worth knowing because they are cheap relative to their effect. Stagger the triggers: jitter cache expiries, spread scheduled jobs, avoid a fleet-wide restart at the same instant. Anything that happens to all instances at exactly the same time is a synchronised event waiting to become a synchronised failure — and this is where jitter belongs, not only in retry backoff.
Stagger the deploys: canary and progressive rollout exist precisely to break the correlation between "a change is bad" and "every instance has it". The value is not that the change is tested; it is that the blast radius of a bad change is bounded by the rollout percentage at the time it is detected.
Diversify where the cost is justified: two DNS providers, two certificate authorities, a fallback path that does not share the primary’s dependencies. This is expensive and rarely worth it for internal components, but for a small number of externally-facing dependencies — DNS is the classic — the correlation is total and the mitigation is well understood.
Vary the timing of resource exhaustion: instances started at the same moment with the same leak reach their limit at the same moment. Staggered start times convert a simultaneous failure into a sequence, which is survivable.
What cannot be engineered away is shared implementation. Every replica runs the same code, so every code-triggered failure is correlated. That is the argument for progressive delivery being a reliability mechanism rather than a release convenience.
- Jitter every periodic event: cache TTLs, scheduled jobs, health probes, token refreshes.
- Stagger deploys so a bad change reaches a bounded fraction before detection.
- Stagger process start times so resource-exhaustion failures do not coincide.
- Keep headroom sized for domain loss plus the retry surge that follows it.
- Diversify only the few external dependencies whose failure is total and whose alternatives are real.
Reading availability numbers honestly
The practical consequence for design review is to stop multiplying availability figures and start asking what a component’s replicas share. A useful exercise is to take a proposed design and, for each redundant group, list the causes that would take all members simultaneously. The list is always longer than expected, and it is usually the actual availability story — far more informative than the arithmetic.
It also reframes where effort goes. If a service’s replicas share a configuration store, adding a fourth replica improves nothing and adding a cached fallback for the configuration store improves a great deal. If a fleet fails together on a bad deploy, canary deployment is worth more than any amount of instance-level redundancy. Effort should go to breaking correlation, not to adding copies, because copies only address the failure mode that is already the least common.
One caveat in the other direction: correlation is not always bad. Deliberately shared fate can be simpler and more predictable than partial failure — a system that fails entirely is easier to reason about and recover than one in which an unknown subset failed. The goal is not zero correlation; it is knowing where the correlation is and choosing it.
Key points
- Availability arithmetic assumes independence, and the failures that matter are not independent.
- Identical software, identical config, shared dependencies and shared triggers all defeat replication entirely.
- Load redistribution is anti-independent: the first failure makes the next one more likely.
- Correlation is reduced by staggering — jitter, canary deploys, varied start times — not by adding replicas.
- For each redundant group, listing what its members share is more informative than any availability calculation.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A cause exists that affects all members of a redundant group: a shared input, a shared configuration, a shared dependency, or a shared moment.
- • The cause is triggered, and every member is exposed to it at once.
- • Members fail simultaneously or in rapid succession; redundancy provides no margin because there is no unaffected member.
- • Where the mechanism is load, each failure increases the load on survivors, accelerating the rate.
- • Recovery is also correlated, because every member attempts to restart against the same dependency at the same moment.
- • A single malformed request reaches every replica and crashes each one.
- • A configuration change is applied fleet-wide within seconds.
- • A certificate with one expiry date expires everywhere at once.
- • Cache entries created together expire together, and every instance misses simultaneously.
- • Instances started together leak at the same rate and hit their memory limit together.
- • Poison request: every replica crashes within seconds of each other. The operator sees a synchronised restart across the fleet and, in the logs, the same request id immediately before each crash.
- • Config-push outage: the fleet becomes unhealthy in the order the change rolled out. The operator sees error rate tracking rollout percentage — visible only if deploy and config changes are marked on the dashboards.
- • Cascading capacity loss: instances fail in accelerating succession. The operator sees an interval between failures that shortens, and per-instance load climbing with each one.
- • Thundering restart: the fleet restarts together and overwhelms a shared dependency, preventing any of them from starting. The operator sees a system that cannot recover from a failure it had already survived.
- • Synchronised cache expiry: a latency spike at a regular interval with no traffic change. The operator sees a sawtooth in cache hit rate exactly aligned to a TTL boundary.
- • Correlation is often *created* by coordination: anything that makes instances act in lockstep — a shared schedule, a fleet-wide config push, a synchronised deploy — converts independent components into a single one.
- • Reducing correlation therefore usually means removing coordination: jitter, independent decisions, per-instance staggering.
- • The exception is deliberate coordination that limits exposure, such as a rollout controller that stops on a health signal — coordination used to bound blast radius rather than to synchronise behaviour.
- • During a correlated failure, redundancy provides nothing; the system behaves as if it had one replica.
- • Components that genuinely do not share the cause continue normally, which is what makes cell-based isolation valuable.
- • Recovery is slower than expected because every instance recovers at the same time against the same dependencies.
- • Detect: alert on simultaneity itself — several instances failing within a short window is a different event from one failing, and should page differently.
- • Contain: stop the propagating mechanism first, whether that is a rollout, a poison message, or a retry storm.
- • Recover: bring instances back in a staggered fashion, because a synchronised restart recreates the correlation.
- • Reconcile: identify the common cause before restoring full traffic; restoring into an unchanged cause simply repeats it.
- • Verify: confirm that the mitigation actually breaks the correlation rather than reducing its probability — a jittered TTL is a fix, a longer TTL is not.
- • Time between instance failures within a group; a shrinking interval is the signature of load-driven cascade, and simultaneity is the signature of a common trigger.
- • Per-instance load headroom, so you can see whether the survivors can absorb another loss before it happens.
- • Rollout and config-change markers on every dashboard, since correlated failures are frequently change-triggered and the correlation with rollout percentage is the fastest diagnostic.
- • Distribution of periodic-event timing — TTL expiry, scheduled jobs, token refresh — where clustering shows synchronisation that has not yet caused an incident.
- • Any design review where redundancy is being counted as an availability argument.
- • Any post-incident review where several instances failed together, which is most of them.
- • Chasing independence for its own sake produces heterogeneous fleets that are harder to operate, and operational complexity is itself a source of outages.
- • Diversifying internal components rarely repays the cost; the correlation there is usually better addressed by staged rollout.
- • Reduce blast radius instead of correlation: cells or shuffle sharding bound the damage without requiring independence.
- • Add headroom rather than replicas, so that losing members does not push survivors past their limit.
- • Make recovery fast rather than failure rare — a correlated failure with a 90-second recovery may beat an uncorrelated one with a 30-minute one.
- • Accept the correlation and plan for total loss, which is sometimes simpler and more honest than defending a partial-failure story that does not hold.
Why all three replicas failed at once
What people believe, and what is true
Three replicas gives us nine nines.
Only against independent failures. Against a shared cause, three replicas give exactly the availability of one, at three times the cost.
Adding a fourth replica improves availability.
It improves tolerance of independent failures, which is already the least likely mode. Breaking a correlation is usually worth more than another copy.
The replicas are in different zones, so the failure cannot be correlated.
Physical separation addresses physical causes. Software, configuration, certificates and shared services are unaffected by geography.
Correlated failure is always a design flaw.
Sometimes shared fate is a deliberate simplification. The flaw is not knowing where the correlation is and counting availability as if it were absent.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Replicas fail together when they share a cause — the same code, the same config, the same dependency, the same moment. Availability arithmetic assumes they do not, which is why it is usually wrong.
Practical
For each redundant group, list what all members share and what would take them all at once. Then jitter every periodic event, stagger deploys and start times, keep headroom sized for the loss plus its retry surge, and mark changes on every dashboard so the rollout correlation is visible in the first minute of an incident.
Advanced
The sharpest version is the metastable failure: a system with two stable states, one healthy and one where retry and queueing load sustain themselves after the original trigger is gone. A correlated failure supplies the push, load redistribution supplies the amplification, and the system settles into the bad state and stays there — restarting it returns it to the bad state within seconds, because the queued work and pending retries are still there. Breaking out requires reducing the offered load below the threshold: shedding traffic, draining queues, or capping retries. This is why "we restarted everything and it came back down" is such a common incident narrative, and why load shedding is a recovery tool and not only a protection one.
Apply it
- ⚡ A fleet of forty instances fails within ninety seconds with no deploy and no dependency alert. Enumerate the hypotheses and the signal that distinguishes each.
- ⚡ Your cache TTL is 300 seconds and you see a latency spike every five minutes. Explain the mechanism and the one-line fix.
- 💬 Why is multiplying availability figures across replicas usually wrong?
- 💬 Give three causes that take all replicas of a service simultaneously.
- 💬 Instances are failing one after another with a shortening interval. What is happening and what stops it?