The question this answers
Every tier retries three times, which seems reasonable. Why is the bottom of my stack seeing thirty times the traffic?
None, and that is the finding. A per-tier retry policy bounds attempts *at that tier only*. Across a chain of d retrying tiers with a attempts each, the work reaching the bottom is bounded by a^d times the original — a bound that grows exponentially in the depth of the call graph, which is not a bound anyone intended to set.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A tier knows how many attempts *it* has made for the request in its hand. It does not know how many attempts its caller has already made for the same user action, how many attempts its callee will make below it, or how many of the requests currently arriving are duplicates of each other. Nothing in a standard RPC carries that information, which is precisely why the multiplication is invisible from every individual vantage point.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The multiplication, worked through
Performance covers what a retry storm does to a single dependency — the load you generated yourself. The distributed question is different and comes one layer earlier: how much load is there to generate? The answer is set by the shape of the call graph, and it is multiplicative.
Take a mundane architecture. The gateway calls the order service, which calls the inventory service, which calls the database. Every hop is configured with the industry-standard "retry up to 3 times", which every team involved considers conservative. Now count what one user request can produce at the bottom.
The gateway’s three attempts each cause the order service to make three attempts, each of which causes the inventory service to make three: 3 × 3 × 3 = 27 database calls for one user action. Add fan-out — the order service calls inventory once per line item, say 8 — and one user request is up to 216 database calls where the happy path needed 8. The retry factor and the fan-out factor multiply too.
The property that makes this dangerous rather than merely wasteful: the multiplication is worst exactly when the system is least able to take it. During health, retries almost never fire and the amplification factor is ~1. During a partial failure at the bottom, every tier retries, and the factor jumps to its maximum in seconds. The load function is a step function keyed to the health of the thing it will crush.
| Tier | Fan-out per call | Attempts | Calls issued at this tier | Cumulative multiplier |
|---|---|---|---|---|
| Client → Gatewayassumption | 1 | 3 | 3 | 3× |
| Gateway → Order serviceassumption | 1 | 3 | 9 | 9× |
| Order → Inventory (per item)assumption | 8 | 3 | 216 | 216× |
| Inventory → Databaseassumption | 1 | 3 | 648 | 648× |
Nested timeouts make it worse than the arithmetic suggests
The a^d figure assumes each attempt completes before the next begins. Under the timeout configurations most systems actually have, they overlap — and the concurrent load is higher than the total-work calculation implies.
The mechanism is a timeout inversion. Suppose the gateway waits 2 seconds and the order service also waits 2 seconds on inventory. The gateway’s first attempt times out at 2s and it retries — but the order service’s original call is still in flight, because nothing told it to stop. Now there are two live chains doing the same work. At 4s there are three. The database sees not 27 sequential calls but a growing set of *simultaneous* ones, each holding a connection, and the connection pool is what fails first.
This is why A Deadline Is Divided Across the Call Chain, Not Repeated at Every Hop and Pass the Remaining Budget Down, Not a Fresh One are load-control mechanisms and not merely latency hygiene. A propagated deadline makes the abandoned work stop: when the gateway gives up, the remaining budget downstream is zero and every tier drops its in-flight attempt instead of continuing to consume capacity for a caller who has left.
Retry at one layer only, and choose which
The fix that actually holds is structural: decide which single layer owns retrying for a given failure, and make every other layer pass the failure through. Multiplication requires at least two retrying layers; remove one factor and the exponent collapses.
Which layer should own it depends on what it can see. The layer nearest the failure retries fastest and recovers from the most transient faults — a connection reset is best retried by the client that owns the connection. The layer nearest the user knows whether a retry is still worth doing, because it holds the deadline and knows the user is still there. Choosing the bottom means fast recovery from blips and no knowledge of whether anyone still cares; choosing the top means the opposite.
The practical settlement most large systems reach: retry at the bottom for *connection-level* faults only (reset, refused, DNS), where the attempt is cheap and provably did not reach the application; retry at the top for *application-level* faults, where a deadline exists to bound the effort. Middle tiers do not retry at all. It reads as under-engineering and it is the thing that keeps the exponent at 1.
Everything else in this module is damage control for the cases where you cannot get that structure: Cap Retries as a Fraction of Traffic, Not as a Count per Request caps the aggregate, Without Jitter, Every Client That Failed Together Retries Together decorrelates the timing, and Architecture’s circuit breaker cuts the branch entirely.
1// Middle tiers do not retry. They classify and pass through, and they2// mark responses so an upper tier can decide whether a retry is sound.3type Outcome =4 | { kind: 'ok'; value: unknown }5 | { kind: 'retryable'; reason: 'connect' | 'unavailable'; sideEffectPossible: false }6 | { kind: 'ambiguous'; reason: 'timeout'; sideEffectPossible: true }7 | { kind: 'terminal'; reason: 'invalid' | 'forbidden' }8 9// Only two places in the whole chain are allowed to loop:10// 1. the connection layer, for faults that provably never reached the app11// 2. the edge, which holds the deadline and the user12// 'ambiguous' is retryable ONLY if the operation carries an idempotency key —13// otherwise a retry here is a duplicate side effect, not a recovery.14function mayRetryHere(o: Outcome, layer: 'connection' | 'middle' | 'edge', idempotent: boolean): boolean {15 if (layer === 'middle') return false16 if (o.kind === 'retryable') return layer === 'connection'17 if (o.kind === 'ambiguous') return layer === 'edge' && idempotent18 return false19}Key points
- Retries at independent tiers multiply:
aattempts acrossdtiers isa^dwork at the bottom, nota. - Fan-out multiplies with the retry factor — 8 line items and 3 attempts per tier is hundreds of calls for one user action.
- The amplification factor is ~1 while healthy and jumps to maximum during a partial failure: the load spikes exactly when capacity is lowest.
- Without propagated deadlines and cancellation, retried attempts overlap the originals, so peak concurrency rises as well as total work.
- The structural fix is to retry at exactly one layer. Budgets, backoff and breakers are mitigations for when you cannot.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A leaf dependency degrades: latency rises past the timeout of the tier above it.
- • That tier records a failure and retries, per its own policy, without knowing whether its caller is also retrying.
- • Each retry re-enters the tier below, which applies its own retry policy to each of them.
- • Because no attempt is cancelled when its caller gives up, retried attempts run concurrently with the originals.
- • Offered load at the leaf multiplies by the product of the per-tier attempt counts, sustaining the degradation that triggered it.
- • Every tier is configured independently, often by different teams, and no artefact records the composed factor.
- • A default retry policy inside a client SDK adds a factor nobody configured or knows about.
- • Timeouts are equal or increasing down the chain, so upper-tier retries overlap lower-tier work in flight.
- • A load balancer or service mesh retries as well, adding a hidden tier to the product.
- • Retries of ambiguous outcomes duplicate side effects as well as load.
- • The leaf never recovers after the trigger passes: database CPU stays at 100% and query rate stays at 20× baseline long after the original slow query was fixed. The load is now entirely self-generated.
- • Connection pool exhaustion at a middle tier while its own request rate looks normal — the tier is holding concurrent duplicate calls, which shows as in-flight count far above arrival rate × latency.
- • Ratio inversion in the traffic graph: requests-per-second rises as you descend the stack even though every tier reports the same error rate. Plot rps by tier during an incident and the multiplier is directly visible.
- • Duplicate side effects at scale: 27 charge attempts for one order, all with different request ids, visible only in the ledger and in no service’s error log.
- • No coordination is needed to cause this — that is what makes it a distributed-systems problem rather than a configuration problem. Each tier acts locally and correctly.
- • Bounding it requires shared information: a propagated deadline, an attempt counter carried in the request context, or a retry budget scoped to the dependency rather than the request.
- • The cheapest shared information is the deadline, because it is a single number that every tier can act on unilaterally and it decays on its own.
- • Correctness of an individual admitted request is unaffected — this is a load pathology, not a consistency one, unless the retried operations are non-idempotent.
- • The system does not self-recover: because the load is generated by the failure, removing the original trigger does not remove the load.
- • Capacity added during the incident is consumed by amplified retries, so scaling up often fails to help until the retries are capped.
- • Detect: chart request rate per tier on one graph. Amplification is the only thing that makes rate *increase* down the stack while user traffic is flat.
- • Contain: disable retries at middle tiers first — the fastest incident lever, and it removes a whole factor from the product.
- • Recover: shed at the leaf so it can drain, and keep shedding until the retry traffic decays. The system will not exit the equilibrium on its own.
- • Reconcile: audit for duplicate side effects created during the storm; the retry count is a lower bound on how many duplicates were possible.
- • Verify: write down the composed multiplier as a number (
a^d × fan-out) and keep it in the same place as the architecture diagram. If nobody can state it, it is not controlled.
- • Requests per second at each tier, on a single chart, normalised to user-facing request rate — the multiplier as a directly readable line.
- • Retry ratio per dependency: attempts divided by distinct logical requests. Healthy is a percent or two; an incident takes it into the hundreds.
- • In-flight concurrency versus arrival rate × mean latency; a gap means abandoned work is still running.
- • A propagated attempt counter in the request context, logged at the leaf. This is the only way to see the *composed* count rather than each tier’s local one.
- • Understanding this is not optional for any call graph deeper than two hops — which is every microservice system.
- • It is most valuable when designing timeout and retry defaults for a shared client library, because that is where a single decision multiplies across an organisation.
- • The analysis is overkill for a two-tier system with a single retry point, where the factor is what it says on the tin.
- • Removing all retries because of this lesson is the wrong lesson: transient faults are real and a system with no retries at all has a worse success rate. The point is to have exactly one retrying layer, not zero.
- • Retry only at the edge, where the deadline lives — simplest to reason about, slower to recover from transient blips.
- • Retry only at the connection layer for faults that provably never reached the application, and fail everything else through. Fast, safe, and does not multiply.
- • Cap the aggregate instead of the structure with a retry budget — the right move when you cannot change every tier. See Cap Retries as a Fraction of Traffic, Not as a Count per Request.
- • Cut the branch with a circuit breaker so failed dependencies stop receiving attempts at all; Architecture owns that mechanism.
Retry storm: the retries become the load
| Tier | Load at the stated failure rate | Amplification | When everything fails |
|---|---|---|---|
| tier-1 | 202/s | 1.01× | 600/s · 3.00× |
| tier-2 | 204/s | 1.02× | 1,800/s · 9.00× |
| tier-3 | 205/s | 1.03× | 5,400/s · 27.0× |
What people believe, and what is true
Three retries is a conservative setting.
Three retries at one tier is conservative. Three retries at each of four tiers is 81× and nobody configured that number.
Retries only add load when things are already broken, so they cannot cause the outage.
They cannot cause the *trigger*. They routinely cause the outage, by converting a recoverable ten-second blip into a sustained 20× load the leaf can never drain.
Adding capacity during the incident will help.
Amplified retry load scales with the failure, not with user demand. New capacity is absorbed by retries until the retry traffic is capped.
Our retries are sequential so peak load is unchanged.
Only if abandoned attempts are cancelled. Without propagated deadlines the upper retry starts while the lower attempt is still running, so peak concurrency multiplies too.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Each tier that retries multiplies the load reaching the tier below. Three tiers retrying three times each is 27× at the bottom, not 3×.
Practical
Write down your call graph, mark every layer that retries — including SDK defaults and the service mesh — and multiply the attempt counts. Then remove retries from every layer except one. Verify by plotting request rate per tier during a load test with an injected leaf failure.
Advanced
Model it as a branching process: with per-tier attempt count a and leaf failure probability p, the expected number of leaf calls per user request is Σ (a·p)^k over the depth, which converges while a·p < 1 and diverges above it. That inequality is the phase transition — below it retries are a rounding error, above it the system has a self-sustaining load source that outlives its trigger. Retry budgets work precisely by forcing a·p back below 1 regardless of p.
Apply it
- 💬 One user request, four tiers, each retrying three times, 8-way fan-out at tier three. How many database calls in the worst case, and when does that worst case occur?
- 💬 A database incident ends but the load stays at 20× for an hour. What is generating it and how do you stop it?
- 💬 Which single layer should own retries in your architecture, and what do you give up by choosing it?