DebuggingGENERALSCALE-SPECIFICFRAMEWORK-SPECIFIC

Retry Storms

How a reasonable retry policy turns a dependency's brief degradation into a sustained outage, and what bounds it.

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

A dependency got slow and now it is completely down, with our request rate against it several times normal. What did we do?

The requirement

Calls to the payments provider occasionally fail transiently. The team wants them retried so a blip does not become a failed checkout.

The obvious build

Wrap every outbound call in a retry: three attempts, short fixed delay. It is a few lines, and it makes transient failures invisible.

Why it breaks

The moment the dependency degrades, every caller retries at once. Its offered load multiplies by the retry count exactly when it has least capacity.

How it breaks in production

The moment the dependency degrades, every caller retries at once. Its offered load multiplies by the retry count exactly when it has least capacity.
Retries stack across layers: the client retries, your gateway retries, your service retries, your HTTP library retries. Three layers of three attempts is twenty-seven requests for one intent.
Fixed delays synchronise. Every failed caller waits the same interval and returns simultaneously, producing a pulse rather than a spread.
Retrying a timeout means the original request may still be running. The dependency now processes both, doing more work per user intent while it is already overloaded.
Non-idempotent operations retried on timeout produce duplicates — double charges, double orders (Idempotency in Backends).
Your own service degrades too: each retry occupies a connection, a thread or a loop slot for the whole retry sequence, so your capacity falls while your load rises.

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

A retry policy is a positive feedback loop. Load causes failure, failure causes retries, retries cause load. Below a threshold the loop damps out; above it, the system does not recover on its own even after the original trigger is gone.
That is why retry storms outlive their cause: the dependency recovers capacity, is immediately hit by the accumulated retry load, fails again, and the loop re-arms. This is metastable failure — the system stays broken after the trigger is removed.
Amplification is multiplicative across layers, not additive. Nested retries multiply, which is why the effective factor is almost always higher than anyone intends (Cascading Failure).
Fixed intervals cause synchronisation: independent callers become correlated because they all failed at the same instant. Jitter breaks the correlation (Backoff and Jitter).
A retry is only safe when the operation is idempotent. "Retryable" (the error suggests a retry might succeed) and "safe to retry" (repeating it cannot cause harm) are different properties, and conflating them is how duplicates happen (Idempotency Keys).
The bound that actually works is a retry budget: a cap on retries as a fraction of total requests over a window, so amplification is limited globally rather than per call site (Rate Limiting).

The loop that outlives its trigger

A retry storm is not "too many retries". It is a feedback loop that crosses a threshold. Below the threshold, retries absorb transient failures and the system self-corrects. Above it, retries generate more failures than they recover, and the loop sustains itself after the original problem is gone.

This is why the incident does not end when the dependency's own incident ends. Their capacity returns, your accumulated retry load hits it immediately, it fails again, and everyone concludes their fix did not work.

Amplification multiplies across layers; it does not add.
Fixed delays synchronise callers who failed at the same moment.
Timeout-then-retry means the dependency may be doing the work twice per intent.
Retries consume the caller's capacity too — connections and slots held for the whole sequence.
The loop can be metastable: removing the trigger does not restore the system.

Amplification across layers

UserLLMAgentToolDataDecisionHumanGuardrail

What each control actually bounds

The controls are often described as a bundle, which hides the fact that each one bounds a different quantity. Adding all of them without understanding which is doing the work produces configuration nobody dares change.

Control, what it bounds, and how it fails

Trigger	Symptom	Cause	Response
No bound at all	Request rate to a failing dependency multiplies	Retries at every layer with defaults nobody audited	Inventory retry behaviour per layer; disable all but one
Attempt cap only	Bounded per request, unbounded across the fleet	A cap limits one call site, not aggregate load	Add a retry budget expressed as a share of total requests
Fixed delay	Load arrives in synchronised pulses	All callers failed at once and wait the same time	Full jitter — randomise across the whole interval, not around it
Backoff without a deadline	Requests occupy resources for tens of seconds	Per-attempt timeouts do not bound the sequence	One deadline for the whole operation, propagated to every attempt (Timeouts)
No breaker	Traffic continues against a dependency that is fully down	Every request pays the full timeout before failing	Open on sustained failure; probe with limited concurrency before closing (Circuit Breakers)
Breaker closes fully at once	Recovery followed immediately by a second collapse	All suppressed load released simultaneously	Half-open with a small probe allowance, then ramp
Retrying non-idempotent writes	Duplicate orders or charges after a dependency blip	"Retryable" confused with "safe to retry"	Idempotency key per intent, honoured by the receiver (Idempotency Keys)
Dead-letter re-enqueued immediately	A queue that never drains and a consumer at 100%	Failure loop moved from HTTP to the queue	Delay and cap redelivery; park poison messages (Dead-Letter Queues)

A policy with every bound in place

SIMPLIFIEDShown as one function for readability. In practice this belongs in a shared client wrapper or a service mesh, so the budget is genuinely shared and the policy cannot be reimplemented differently at each call site.

Written out, a safe retry is not much longer than an unsafe one. The difference is that each line bounds a specific quantity: attempts, elapsed time, aggregate share, and whether the operation qualifies for a retry at all.

Bounded retry: deadline, jitter, budget, idempotency

1async function callWithRetry<T>(
2  op: (signal: AbortSignal) => Promise<T>,
3  opts: { deadlineMs: number; maxAttempts: number; key?: string },
4): Promise<T> {
5  const deadline = Date.now() + opts.deadlineMs
6  let attempt = 0
7 
8  for (;;) {
9    attempt++
10    if (attempt > 1) {
11      // one global budget for the whole process, not per call site
12      if (!retryBudget.tryConsume()) throw new BudgetExhausted()
13      metrics.inc('outbound.retry', { attempt: String(attempt) })
14    }
15 
16    const left = deadline - Date.now()
17    if (left <= 0) throw new DeadlineExceeded()
18 
19    const ctl = new AbortController()
20    const timer = setTimeout(() => ctl.abort(), Math.min(left, PER_ATTEMPT_MS))
21    try {
22      return await op(ctl.signal)
23    } catch (err) {
24      // retryable describes the error; safe-to-retry describes the operation
25      const retryable = isTransient(err)
26      const safe = opts.key !== undefined || isReadOnly(op)
27      if (!retryable || !safe || attempt >= opts.maxAttempts) throw err
28 
29      // full jitter: uniform over [0, backoff], not backoff +/- noise
30      const backoff = Math.min(BASE_MS * 2 ** (attempt - 1), MAX_BACKOFF_MS)
31      await sleep(Math.random() * Math.min(backoff, deadline - Date.now()))
32    } finally {
33      clearTimeout(timer)
34    }
35  }
36}

Four independent bounds, and each one fails differently if removed: without the deadline a request can occupy a slot indefinitely; without the budget the fleet amplifies even though each call site is capped; without full jitter every caller returns together; without the idempotency key a retried write becomes a duplicate.

How to build it

Most important first.

Retry only what is idempotent, and make writes idempotent with a key so they qualify (Idempotency Keys).
Retry at exactly one layer. Pick it deliberately — usually the outermost one that knows the business intent — and disable retries in the libraries and proxies below it.
Exponential backoff with full jitter. The jitter matters more than the exponent; it is what stops the synchronised pulse (Backoff and Jitter).
Cap total attempts and total elapsed time. A deadline for the whole operation is stronger than a per-attempt timeout, because it bounds the sequence (Timeouts).
Add a retry budget: stop retrying when retries exceed some small share of requests in the window. This is the control that limits amplification when everything is failing at once.
Add a circuit breaker so that a sustained failure stops producing traffic at all, giving the dependency room to recover (Circuit Breakers).
Do not retry a 4xx. Retrying a validation error or an authorization failure will never succeed and is pure amplification (Status Codes From the Server's Side).
Shed load rather than queueing it. Failing fast when the breaker is open is better for everyone than holding requests that will fail later (Backpressure).

What can go wrong

Failure modes

A circuit breaker that closes all at once and sends the full accumulated load into a dependency that has recovered only partially — the breaker itself becomes the pulse generator. Half-open probes with limited concurrency exist for this.
A retry budget set per instance, so the effective fleet-wide budget multiplies by instance count.
Backoff added at one layer while another layer retries tightly, leaving amplification untouched.
Jitter implemented as a small random addition to a fixed delay, which does not decorrelate enough to matter.
Retries that consume the caller's own capacity: connections held for the entire retry sequence, so the caller exhausts its pool defending against the callee (Connection Pool Exhaustion).
Dead-letter handling that immediately re-enqueues, turning a queue into an infinite retry loop (Dead-Letter Queues).

What can race

A retry can race the original request when the first attempt timed out but is still executing, so both run concurrently against the same state (Backend Races).
Breaker state is shared: concurrent requests read and update it, and a naive implementation lets many requests through in the instant it half-opens.

Security

Retries against an authentication endpoint can trip account lockout, turning a transient failure into a denial of service against your own users (Credentials and Password Handling).
Retried non-idempotent financial operations are a double-charge bug and a compliance problem, not merely a correctness one.
An attacker who can induce failures on a cheap endpoint can use your retry policy as an amplifier against a downstream service — your service becomes the attack tool (Rate Limiting).
Retry logic that reuses a signed request must respect the signature's validity window; blindly replaying an expired signed request produces confusing auth failures (Webhook Signature Verification).

Misreads

"Retries make the system more reliable." They make *transient, isolated* failures survivable and *correlated* failures worse. The distinction is the whole lesson.
"Exponential backoff is enough." Without jitter, callers stay synchronised; without a budget or a breaker, backoff only delays the pulse (Backoff and Jitter).
"A 500 is retryable, so retrying is safe." Retryable describes the error. Safe describes the operation. A 500 from a payment capture may mean the capture succeeded.
"We only retry three times." Count the layers. Three at each of three layers is twenty-seven.
"The dependency recovered but we are still failing, so it must be something else." That is exactly what metastable failure looks like — the loop is now self-sustaining.

Operating it

How you see it in production

Outbound attempts and outbound requests as two separate counters per dependency. Their ratio is the amplification factor, and it is the single most useful number here (Retries).
Retry budget consumption as a gauge — how close the service is to suppressing retries.
Circuit breaker state transitions as events on the dashboard, alongside dependency error rate.
Attempt-number distribution: a healthy system does almost everything on attempt one, and a shift in that distribution is an early warning.
Your own in-flight outbound request count. A storm shows as a rising in-flight count with falling completion rate.
Duplicate detection hits — a rising rate means retries are reaching non-idempotent paths (Duplicate Detection).

What changes at 10x and 100x

Amplification is proportional to caller count. With 10 instances a storm is annoying; with 500 it is an outage the dependency cannot serve out of.
At scale, per-call-site retry configuration becomes unmanageable and the budget has to be enforced centrally — in a shared client library or a service mesh.
A small service calling a large one rarely causes a storm; a large fleet calling a small internal service does so easily. The asymmetry decides how much of this you need.
At very small scale, three retries with jitter and a timeout is genuinely sufficient, and a breaker plus a budget is over-engineering.

What this costs

Every retry restraint reduces success rate for genuinely transient failures. Fewer retries means more errors surfaced to users during blips, which is the price of not amplifying real outages.
Circuit breakers fail requests that might have succeeded, and a badly tuned threshold opens on noise.
Idempotency keys make retries safe and add storage, expiry policy and a lookup on the write path (Idempotency Storage).
Centralised retry policy is safer and less flexible; teams with unusual requirements will work around it.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALThe feedback loop is protocol- and language-independent; it appears in HTTP clients, queue consumers and database drivers alike.
SCALE-SPECIFICBelow a few instances, retries rarely generate enough load to hurt a dependency and simple backoff is adequate. Above a large fleet, an unbudgeted retry policy is a latent outage — the same configuration is correct in one place and dangerous in the other.
FRAMEWORK-SPECIFICMany HTTP clients, SDKs, proxies and service meshes retry by default and do not say so prominently. The amplification factor of a system is the product of every layer's default, so auditing those defaults is part of the design, not an optimisation.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designretryability retries-and-timeouts idempotency-keys quotas-vs-rate-limits

Architecturecircuit-breaker reliability-patterns backpressure

Performanceretry-storms saturation tail-latency

Concurrencyconcurrent-backpressure

Domains that do not exist yet

— Distributed Systems — metastable failure, load shedding and why a system can stay in a failed state after the trigger is removed.