IntegrationsGENERALPROTOCOL-SPECIFICSCALE-SPECIFIC

Retries

"Retryable" and "safe to retry" are different properties, and confusing them is how a transient error becomes a duplicate charge.

What actually happens How to build it

The requirement, the obvious build, and why it breaks

Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.

The question

When should I retry a failed call, and what has to be true before retrying is safe?

The requirement

The payment provider occasionally returns a 503 that succeeds a second later. We want those to succeed without a user seeing an error — without ever charging anyone twice.

The obvious build

Wrap the call in a loop: try three times, then give up. Transient failures disappear and the code is four lines.

Why it breaks

A 500 from a POST /charges is retried. The first attempt actually succeeded and the response was lost. The customer is charged twice, and you find out from their bank (Idempotency in Backends).

How it breaks in production

A 500 from a POST /charges is retried. The first attempt actually succeeded and the response was lost. The customer is charged twice, and you find out from their bank (Idempotency in Backends).
The dependency is overloaded. Every client retries. The load triples at the exact moment the dependency is least able to absorb it, and a recoverable degradation becomes a full outage (Retry Storms).
The SDK retries three times, your wrapper retries three times, and the caller retries three times: twenty-seven attempts for one user action.
A 400 from a malformed request is retried three times, wasting the request budget on an error that will never change.
A 429 is retried immediately, which is the one response that explicitly tells you not to (Rate Limiting).
Retries with per-attempt timeouts exceed the caller's deadline, so the user gets a 504 while your service is still on attempt two (Timeouts).

Requirement→API Contract→Application Logic→Data Access→External Deps→Concurrency→Failure→Security→Observability→Deployment→Scale

What is actually happening

Two different properties get collapsed into one word, and separating them is the entire lesson. Retryable is a property of the *error*: is this failure likely to be transient? Safe to retry is a property of the *operation*: does executing it twice have the same effect as executing it once?
A 503 is retryable. POST /charges is not safe to retry. Both can be true at the same time, and retrying anyway is how money moves twice. You need both properties before a retry is correct (Retryability: Telling Clients What To Do Next in API Design covers the contract side).
The ambiguous errors are the important ones. A timeout, a connection reset or a 502 tells you nothing about whether the other side acted — so they are simultaneously the most worth retrying and the most dangerous to retry blindly.
Idempotency is what converts "not safe" into "safe". An idempotency key sent with the request lets the dependency recognise a repeat and return the original outcome instead of performing the operation again (Idempotency Keys).
Retries multiply load precisely when the dependency is weakest. A service at capacity that starts failing 50% of requests sees its incoming load rise by 50% if every client retries once — which pushes the failure rate up, which triggers more retries. This is a positive feedback loop and it does not stabilise on its own.
A retry budget breaks that loop by capping retries as a fraction of total traffic — for example, retries may not exceed a small percentage of successful requests in a rolling window. Under normal conditions the budget is never touched; during an outage it stops retries almost entirely, which is exactly what you want.
Retries compose multiplicatively across layers. The rule is one retrying layer per call path, chosen deliberately, with every other layer configured not to retry.

Two questions, not one

The table below is the core of this lesson. The two centre columns are independent properties, and a retry is correct only when both are satisfied. The interesting rows are the ones where they disagree — where the error invites a retry and the operation forbids it.

Notice that the fix in almost every disagreeing row is the same: an idempotency key converts an unsafe operation into a safe one, and it is the only thing that does. Waiting longer, retrying fewer times or hoping the timeout was clean do not change the safety property at all.

Situation	Retryable? (the error)	Safe to retry? (the operation)	Correct action
`GET /prices` returns 503	Yes — transient	Yes — a read	Retry with backoff.
`POST /charges` returns 503	Yes — transient	No — may have charged	Retry only with an idempotency key (Idempotency Keys).
`POST /charges` times out	Yes — ambiguous	No — outcome unknown	Retry with the same key, or reconcile against their records.
`POST /charges` returns 400	No — permanent	Irrelevant	Fail. Fix the request; retrying wastes budget.
Any call returns 401	No — not without action	Irrelevant	Refresh the credential once, then retry. Never loop (API Keys).
Any call returns 429	Yes — after a delay	Depends on operation	Wait for `Retry-After`; do not retry immediately (Rate Limiting).
`PUT /users/42` returns 502	Yes — transient	Yes — naturally idempotent	Retry. A `PUT` of the same body twice is one outcome.
`DELETE /orders/9` returns 500	Yes — transient	Yes — already-deleted is the same end state	Retry; treat 404 on retry as success.
Send email returns a socket error	Yes — transient	No — may have sent	Deduplicate on a message key, or accept a rare duplicate (Email and Notifications).

Why retries make outages worse

GENERALThe structure is language-independent. What differs is where the budget lives: in a single process it can be an in-memory token bucket, but across many instances it is only a true service-wide budget if it is held in a shared store with atomic decrement — otherwise each instance enforces its own and the effective budget multiplies by instance count (Stateless Services).

A dependency at capacity starts shedding load. Every client responds by sending more. The dependency's effective load rises by the retry multiplier at the moment its capacity has fallen, so its failure rate rises, so the multiplier rises. Nothing in that loop pushes back.

A retry budget is the pushback. Expressed as a ceiling on retries relative to successful requests over a rolling window, it is invisible in normal operation — a healthy service almost never retries — and it collapses retry traffic to nearly zero during an outage. That is precisely the behaviour you want and precisely the opposite of a fixed per-call attempt count, which retries hardest when things are worst.

A retry policy with the properties that matter

1type Attempt = { attempt: number; deadline: Deadline }
2 
3async function callWithRetry<T>(
4  op: (a: Attempt) => Promise<T>,
5  {
6    idempotencyKey,     // generated ONCE per logical operation, outside this call
7    deadline,           // absolute; retries must fit inside it
8    maxAttempts,
9    budget,             // shared across the service, not per call site
10    classify,           // error -> 'permanent' | 'transient' | 'throttled'
11  }: RetryPolicy<T>,
12): Promise<T> {
13  let attempt = 0
14  for (;;) {
15    attempt++
16    try {
17      return await op({ attempt, deadline })
18    } catch (err) {
19      const kind = classify(err)
20 
21      // 1. the error is not retryable at all
22      if (kind === 'permanent') throw err
23 
24      // 2. the operation is not safe to repeat and we cannot make it safe
25      if (!idempotencyKey && mutates(op)) throw err
26 
27      // 3. out of attempts, out of time, or out of budget
28      if (attempt >= maxAttempts) throw err
29      if (!budget.tryConsume()) throw new RetriesBudgetExhausted(err)
30      const wait = kind === 'throttled'
31        ? retryAfterMs(err)              // obey the dependency
32        : jitteredBackoff(attempt)       // do not synchronise with everyone else
33      if (deadline.remainingMs() <= wait) throw new DeadlineExceeded(err)
34 
35      await sleep(wait)
36    }
37  }
38}

Three guards do the real work and none of them is the attempt count. Check 2 is the retryable/safe distinction made executable; check 3's budget is what prevents your retries from becoming the dependency's next problem; and the deadline check is what stops retries from outliving the request that wanted them.

The failures retries cause

Every row here is a retry mechanism producing an outcome worse than not retrying at all. They are not exotic; they are the standard results of the four-line loop.

When the mitigation is the incident

Trigger	Symptom	Cause	Response
Ambiguous failure on a non-idempotent write	Duplicate charge, duplicate order, two emails.	"Retryable" treated as "safe to retry".	Idempotency key generated once per operation and reused across attempts.
Dependency degrades to 50% errors	Dependency goes from degraded to fully down within minutes.	Every client amplifying load at the worst moment.	Retry budget plus a breaker; retries handle blips, breakers handle outages (Circuit Breakers).
SDK retries plus your retries plus gateway retries	Vendor reports far more calls than you sent; quota exhausted.	Multiplicative retries across layers.	Choose one retrying layer; explicitly disable the others.
All clients fail at the same instant	Traffic arrives in synchronised waves after each backoff interval.	Deterministic backoff with no jitter (Backoff and Jitter).	Randomise the wait; never let clients share a schedule.
429 retried immediately	Rate limit never clears; the key gets blocked.	Treating throttling as a generic transient error.	Honour `Retry-After`; shed or queue instead of retrying (Rate Limiting).
Retries exceed the caller's deadline	User gets a 504 while your service is on attempt two, doing work nobody will receive.	Attempt count bounded, elapsed time not.	Check the remaining deadline before every attempt (Timeouts).
Key regenerated inside the loop	Idempotency present and useless; duplicates anyway.	The key identifies the attempt rather than the operation.	Generate at the boundary where the intent originates (The Idempotency Key Flow).

How to build it

Most important first.

Classify the error before deciding anything. Permanent client errors (400, 404, 422) must never be retried; authorization failures (401, 403) must not be retried without re-authenticating; 429 is retryable but only after the delay the response asks for; 5xx and network errors are retryable; timeouts are retryable and ambiguous (An Error Taxonomy That Maps Cause to Response).
Ask separately whether the operation is safe. Reads generally are. Writes generally are not, unless the dependency supports an idempotency key or the operation is naturally idempotent (a set to a fixed value, a delete, an upsert keyed on your id).
Send an idempotency key on every mutating call that you might retry, generated once for the logical operation and reused across all attempts of it. A key regenerated per attempt is not an idempotency key (The Idempotency Key Flow).
Bound retries by budget and deadline, not only by attempt count. Total elapsed time across attempts must fit inside the request's remaining deadline, and retries across the service must stay under a global fraction of traffic.
Back off exponentially with jitter between attempts, and respect Retry-After when the dependency provides it (Backoff and Jitter).
Pick exactly one layer to retry at and turn it off everywhere else. Disable SDK-internal retries explicitly, and write down which layer owns it.
Combine retries with a breaker so that sustained failure stops producing attempts at all — retries handle blips, breakers handle outages (Circuit Breakers).
For operations that are neither safe nor keyable, do not retry in the request path. Enqueue them, where an idempotent consumer can retry safely and slowly (Job Idempotency).

What can go wrong

Failure modes

Retrying a non-idempotent write on an ambiguous error: duplicate charges, duplicate orders, duplicate emails.
Retry storms — the mitigation causing the outage it was meant to survive (Retry Storms).
Retries hidden in an SDK, invisible in your metrics, so your dashboards show one call and the vendor sees three.
An idempotency key generated inside the retry loop, so every attempt looks like a new operation to the dependency.
Retrying 401 without refreshing the credential, turning an auth problem into a lockout or an abuse flag.
A retry budget configured but never observed, so nobody notices when it is permanently exhausted and retries silently stopped working.
Retries applied to a request that already timed out at the caller, doing work for a client that is gone (Timeouts).

What can race

A retry overlapping an original attempt that has not actually failed — two concurrent executions of one operation, which sequential reasoning about retries does not predict (Duplicate Detection).
Many clients retrying in lockstep after a shared failure, synchronising into a burst that the dependency sees as a coordinated attack (Backoff and Jitter, Thundering Herd in Concurrency).
Two attempts of one operation arriving at two instances of the dependency simultaneously, where an idempotency key must be enforced with an atomic claim rather than a read-then-write check (Atomic Operations).

Security

Repeated authentication failures look like credential stuffing from the dependency's side and can get your API key throttled or suspended. Never retry a 401 without a fresh token (API Keys).
Retries against your own endpoints amplify any request an attacker can make expensive — a retrying gateway turns one hostile request into several (Rate Limiting).
Retried mutating operations without idempotency keys are a duplicate-transaction risk with direct financial consequences, and they are frequently in scope for payment compliance review.
Retry loops that log the full request on each attempt multiply exposure of whatever the payload contained (Secrets in Logs).

Misreads

"A 500 is retryable, so I can retry it." Retryable describes the error. Whether it is *safe* depends on the operation. A 500 from a charge endpoint may mean the charge succeeded and the response failed — retrying that without a key charges twice.
"Timeouts are safe to retry because nothing happened." A timeout is the case where you have the least information. The work may have completed in full.
"Three attempts is standard." Attempt count is the least important parameter. Whether the operation is idempotent, whether there is a budget, and whether backoff has jitter all matter more.
"Retries make the system more reliable." They make it more reliable against transient failure and less reliable against saturation, because they add load exactly when there is none to spare.
"The SDK retries for me, so I am covered." You are exposed: its retries are invisible to your metrics, and it does not know whether your operation is idempotent — so it may be retrying things it should not.
"Idempotency keys are the payment provider's concern." They are a contract between you and the dependency, and they only work if *you* generate the key once per logical operation and reuse it across attempts.

Operating it

How you see it in production

Attempt count distribution per dependency, not just a retry counter. Most calls should be one attempt; a shift toward two and three is the dependency degrading before the error rate moves.
Retry budget utilisation as a gauge. Approaching the cap during an incident is expected; sitting near it during normal traffic means the budget or the policy is wrong.
Separate counters for "retried and eventually succeeded" and "retried and gave up". The first is the value the mechanism delivers; the second is what users experienced.
Compare your outbound request count with the vendor's reported count. A persistent gap is a retrying layer you did not know about.
Duplicate-suppressed responses reported by the dependency (many payment APIs return a header or a replayed flag). Non-zero means your retries are being caught by idempotency, which is the system working (Idempotency Keys).

What changes at 10x and 100x

At 10x traffic, a retry policy that was harmless becomes a meaningful fraction of the dependency's load, and your worst-case amplification factor turns into their capacity planning problem.
At 100x, retry budgets and breakers stop being optional: without them, any dependency wobble is amplified into an outage by your own traffic (Cascading Failure).
Client-side retries interact with autoscaling badly. Retries inflate the observed request rate, the system scales up to serve work that is duplicated, and the bill grows without the useful throughput doing so (Autoscaling a Backend).
At high fan-out, one user request that touches ten dependencies with three attempts each has a worst case of thirty outbound calls, which is a different system from the one on the architecture diagram.

What this costs

Retries raise success rate under transient failure and raise load under sustained failure. You cannot have the first without the second; budgets and breakers are how you bound the second.
Idempotency keys make retries safe and require the dependency to support them, plus key generation, propagation and storage on your side (Idempotency Storage).
A single retrying layer is correct and requires knowing what every SDK and proxy in the path does, which is genuine investigative work.
Not retrying is a legitimate choice for degradable dependencies, and it is often the better one: a fast, clean failure with a fallback beats three attempts and a slow failure.

Where this applies

Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.

GENERALThe retryable/safe distinction, budgets and the amplification dynamic hold for every protocol and language.
PROTOCOL-SPECIFICHTTP gives you status codes and Retry-After to classify with. Over a raw socket or a database protocol you get connection errors and vendor error codes instead, and the classification table has to be built by hand from the driver's documentation.
SCALE-SPECIFICAt low traffic, naive retries are harmless because your amplification is negligible against the dependency's capacity. The feedback loop only becomes self-sustaining once your retried load is a significant fraction of what the dependency can serve — which is why this fails suddenly rather than gradually.

Where the depth lives

This domain teaches the application-side mechanism and hands the rest off.

API Designretryability retries-and-timeouts idempotency-keys idempotency-vs-deduplication error-model

Architecturereliability-patterns circuit-breaker idempotency

Performanceretry-storms tail-latency saturation

Concurrencythundering-herd deadlines-vs-timeouts

Agentictool-errors-retries-timeouts

Domains that do not exist yet

— Distributed Systems — the impossibility of distinguishing a lost request from a lost response, and why that forces idempotency into the application layer.