Reliabilityretriestimeoutsbackoffjitterretry-afterretry budget

Retries and Timeouts as Contract Guidance

A timeout is not a failure — it is the absence of an answer. The contract owes clients the missing half of their retry loop: what is retryable, how long to wait, how to back off, and what the server will do to protect itself when everyone retries at once.

Follow the failure

Frame the contract

API design starts with a consumer, a design question and a guarantee — never with a URL.

Design question
The request failed or timed out — should this client try again, when, how many times, and does the server survive everyone deciding "yes"?
Consumers
Every SDK author choosing retry defaults, every integration with a runbook, every service with your API in its critical path — and, adversarially, all of them at once during your partial outage, when their collective retry policy becomes your load profile.
The promise
For every failure mode the API can produce, the contract answers: retryable or not, after how long, with what ceiling — and the server enforces the same rules it publishes, so well-behaved clients are never out-competed by aggressive ones.
Requirement→Consumers→Resource Model→Style→Contract→Validation→Authorization→Errors→Idempotency→Pagination→Versioning→Observability→Evolution→Trade-offs

Timeouts: the answer that never came

Every failure a client sees is one of two kinds, and the difference governs everything downstream. A definitive failure — 400, 403, 422 — is an answer: the server judged the request and rejected it, and retrying the identical request is pure waste. A timeout is *no answer*: the request may have never arrived, arrived and failed, or arrived and fully succeeded with the response lost in flight. The client cannot distinguish these, which is why retrying a timed-out mutation is only safe under Idempotency: Surviving the Retry — the timeout is the precise scenario idempotency keys exist for.

The client's timeout is a contract-adjacent number, and the API should inform it. Publish latency expectations per operation class (reads p99 < 500ms, writes p99 < 2s) so clients can set timeouts above real p99 instead of guessing: a timeout below server p99 manufactures failures out of successes — the client abandons requests the server completes, then retries them, doubling load and duplicating effects. And keep your own upstream budget shorter than what you expect clients to wait, so you can return a real 504 with guidance instead of letting the client's deadline fire first and learn nothing (Status Codes Clients Can Branch On).

What a client can conclude from each failure class
Response         Meaning                       Retry the same request?
─────────────    ──────────────────────────    ──────────────────────────
400 / 422        judged and rejected           no — fix the request first
401 / 403        identity/permission verdict   no — fix credentials, not timing
404              target absent                 no — unless eventual visibility is documented
409 / 412        state conflict                not blindly — refetch, then retry ([[optimistic-concurrency]])
429              you, specifically, slow down  yes, after Retry-After — reduce rate
500              server failed to process      maybe — per documented retryability
502 / 503        dependency/capacity           yes, with backoff; honor Retry-After
504 / timeout    NO ANSWER — outcome unknown   only if idempotent; else check-then-retry

The retry loop the contract should dictate

Left to defaults, every client invents its own loop, and the bad ones share a shape: immediate retry, fixed interval, unlimited attempts. Immediate retry hits a server still in the same failing state. Fixed intervals synchronize clients that failed together into waves that arrive together. Unlimited attempts convert a five-minute incident into an hour of self-sustained load. The fix is a published recipe: exponential backoff (double the wait per attempt: 1s, 2s, 4s, 8s…), full jitter (each wait is random(0, base × 2^attempt) — the randomness is what breaks up the waves), a cap on both single-wait and total attempts, and a retry budget (retries as a bounded fraction of a client's traffic, ~10–20%, so retry load can never dominate first-try load).

Server-sent timing beats client math wherever you can provide it: Retry-After on 429 and 503 is you telling clients exactly when capacity returns, which no backoff formula can know (The Rate-Limit Contract). And publish where retries should *stop* mattering: after the budget is spent, the correct client behavior is to surface the failure — to a queue, a dead-letter, a human — not to keep pounding. Clients with a circuit-breaking layer (Circuit Breaker) formalize exactly this.

One structural rule prevents the worst emergent behavior: retry at one layer. When the SDK retries 3×, the calling service retries 3×, and its caller retries 3×, one user action becomes 27 requests — retry amplification that turns a hiccup into an outage. The contract should say where the retry responsibility lives (usually: the SDK, with everything above it failing fast), because no single team can see the multiplication from inside its own layer.

The loop the docs should print — capped, jittered, budgeted, idempotency-aware
1MAX_ATTEMPTS = 4        # 1 try + 3 retries
2BASE = 1s               # grows 1s → 2s → 4s
3CAP = 30s
4 
5attempt(req):
6  for n in 0 .. MAX_ATTEMPTS-1:
7    resp = send(req, timeout = op.p99 * 2)
8    if resp is definitive-failure:  return resp    # 4xx: answer, not obstacle
9    if resp is success:             return resp
10    if not budget.allow():          return resp    # retries ≤ 20% of traffic
11    if resp.retry_after:            wait = resp.retry_after   # server knows best
12    else:                           wait = random(0, min(CAP, BASE * 2**n))
13    if req.mutating and not req.idempotency_key:
14      return resp                                  # unsafe to retry blind
15    sleep(wait)
16  return last_resp                                 # budget spent: surface, don't pound

Retry behavior is part of the API surface

Everything above becomes real only when it is written into the contract per failure mode — a retryability column in the error table, not a paragraph of general advice (Retryability: Telling Clients What To Do Next covers the response-side signaling; An Error Taxonomy Clients Can Branch On the classification it hangs on). The reference APIs ship this as SDK behavior, which is the strongest form of documentation: Stripe's and AWS's SDKs retry idempotent operations with jittered backoff by default, which means the *median* integration is well-behaved without its author ever reading the retry section.

The server side must then enforce what it published, because during an incident you will meet the clients who ignored it. Rate limits that apply to retry storms (The Rate-Limit Contract), load shedding that answers excess load with fast 503 + Retry-After instead of slow timeouts, and idempotency machinery sized for retry bursts (Idempotency Keys: The Mechanism) are the enforcement half. A contract that politely requests backoff while the server melts under whoever ignores it is a contract that punishes exactly the compliant.

  • Per-error retryability — every documented error code carries retryable: yes / no / after-delay (The Error Model: Structure Over Apology).
  • `Retry-After` on 429 and 503 — server-known timing beats client-side guessing; clients must be told to honor it.
  • Published latency classes — client timeouts should clear server p99 with margin; state the p99 per operation class.
  • Idempotency prerequisites — the docs must say plainly: do not retry mutations without a key (Idempotency Keys: The Mechanism).
  • SDK defaults as policy — encode the loop in official SDKs so the default integration is the well-behaved one (SDK Design: The Contract's User Interface).
  • One retrying layer — declare where retries live in the stack; layers above it fail fast.

Key points

  • A 4xx is an answer; a timeout is the absence of one — the outcome is unknown, and only idempotency makes retrying it safe.
  • Client timeouts below server p99 manufacture failures: the client abandons and retries work the server completed.
  • The safe loop is exponential backoff + full jitter + attempt cap + retry budget; jitter is what prevents synchronized retry waves.
  • Retry-After from the server beats any client formula — send it on 429/503 and require clients to honor it.
  • Retry at one declared layer; stacked 3× retries compound into 27× amplification that turns hiccups into outages.
  • Publish retryability per error code and enforce it server-side — unenforced politeness punishes compliant clients.

Follow the failure

How the contract fails or gets misused, hop by hop — and what it costs when it completes.

  1. 1
    Team → docs: ships an error table with no retryability column; SDK has no retry logic, so each integrator writes their own.
  2. 2
    Integrators → clients: most write immediate-retry-3× loops; a few write while-true loops; none add jitter.
  3. 3
    Dependency → API: a downstream slows down; p99 crosses client timeouts and thousands of in-flight requests become "failures" simultaneously.
  4. 4
    Clients → API: synchronized retries triple the load on an already-degraded system; latency rises further, breeding more timeouts and more retries.
  5. 5
    Team → incident: the original blip lasted 90 seconds; the retry storm sustains the outage for an hour, and the postmortem blames "client behavior" the contract never specified.
What breaks
  • Retry storms convert partial degradation into full outage — the failure amplifies through exactly the mechanism meant to handle it.
  • Non-idempotent mutations retried on timeout duplicate charges, emails and orders precisely during incidents, when reconciliation capacity is lowest.
  • Compliant integrations starve: clients that back off yield capacity to clients that hammer, so the contract's good citizens get the worst experience.

Design, observe, evolve

A contract decision is incomplete until you know how you would notice it failing and how it changes later.

Design the contract
  • • Document retryability per error code and ship the backoff-jitter-budget loop as the default in official SDKs — policy as code, not prose.
  • • Send `Retry-After` on 429 and 503, sized from real recovery estimates, and honor it in your own internal clients.
  • • State latency expectations per operation class so client timeouts can be set above real p99 with margin.
  • • Enforce the published behavior: rate-limit retry bursts, shed load with fast 503s instead of slow timeouts, and require idempotency keys where blind retries would mutate.
Observe in production
  • • Track retry rate as a first-class metric (attempt-number header or key-replay rate): rising retry share is degradation visible before error rates move.
  • • Watch inter-arrival patterns of identical requests during incidents — evenly spaced spikes mean fixed-interval clients, your next doc-and-SDK fix.
  • • Measure the gap between client-abandoned requests (connection closed) and server completion; a widening gap means client timeouts are set below your real latency.
Evolve without breaking
  • • Retryability can loosen safely (no → after-delay) as operations gain idempotency; tightening (retryable → not) breaks client loops built on the promise and needs a deprecation path.
  • • SDK retry defaults can be tuned per release — one of the few levers that upgrades the whole ecosystem's behavior without any integrator editing code.
  • • Adding `Retry-After` where it was absent is additive; clients that ignore it are no worse off, clients that honor it immediately behave better.
What it costs
  • • Backoff trades recovery speed for stability: after a blip, jittered clients return over tens of seconds rather than instantly — the p99 of recovery is the price of not re-toppling the server.
  • • Retry budgets mean some retryable failures are surfaced to users who would have succeeded on attempt two; the budget protects the fleet at the cost of individual requests.
  • • Publishing latency classes turns them into commitments — a regression in p99 is now a contract conversation, not just a performance ticket.

Misconceptions

Claim
“Retries are the client's business — the API just serves requests.”
Reality
The sum of client retry decisions is the server's load profile during every incident. An API that doesn't specify retry behavior has still chosen one: whatever the worst integrator wrote. Specifying and enforcing it is self-defense.
Claim
“More retries mean more reliability.”
Reality
Retries only help failures that are transient and load-independent. Against overload they are anti-reliability: each round of retries deepens the condition that caused the failures, which is how 90-second blips become hour-long outages.
Claim
“A timeout means the request failed, so retrying is like retrying any error.”
Reality
A timeout means the outcome is unknown — the work may have fully succeeded. Retrying it without idempotency is choosing possible duplication, not repeating a no-op. The distinction is the entire reason Idempotency Keys: The Mechanism exist.

Apply it

⚡ Break: Timeout, Then Retry
← There Is No Transaction Across APIs