Idempotency Keys: The Mechanism
A client-generated key turns "did my POST land?" into a question the server can answer: check the store, replay the saved result or process and save. The hard parts are scope, expiry, parameter mismatches, and two identical requests in flight at once.
Frame the contract
API design starts with a consumer, a design question and a guarantee — never with a URL.
The flow: check, process, store, replay
The client generates a unique key per *logical operation* — one checkout attempt, one refund — typically a UUID minted when the user intent is created, not when the HTTP request is sent. That distinction is the whole trick: the retry reuses the key because it is the same intent, while a genuinely new attempt mints a new key. A key generated per HTTP request protects nothing.
Server-side, the handler brackets the work: look the key up; if a stored response exists, return it verbatim without touching the domain; otherwise reserve the key, execute, store the outcome against the key, and return it. The reservation step matters — key insertion and the duplicate check must be one atomic operation (a unique-constraint insert, not a read-then-write), or two concurrent requests both pass the check and you have reimplemented the bug with extra steps.
The four hard edges
The happy path fits on a napkin; the contract earns its keep on the edges. Each edge below is a question your documentation must answer, because a client will hit every one of them.
The nastiest is the in-flight duplicate: the retry arrives while the original is still executing — common, because the retry was *triggered by* the original being slow. Replaying is impossible (there is no stored result yet) and re-executing is the bug. The two honest contracts are: return 409 with a retry hint, or hold the second request until the first completes and then replay. Stripe does the former; either is fine, silence is not.
- Scope — keys are unique per endpoint per principal, not globally: client A's
key-1must not collide with client B's, and a key used on/paymentsshould not block one on/refunds. Scope by (credential, route, key). - Expiry — the store cannot grow forever. 24–72h covers real retry windows (Stripe uses 24h); document it, because a replay after expiry executes again. Expiry shorter than your clients' longest retry/queue delay is a latent double-charge.
- Parameter mismatch — same key, different body means a client bug (reused UUID) or an attack. Never execute, never replay the old result as if it matched: reject with
422and a specific error code. Detect it by storing a request-body hash with the key. - Replay of failures — if the first attempt returned
402 card_declined, the retry with the same key gets the stored402, not a fresh charge attempt. A new attempt is a new intent and needs a new key. Document this or clients will "retry" declines forever. - In-flight duplicates —
409 ConflictwithRetry-After, or block-and-replay. Pick one, write it down.
What the wire looks like
The convention is an Idempotency-Key request header, echoed metadata in the response, and a signal for whether this response was freshly computed or replayed. The replay signal costs one header and saves hours of debugging — a client seeing Idempotency-Replayed: true on a "slow" request understands instantly that its first attempt succeeded.
Make the key required on the endpoints that need protection, not optional. An optional key protects only the clients that already understood the problem — the ones that were going to be careful anyway. A missing key on POST /payments should be a 400, which turns the contract into a forcing function.
POST /payments HTTP/1.1
Idempotency-Key: 0b1de8e2-…-4a7c
Content-Type: application/json
{ "amount": 4999, "currency": "EUR", "source": "card_abc" }HTTP/1.1 201 Created
Idempotency-Key: 0b1de8e2-…-4a7c
Idempotency-Replayed: true
Request-Id: req_7g2…
{ "id": "pay_8f3…", "status": "processing" }
# Same id, same body as the lost first response.
# Exactly one charge exists.Implementation shape: where teams get it wrong
The recurring implementation bug is checking and inserting the key in two steps, or storing the key only *after* the operation succeeds. Both leave a window where concurrent duplicates execute twice. The key row must be claimed atomically before the work starts, and the outcome written to it after — which also means deciding what happens if the process crashes between the two (a key stuck "in flight" needs a timeout after which it is retryable, ideally tied to whether the underlying work is verifiable).
Store the response you intend to replay — status, body, relevant headers — not just a "seen" flag. A flag can only tell you a duplicate happened; it cannot answer the client, so implementations with flags end up re-querying domain state to reconstruct a response, which drifts from what the first caller saw.
1handle(req):2 if store.exists(req.key): # step 13 return store.get(req.key)4 result = charge(req.body) # two concurrent requests5 store.put(req.key, result) # both reach here6 return result7# Two timeouts + one retry = both pass exists(),8# both charge. The mechanism made it look safe.1handle(req):2 claim = store.insertIfAbsent( # unique constraint3 key = (principal, route, req.key),4 bodyHash = hash(req.body))5 if claim.existing:6 if claim.bodyHash != hash(req.body): return 422 key_reuse7 if claim.completed: return claim.response # replay8 return 409 retry_after(2s) # in flight9 result = charge(req.body)10 store.complete(claim, response = result)11 return resultThe unique-constraint insert makes "have I seen this?" and "I am handling this" one atomic step, so concurrent duplicates serialize. Storing the full response makes the replay byte-faithful; the body hash catches key reuse.
Key points
- The key identifies a logical intent, minted when the intent is created — a key generated per HTTP request protects nothing.
- Claim the key atomically (unique-constraint insert) before executing; check-then-act reintroduces the race under concurrency.
- Store and replay the full outcome — including failures. A retry of a declined charge gets the stored decline, not a new attempt.
- Scope keys per principal and endpoint; expire them on a documented window longer than any client retry horizon.
- Same key + different body = reject with 422; in-flight duplicate = 409 with Retry-After or block-and-replay. Both cases must be in the docs.
- Make the key required on protected endpoints — optional keys only protect the already-careful.
Idempotency Key Flow
Change the contract and observe which guarantee moves.
—
Follow the failure
How the contract fails or gets misused, hop by hop — and what it costs when it completes.
- 1Team → API: adds an optional
Idempotency-Keyheader, checks it withSELECTthenINSERT, stores only a seen-flag. - 2Client → API: a slow charge triggers the SDK's retry while the original is still executing.
- 3API → store: both requests pass the
SELECT, both execute; the customer is double-charged despite "having idempotency". - 4Second client → API: a different integration never sends the optional key at all and duplicates silently on every timeout.
- 5Team → postmortem: the mechanism gets blamed as "not working"; the fixes — atomic claim, required key, stored response — were contract decisions all along.
- Concurrent duplicates slip through a non-atomic check exactly under the conditions that cause retries — load and slowness.
- Replays after key expiry or from flag-only stores return different data than the original response, corrupting client-side state.
- Key reuse across different bodies, if silently accepted, makes the API return a stored response for a request it never executed — data that is simply wrong.
Design, observe, evolve
A contract decision is incomplete until you know how you would notice it failing and how it changes later.
- • Require the key on every endpoint whose duplicate consequence is money or an unrecallable side effect; reject missing keys with a `400` and a helpful error code.
- • Persist (principal, route, key, body-hash, status, response) with an atomic claim step; define the in-flight answer (`409` + `Retry-After` is the simple one) and the crash-recovery timeout.
- • Document scope, expiry window, mismatch behavior and failure replay in the endpoint docs — the mechanism is only as safe as the client's understanding of it.
- • Provide the replay indicator header so clients and support can distinguish first execution from replay at a glance.
- • Track replay rate per endpoint: it is your measured retry/ambiguity rate, and a spike localizes a network or latency problem.
- • Alert on `422` key-mismatch responses — each one is a client minting keys wrong, which means it is one refactor away from unprotected retries.
- • Watch the age distribution of replayed keys against the expiry window; replays near the boundary mean the window is too short for someone's queue.
- • Introduce the header as optional-but-honored, instrument adoption per consumer, then enforce required per endpoint with notice — a compatible hardening path.
- • Expiry windows and in-flight semantics can loosen safely (longer window, block-and-replay instead of 409) but tightening them is behavioral breakage for clients built against the old answer.
- • A durable store sits on the hot path of your most valuable writes: it must be as available as the API, and its latency is added to every protected call.
- • Storing full responses for 24–72h at payment volume is real storage and a data-retention question (response bodies may hold PII).
- • Required keys push work onto every client — key generation, persistence across app restarts so the retry can reuse it — which is exactly the point, and still a cost.