Idempotency in Backends
Doing the same thing twice must produce the same result as doing it once — because in a network, twice is not optional.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The client sent POST /payments, timed out, and sent it again. Did we just charge the customer twice?
A customer taps "Pay". Their connection is bad and the app retries. They must be charged once, and they must see a confirmation, not an error.
The handler charges the card and creates the payment row. It is a POST, so it creates something — that is what POST means. Retrying is the client's decision and the client should not retry a POST.
The client library retries on timeout by default, because most HTTP clients do. Nobody wrote that behaviour and nobody can turn it off across every SDK, mobile OS and proxy in the path.
- The client library retries on timeout by default, because most HTTP clients do. Nobody wrote that behaviour and nobody can turn it off across every SDK, mobile OS and proxy in the path.
- A timeout tells the client nothing about what happened on the server. The request may have been rejected at the proxy, may be running now, may have committed and lost the response. Three states, one symptom.
- The card is charged, the connection drops before the 201 is written, the client retries, and the card is charged again. Nothing in your logs looks like an error — you have two successful payments (The Request Lifecycle).
- The user, seeing an error, presses "Pay" again themselves. Now the retry is a human one, and it arrives while the first is still executing.
- A load balancer with retry-on-5xx enabled replays the request to a second instance. You did not write that either.
What is actually happening
- Idempotency is a property of an operation: applying it N times has the same observable effect as applying it once. It is not a property of an HTTP method, and it is not a synonym for "safe".
GET,PUTandDELETEare *specified* as idempotent;POSTis not. But specification is a promise about semantics, not a mechanism — aPUThandler that appends to an audit table on every call is not idempotent, whatever the RFC says (Status Codes From the Server's Side).- The reason retries are unavoidable is that the network cannot distinguish "the request was lost" from "the response was lost". Both look like silence. A client that never retries loses work; a client that retries needs the server to be safe (At-Least-Once Delivery).
- Making an operation idempotent means giving repeated executions a way to recognise each other. That requires an identifier that is the same across attempts and different across intents — which the request body alone is not, because a customer may legitimately pay the same amount twice (Idempotency Keys).
- There are three distinct strategies, and confusing them is the source of most half-working implementations: make the operation naturally idempotent (set a value rather than increment it), give it a natural key (a unique constraint on something the domain already has), or add an explicit idempotency key carried by the client.
- "Retryable" and "safe to retry" are different claims. Retryable means the error class suggests trying again might succeed. Safe to retry means the operation is idempotent. A 503 is retryable; whether it is safe to retry depends entirely on your handler (Retryability: Telling Clients What To Do Next).
A timeout is three different outcomes wearing one costume
When a client's request times out, exactly one of three things happened, and the client cannot tell which. The request never arrived. The request arrived and is still executing. The request arrived, completed, committed, and the response was lost on the way back.
In the first case, retrying is necessary. In the third, retrying duplicates the effect. In the second, retrying creates two concurrent executions of the same intent — the worst of the three, because the two attempts can interleave.
No amount of client-side cleverness distinguishes them, because the information simply is not available on that side of the wire. That is the whole argument for idempotency: the only participant who can tell the difference is the server, and it can only do so if the client hands it something that identifies the attempt.
Three ways to be idempotent, and how to choose
Reaching for an idempotency-key table on every endpoint is over-engineering; assuming PUT handles it is under-engineering. The choice is determined by whether the operation already carries something that identifies its intent.
The cheapest strategy is to change the operation so that repetition is harmless — assert a state rather than apply a delta. SET status = 'paid' is idempotent; balance = balance + 100 is not. Where the domain permits this framing, it needs no extra storage and no client cooperation.
The next cheapest is a natural key: something the domain already guarantees unique, enforced by a database constraint. One shipment per order, one invoice per billing period, one row per (user_id, day). Here the database does the deduplication for free and no key needs to travel over the wire (Database Constraints).
Only when neither applies — when two identical requests can legitimately both be intended, which is exactly the payments case — do you need an explicit key generated by the client.
Does the operation already carry something that distinguishes intent from repetition?
when The operation sets state to a value the client already knows: status changes, profile updates, feature flags.
cost Constrains the API shape toward PUT-like assertions; loses the ability to express "add" without reading first.
when The domain already forbids duplicates: one refund per charge, one invoice per period, one vote per user per poll.
cost The constraint must exist in the database, and its violation must be handled as success rather than as an error.
when Two identical requests can both be legitimate: payments, transfers, message sends, order placement.
cost Client cooperation, a key store, expiry policy, scoping rules, and a race to close on concurrent use (Idempotency Storage).
when No state changes and no side effects, including no audit rows or counters.
cost None, provided "no side effects" is actually true. Check the logging and analytics paths before believing it.
Idempotent effect, not idempotent code path
The subtlest failure is an endpoint that is genuinely idempotent in its primary write and not in everything else it does. The payment row is deduplicated correctly. The confirmation email is sent from an event handler with no key. The analytics counter increments. The audit log appends. The customer receives one charge and three emails.
The test that catches this is not "does the second call return 200". It is: run the operation twice and diff the entire observable state — every table, every queue message, every outbound call. Anything that differs between one execution and two is a side effect you have not made idempotent.
This is also why idempotency has to be designed at the level of the operation rather than bolted on at the boundary. A middleware that caches responses by key protects the response; it does not protect anything the handler already did before the cache was consulted.
| Effect | Naive form | Idempotent form | Why the naive form fails |
|---|---|---|---|
| Set a status | UPDATE ... SET status = 'paid' | Unchanged — already idempotent | It is not a failure; this is the easy case |
| Adjust a balance | balance = balance + 100 | Insert a ledger entry keyed by transaction id, sum entries | Repetition compounds the delta (Atomic Operations) |
| Create a record | INSERT INTO payments ... | INSERT ... ON CONFLICT (idempotency_key) DO NOTHING | Two rows, two charges |
| Call a payment API | POST /charges | Send an Idempotency-Key header the provider honours | The provider charges twice; your dedupe cannot reach across |
| Send an email | mailer.send(...) | Record (user_id, template, entity_id) before sending; skip if present | Duplicate email, which the customer notices immediately |
| Enqueue a job | queue.push(job) | Dedupe on a job key, or make the job itself idempotent | The job runs twice (Job Idempotency) |
| Publish an event | bus.publish(event) | Stable event id; consumers deduplicate | Every downstream consumer duplicates its own effect |
| Append an audit row | INSERT INTO audit ... | Usually left non-idempotent, deliberately | Two rows — which may be correct: two attempts genuinely occurred |
How to build it
Most important first.
- Decide, per endpoint, which of the three strategies applies. Naturally idempotent operations need nothing; operations with a natural unique key need a constraint; the rest need an explicit key (The Idempotency Key Flow).
- Treat every state-changing endpoint reachable by an untrusted client as retryable, and therefore as needing an answer to "what happens on the second call?"
- Prefer natural idempotency where the domain allows it.
PUT /users/42 {status: "active"}is idempotent by construction;POST /users/42/activatethat appends a row is not. - Where the effect is external — a card charge, an email — pass your own idempotency key to that provider so the property holds across the boundary you do not control (Idempotency Keys).
- Make the retry return the same response as the original, not a 409. The client asked the same question twice and deserves the same answer; a conflict forces error handling for a case that is not an error (Idempotency Storage).
- Write the test that actually proves it: call the handler twice with the same input and assert on the resulting state, not on the second response code.
What can go wrong
- Idempotency implemented in the handler while a retry hits a different instance — the deduplication state was in process memory (Stateless Services).
- The check and the write being separate operations, so two concurrent retries both pass the check (Duplicate Detection).
- Idempotent at the API boundary and not at the queue boundary, so the job the handler enqueued runs twice anyway (Job Idempotency).
- Partial idempotency: the payment row is deduplicated but the "payment received" email is sent from a separate code path with no key.
- Assuming
PUTis idempotent when the handler also writes an audit row, publishes an event, or increments a counter. The primary effect is idempotent; the side effects are not. - An idempotency mechanism whose failure mode is to fail open — if the key store is unavailable, process anyway — which is precisely the moment duplicates arrive.
- The retry arriving while the original is still executing. This is the defining race of the module and the reason a check-then-write implementation does not work (Duplicate Detection).
- Two different clients — a mobile app and a background sync — issuing the same intent with different keys, which idempotency cannot detect because it looks exactly like two intents.
- The idempotency record and the business effect committing separately, leaving a window where one exists without the other (Idempotency Storage).
- Concurrent retries hitting different instances, so any in-process coordination is invisible to the other (Backend Races).
- Idempotency state is per-caller. A key presented by user A must never return a response computed for user B; scope every key to the authenticated principal (Idempotency Storage).
- Without scoping, a stored response is an information-disclosure primitive: guess a key, receive someone else's payment confirmation.
- An unbounded key namespace is a storage-exhaustion vector. Keys are attacker-supplied strings; bound their length and set expiry.
- Do not make idempotency a bypass for authorization: a replayed key must still belong to a caller who is still permitted to perform the operation.
- "
POSTis not idempotent, so we cannot make it idempotent." The method's specification describes the default expectation; nothing prevents your handler from being idempotent, and payment APIs have been doing exactly that for years. - "Retryable means safe to retry." A 503 says the server thinks a retry might work. Whether that retry duplicates an effect is a property of your handler, not of the status code (Retryability: Telling Clients What To Do Next).
- "We use a queue, so it is handled." A queue moves the duplicate; it does not remove it (At-Least-Once Delivery).
- "Idempotency is the same as deduplication." Deduplication is one implementation. An operation that sets a value rather than incrementing it is idempotent without deduplicating anything.
- "We return 409 on a duplicate, so we are idempotent." You have detected the duplicate and then failed the client. Idempotency means the second call succeeds identically.
Operating it
- A counter of requests by idempotency outcome —
first,replayed,in_progress,conflict. The replay rate is a direct measurement of how often your clients time out. - A rising replay rate on one endpoint usually means that endpoint is slower than the client timeout. It is a latency signal disguised as a correctness metric (Tail Latency: Why p50 Being Fine Does Not Help).
- Log the idempotency key alongside the correlation id, so a customer report of a double charge can be resolved by looking at whether two keys or one arrived (Correlation Ids That Survive Every Hop).
- A reconciliation query that finds duplicate business effects — two payments for the same order within a short window — because that is what the customer actually experiences, and it finds gaps your metrics do not.
- The rate of retries grows super-linearly with load, because load raises latency and latency triggers client timeouts. The endpoint that never needed idempotency at low volume needs it most under stress.
- At 10x, idempotency state becomes a hot write path of its own; at 100x it needs the same design attention as the data it protects (Idempotency Storage).
- Nothing about the *property* changes with scale. What changes is the probability that the race you dismissed happens, and at high enough volume it happens continuously.
- Idempotency adds a write and often a read to every state-changing request, plus a storage system with its own expiry and failure modes. That cost is paid on every request to prevent a fraction of them from duplicating.
- Storing responses to replay them means storing data you have already sent — with the retention and privacy obligations that implies.
- An explicit key pushes work onto clients, who must generate and reuse it correctly. Client-side mistakes become server-side incidents, and you cannot fix them from your side (Idempotency Keys).
- Natural idempotency is the cheapest option and constrains the API design: it favours
PUT-shaped state assertions overPOST-shaped commands, which is not always the right domain model.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe property and the need for it follow from the network, not from any stack. Any system where a response can be lost needs it.
- PROTOCOL-SPECIFICHTTP specifies
GET,PUT,DELETE,HEADandOPTIONSas idempotent andPOST/PATCHas not. That governs what intermediaries may safely retry on their own — some proxies and load balancers will replay an idempotent method automatically and will not replay aPOST— but it constrains nothing about your handler. - SCALE-SPECIFICOn an internal service with one trusted client and low volume, the natural-key strategy is usually sufficient and an explicit key store is overhead. Public APIs with mobile clients need the explicit mechanism from day one, because you cannot change client retry behaviour retroactively.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — why the network cannot distinguish a lost request from a lost response, and what that forces on every protocol built over it.