The question this answers
What makes an operation actually safe to repeat — and why do handlers that "are idempotent" still produce duplicate effects?
For an idempotent operation, the observable state after N ≥ 1 applications equals the state after exactly one, for a stated observer and a stated set of effects. It does not guarantee the same *response* to each application, it does not guarantee anything about interleaving with other operations, and it says nothing about effects outside the enumerated set.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The receiver knows what its own handler does. It does not know how many times the caller has sent this operation, whether another replica is executing it concurrently, or whether an effect it triggered downstream — an event, a webhook, a notification — has already been produced by a previous application. Idempotence is a claim about the *closure* of a handler’s effects, and the handler can only observe the part of that closure inside its own process.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The definition, and the three words that matter
Formally, f(f(x)) = f(x). In practice the useful phrasing is: applying the operation again does not change the observable state. Three words in that sentence carry the weight.
Observable — by whom, and through what interface? A handler that writes the same row twice but appends two audit entries is idempotent to a reader of the row and not idempotent to a reader of the audit log. Both readers are legitimate. You must say which observer the claim is for.
State — not response. An idempotent operation may legitimately return 201 Created the first time and 200 OK the second, or return the same stored response both times. Response equality is a nice property (and what idempotency-key mechanisms provide) but it is not what idempotence means. Conversely, returning identical responses while performing a second effect is the dangerous combination: it looks correct from outside.
Again — over what window? An operation that is idempotent for an hour and then forgets is idempotent for retries that finish within an hour, and not for a replay from a dead-letter queue three days later. That window is a scope decision serious enough to have its own lesson (What Counts as the Same Operation?).
Three ways to get it, in order of cost
Natural idempotence — the operation is absolute rather than relative. SET status = "shipped" is idempotent; INCREMENT attempts is not. PUT /users/42 {name: "Ada"} is idempotent; POST /users {name: "Ada"} is not. This is free and requires no infrastructure, and the first design move for any operation should be to ask whether it can be expressed absolutely. Very often "add 1" can be restated as "record that event E happened", with the count derived.
Conditional idempotence — the operation carries a precondition that only the first application satisfies. UPDATE orders SET status = "paid" WHERE id = 1 AND status = "pending" applies once; the second application matches zero rows. Version numbers and If-Match on an ETag are the same idea at the protocol level. This costs nothing but a column, and it composes with concurrency control: the precondition detects a concurrent change as well as a duplicate.
Synthetic idempotence — the operation is inherently non-repeatable, so you record that it happened under a caller-supplied identity and make the second application a lookup. This is what idempotency keys do. It is the most general and the most expensive: a durable record per operation, retained for a window, with its own availability and its own storage cost. Reach for it when the first two genuinely do not apply, not by default.
| Approach | Example | Cost | Fails when |
|---|---|---|---|
| Natural (absolute)protocol | `SET balance_state = SETTLED` | None | The operation is genuinely relative (money movement) |
| Conditional (CAS)protocol | `… WHERE status = "pending"` | One column | Multiple legal transitions from the same state |
| Synthetic (keyed)assumption | `INSERT INTO ops(key) … ON CONFLICT DO NOTHING` | A durable record per operation | The key expires, or the store is separate from the effect |
1-- The claim and the effect commit together, so a concurrent duplicate2-- cannot slip between a check and a write.3BEGIN;4 INSERT INTO processed_ops (op_key, request_hash)5 VALUES ($1, $2)6 ON CONFLICT (op_key) DO NOTHING;7 -- 0 rows inserted => someone else already claimed this operation.8 -- Skip the effect and return the stored result.9 10 INSERT INTO ledger (op_key, account, delta) VALUES ($1, $3, $4);11COMMIT;12 13-- Contrast with the racy version, which is the common bug:14-- SELECT 1 FROM processed_ops WHERE op_key = $1; -- both attempts: empty15-- ... do the work ...16-- INSERT INTO processed_ops ... -- both attempts: succeedThe effect closure: where handlers stop being idempotent
This is the practical failure that keeps happening to teams who did the work. A handler is written carefully: the write is conditional, the key is claimed atomically, tests pass. Six months later someone adds a line to the same handler that publishes an event, or increments a Prometheus counter, or calls a notification service. The database is still idempotent. The handler is not, and nothing failed a test, because the tests assert on the row.
Enumerate the full effect closure of a handler: rows written, events published, external calls made, metrics emitted, caches invalidated, files created, logs that something downstream parses. Each one is either idempotent, covered by the same atomic claim, or a duplicate waiting to happen. The handler is idempotent only if every element of that closure is — it is a conjunction, and one new line can break it.
The structural defence is to make the claim gate everything: check the claim first, and if the operation was already applied, return the stored result without executing any of the effects. That works only if the claim is the first thing the handler does and the handler has a single exit path for the replay case — which is why frameworks that wrap the whole handler in an idempotency middleware are more robust than per-write checks.
The second defence is to move effects out of the handler and into the outbox pattern: publish by writing a row in the same transaction as the claim. Then the event inherits the claim’s idempotence, and a duplicate handler invocation produces no second event because it produces no second row.
- Row writes — idempotent via absolute set, CAS, or unique key.
- Published events — idempotent only if written through the outbox inside the claiming transaction.
- Outbound HTTP calls — need their own key, propagated deterministically from the operation’s key.
- Counters and metrics — inherently relative; a duplicate invocation over-counts, and this corrupts billing if the counter is billable.
- Emails, notifications, webhooks — not idempotent and not compensable (A Refund Is Not a Rollback); gate them behind the claim or move them after it.
- Cache writes — usually idempotent, but a cache *invalidation broadcast* is an event and follows the event rules.
Idempotent is not the same as commutative
Retries do not only duplicate; they reorder. Two operations that are each individually idempotent can still produce different results depending on the order they land in, and in a distributed system you do not control that order.
SET status = shipped and SET status = cancelled are both idempotent. Applied in one order the parcel ships; in the other it does not. Two replicas that receive them in different orders diverge permanently, and neither is "wrong" locally. Idempotence protects you from duplication; commutativity is what protects you from reordering, and they are different properties needing different mechanisms.
Where you can, choose operations that are both: relative counters (+= 1 / -= 1) commute and, with a per-operation identity to dedupe, are effectively idempotent too — which is exactly the construction behind a counter CRDT. Where you cannot, you need an ordering mechanism: a version number, a sequence per producer, a causal token, or a merge rule. That is Happens-Before: The Only Ordering You Actually Have and What "Eventually Converges" Actually Requires territory, and the reason those lessons sit in this domain rather than being an implementation detail.
| Operation | Idempotent? | Commutative? | What you still need |
|---|---|---|---|
| `SET name = "Ada"`protocol | Yes | No | A version or timestamp to order concurrent sets |
| `balance += 10`protocol | No | Yes | A per-operation identity to dedupe |
| `add_to_set(tag)`protocol | Yes | Yes | Nothing — safe under duplication and reordering |
| `send_email()`protocol | No | No | A claim, and placement after the pivot |
| `SET status = X WHERE status = Y`assumption | Yes | Partially | A state machine that rejects illegal transitions |
Key points
- Idempotent means the observable state after many applications equals the state after one — for a stated observer and a stated set of effects.
- It is about state, not response; identical responses while performing a second effect is the worst case, because it looks correct.
- Get it naturally (absolute writes), conditionally (compare-and-set), or synthetically (a claimed key) — in that order of preference.
- The claim must be atomic with the effect. A lookup followed by a write is a race two concurrent retries will win.
- A handler is idempotent only if *every* effect it produces is — one added line publishing an event breaks it silently.
- Idempotence protects against duplication; commutativity protects against reordering. You often need both.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • Enumerate the handler’s full effect closure: writes, events, outbound calls, metrics, notifications.
- • For each effect, choose absolute, conditional or keyed idempotence, preferring the cheapest that works.
- • Claim the operation identity atomically — a conditional insert or unique constraint in the same store and transaction as the primary effect.
- • If the claim fails, the operation was already applied: return the stored result and execute none of the effects.
- • Route event publication through an outbox row written in the claiming transaction, so events inherit the claim.
- • Give outbound calls a key deterministically derived from the operation key, so retries at any level converge on the same identity.
- • Record the result against the claim so a later replay returns the same answer rather than re-executing.
- • A concurrent duplicate passes a non-atomic check and both attempts execute.
- • The claim record lives in a different store from the effect, so a crash between them leaves them disagreeing.
- • A new effect is added to the handler and is not covered by the claim.
- • A relative operation (a counter, a balance delta) is treated as idempotent because the surrounding row write is.
- • Two idempotent operations arrive in different orders at different replicas and the replicas diverge.
- • The claim expires and a late replay is treated as a new operation (What Counts as the Same Operation?).
- • The stored response is returned for a request whose body differs from the original, silently dropping a real second operation.
- • Duplicate downstream events with correct database rows: the operator sees one order row, two
OrderCreatedevents, and two shipments. The database team’s idempotency tests all pass because they assert on rows. - • Billing over-count: an idempotent write paired with a non-idempotent usage counter means customers are charged for retried requests. The discrepancy is proportional to the dependency’s timeout rate and is discovered by a customer, not by monitoring.
- • Concurrent-duplicate window under load: duplicates appear only at high concurrency, because that is when two attempts overlap enough to both pass a read-then-write check. Unreproducible in staging.
- • Divergent replicas after a reorder: two nodes show different final statuses for the same entity, both consistent with their own histories, and no error was raised anywhere (Two Writes, No Order, One Answer Required).
- • Silent second effect behind an identical response: the caller sees the same 200 and the same body on the retry, so nothing looks wrong, while a second notification went out.
- • The atomic claim is the coordination point, and keeping it in the same store as the effect makes it nearly free — it rides an existing transaction rather than adding a round trip.
- • Moving the claim to a separate store (a cache, a dedicated dedup service) adds a dependency to every write and creates a window where claim and effect disagree.
- • Natural and conditional idempotence require no coordination at all, which is why they are strictly preferable when available.
- • Ordering, unlike deduplication, cannot be solved locally: preventing divergence under reordering needs versions, sequences or causal metadata carried between nodes (Version Vectors: Making the Conflict Visible).
- • A claimed-but-uncommitted operation vanishes on crash, so the retry proceeds normally — the correct behaviour, and the reason the claim must share the effect’s transaction.
- • A claim committed with its effect survives any subsequent failure, so replays remain safe indefinitely within the retention window.
- • Effects outside the claiming transaction — an email already sent, an HTTP call already made — are unaffected by rollback and may duplicate.
- • Under partition, replicas may each accept the same operation independently; idempotence per replica does not prevent two effects across replicas unless the claim is globally scoped.
- • Detect: instrument the claim path so replays are counted. A replay rate of zero usually means the mechanism is not working, not that there are no retries.
- • Contain: gate the entire handler behind the claim, so a replay short-circuits before any effect rather than being handled per-effect.
- • Recover: for handlers already found non-idempotent, dedupe downstream by natural key rather than re-running the operation (Deduplication: Bounded Memory Against an Unbounded Stream).
- • Reconcile: compare counts of operations against counts of each effect they should produce; a ratio above one localises which effect escaped the claim.
- • Verify: add a test that invokes the handler twice with identical input and asserts on the *full* effect closure — rows, events, outbound calls, metrics — not only on the row.
- • Replay rate at the claim: how often an operation was recognised as already applied. Normally small and non-zero; zero is suspicious.
- • Ratio of effects to operations for each effect type — events published per operation, emails sent per operation. Any ratio above one is a broken closure.
- • Claim conflicts resolved concurrently versus sequentially, which distinguishes overlapping retries from late ones.
- • Handler execution count versus distinct operation count, which is the direct measurement of duplicate processing.
- • Divergence checks between replicas for entities updated by absolute writes, which catches reordering rather than duplication.
- • Any write path reachable by a retry — which, after The Retry Is a Decision, Not a Reflex, is every write path.
- • Message consumers, where at-least-once delivery makes duplicates routine rather than exceptional (Where You Put the Acknowledgement Decides Everything).
- • Webhook receivers, where the sender’s retry policy is not yours to control.
- • Anywhere an operation can be re-driven by an operator, a replay tool, or a dead-letter queue drain days after the fact.
- • When synthetic keying is applied to operations that were already naturally idempotent, adding a store and a dependency for nothing.
- • When the claim store is separate from the effect store, so every write gains a network hop and a new failure mode.
- • When "idempotent" is asserted for a handler nobody has enumerated the effects of, giving false confidence that removes the pressure to dedupe downstream.
- • When idempotence is treated as sufficient and ordering is ignored, so the system stops duplicating and starts diverging.
- • Redesign the operation to be naturally idempotent — absolute rather than relative — which removes the problem instead of managing it.
- • Dedupe at the sink by natural business key with an upsert, so no separate dedup state exists at all (Deduplication: Bounded Memory Against an Unbounded Stream).
- • Use optimistic concurrency with a version check, which handles duplicates and concurrent modification with one mechanism.
- • Accept duplicates and reconcile downstream, where the effect is cheap and the reconciliation is easy.
- • Move the effect behind a queue keyed by the operation identity, so the queue collapses duplicates before the handler sees them.
Apply the handler N times and count what the world has
| Implementation | Observer | After 3 applications | Verdict | |
|---|---|---|---|---|
| write the order rowprotocol | INSERT … ON CONFLICT (order_id) DO UPDATE | the database | 1× — the row is a function of the request, not of how many times it arrived | idempotent |
| increment the usage countertypical | UPDATE accounts SET used = used + 1 | the invoice | 3× — read-modify-write: every application adds one, and the customer is billed for your retries | not idempotent |
| append an audit rowtypical | INSERT INTO audit … | the audit log | 3× — append is never idempotent — that is what append means | not idempotent |
| publish OrderCreatedtypical | broker.publish(OrderCreated) | every downstream consumer | 3× — a fresh message id per call: two events, two shipments | not idempotent |
| send the confirmation emailtypical | mailer.send(...) | the customer's inbox | 3× — you cannot un-inform someone; two emails is two emails | not idempotent |
What people believe, and what is true
Idempotent means the operation has no side effects.
That is *safe* (or nullipotent), a stronger and different property. An idempotent operation may have large side effects — it just does not add more of them when repeated.
Idempotent means the same response every time.
It means the same state. Response equality is a separate, useful property that keyed idempotency mechanisms provide on top.
PUT is idempotent, so my PUT handler is.
The method’s semantics are a contract you can violate. If the handler appends an audit row or emits an event per call, it is not idempotent regardless of the verb.
Our writes are idempotent, so duplicate messages are harmless.
Only if the write is the only effect. Events, emails, metrics and outbound calls in the same handler each need their own answer.
We check whether the key exists before doing the work, so we are safe.
Two concurrent retries both check, both find nothing, and both proceed. The check and the effect must be one atomic operation.
If every operation is idempotent, the system is correct under retries.
It is correct under duplication. Reordering is a separate hazard, and idempotence does nothing about it.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Doing it twice should leave the world the same as doing it once. Prefer writes that set a value over writes that change it by an amount.
Practical
List everything a handler does — rows, events, HTTP calls, metrics, emails. Make the claim atomic and put it first, so a replay returns the stored result without running any of them. Publish events through an outbox in the same transaction. Test by invoking the handler twice and asserting on every effect, not just the row.
Advanced
Idempotence and commutativity are orthogonal, and retries introduce both duplication and reordering. Operations that are both — set union, counters with per-operation identity, last-writer-wins with a proper version — are the ones that survive an asynchronous network without coordination. That is the design space CRDTs formalise, and choosing operations from it is how you avoid needing ordering guarantees you would otherwise have to buy with coordination.
Apply it
- 🔧 Take a handler you believe is idempotent, enumerate its full effect closure, and find the effect that is not covered.
- 🔧 Write a test harness that invokes a handler twice and asserts on rows, published events, outbound HTTP calls and metrics. Run it against three handlers in your codebase.
- 🔧 Convert a relative counter into a set of identified events with a derived count, and show that duplicates no longer over-count.
- ⚡ Customers report being charged the correct amount but receiving two confirmation emails, occasionally. Where is the bug?
- ⚡ Usage-based billing over-charges by an amount that correlates with the payment provider’s latency. Explain the chain.
- 💬 Define idempotent precisely. What are the two qualifiers your definition needs?
- 💬 Give three ways to make a non-idempotent operation safe to repeat, and rank them by cost.
- 💬 Why is "check whether the key exists, then do the work" wrong?
- 💬 A handler writes a row, publishes an event and increments a usage counter. Which parts are idempotent and what would you change?
- 💬 Two nodes each receive the same two idempotent updates in different orders. What happens, and does idempotence help?