Idempotency by Design
Idempotency is a property of a signature, not a feature you add later. createPayment(commandId, amount) has the id in it because the failure model put it there — and no discipline around createPayment(amount) can substitute.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind survives until the requirement changes.
What has to be in this operation's signature for repeating it to be safe?
A retry on a flaky network created a second payment. The proposed fix is "deduplicate in the payment service". Nobody has said what makes two payments the same payment.
Deduplicate on the server by hashing the request: same amount, same method, same customer, within five minutes — treat it as the same payment.
It is wrong in both directions and there is no window that fixes it. Too short and a slow retry creates a duplicate; too long and a customer legitimately buying the same coffee twice gets one payment. Content is not identity (Value Objects).
- It is wrong in both directions and there is no window that fixes it. Too short and a slow retry creates a duplicate; too long and a customer legitimately buying the same coffee twice gets one payment. Content is not identity (Value Objects).
- It puts the identity decision on the wrong side. The server is guessing at intent that the client knows for certain — and a guess about intent is exactly the thing you cannot make correctly from data.
- It fails silently in the expensive direction. The customer who was refused their second coffee does not file a bug; they file a complaint about being charged once for two coffees, six weeks later.
- It cannot express "return the original result", because there is no key to look the original up by. So the second call either creates a second payment or errors, and neither is what the retrying client needs.
What limits the solution, and what must never stop being true
This domain leads with these two. A design that ignores its constraints is not a design, and an invariant nobody named is one nothing is protecting.
- The operation crosses a network, so retries will happen whether or not you design for them (What Changes at the Network Boundary).
- Clients include a mobile app that retries automatically, a job scheduler that retries on crash, and a human clicking a button.
- The service is already live with
createPayment(amount, method), so any change has to be compatible for existing callers (Backward Compatibility as a Constraint). - Whatever stores the dedup record has to survive a process crash, so it is a durable store and not a cache.
- One customer intent produces at most one payment, no matter how many times any layer retries.
- A repeat of a completed command returns the original result — not a new one, and not an error. A retry that fails on the second attempt is not idempotent, it is merely guarded.
- The identity of a command is decided by the client, because only the client knows whether this is a new intent (State Ownership).
Who owns what, and where the seams fall
Responsibilities decide boundaries; boundaries decide what an interface has to say.
- The client owns generating the command id, once, at the moment intent is formed — before the first attempt, not inside the retry loop (Stable Identifiers).
- The service owns recording that id durably, with the result, before or atomically with the effect.
- The signature owns making the whole arrangement non-optional. A parameter can be forgotten in documentation; it cannot be forgotten in a signature that does not compile without it.
- Something owns expiring old command records, because an unbounded dedup table is a slow-motion outage (Data Retention in Data Engineering).
- The boundary is the operation, and the scope of the id is that operation. A command id for "create payment" says nothing about "capture payment"; scoping it too broadly makes distinct operations collide (What Counts as the Same Operation? in Distributed Systems).
- The dedup record and the effect must be inside one transaction, or you have moved the problem rather than solved it (Consistency Boundaries).
- The seam is where a retry can be introduced by anyone: an HTTP client, a proxy, a queue, a job runner, a user. All of them are behind the same boundary and none of them can be trusted not to retry.
The parameter is the design
Compare the two signatures and notice how little of the argument is about implementation. The first cannot be made safe by any caller, any middleware or any amount of care, because the information needed to deduplicate is not present. The second is safe by construction, and the implementation is almost mechanical.
This is why §170 insists on not retrofitting. The retrofit is not hard because the dedup logic is hard; it is hard because the id has to be threaded backwards through every caller to the point where intent was formed, and in a job runner or a queue consumer that point is often in another system.
1// cannot be made retry-safe by any caller2async function createPayment(amount: Money, method: Method): Promise<Payment>3 4// safe by construction; the id is not optional and not inferred5async function createPayment(6 commandId: CommandId, // minted where intent is formed7 amount: Money, method: Method,8): Promise<Payment> {9 return db.tx(async (t) => {10 const seen = await t.commands.byId(commandId)11 if (seen) return seen.result // replay: original answer12 13 const payment = await charge(amount, method)14 15 await t.commands.insert({ // UNIQUE(commandId)16 id: commandId, result: payment,17 }) // same transaction as the effect18 return payment19 })20}21 22// the caller, and the line that actually matters:23const commandId = uuid() // ONCE, when the button is pressed24await retry(() => createPayment(commandId, amount, method))25// ^ the id is OUTSIDE the retry. Inside it, none of this works.Three details carry the correctness: the unique constraint rather than the lookup is what arbitrates concurrent duplicates, the insert is in the same transaction as the effect, and the id is minted outside the retry loop. The last is the one implementations get wrong most often, and it is invisible in review unless you are looking for it (Backend Races in Backend).
A command has a lifecycle, and it is not two states
Most implementations model a command as present or absent, which cannot express the case that actually occurs under load: a second attempt arriving while the first is still in flight. At that moment the record exists, has no result, and the correct behaviour is neither "replay" nor "proceed".
Making the lifecycle explicit turns that from a race into a state with a defined answer.
| From | On | To | Guard | Effect |
|---|---|---|---|---|
| ABSENT | first attempt | IN_FLIGHT | insert wins the unique constraint | the row is the lock; a concurrent attempt loses here |
| IN_FLIGHT | effect completes | SUCCEEDED | — | result stored in the same transaction as the effect |
| IN_FLIGHT | provider declines | FAILED_PERMANENT | — | the refusal is stored so retries get the same answer |
| IN_FLIGHT | duplicate arrives while running | IN_FLIGHT | lease not expired | wait, or return 409 in-progress — never a second effect |
| IN_FLIGHT | lease expires after a crash | ABSENT | the effect is verified not to have happened | reconciliation, not a timer (Designing for Failure) |
- IN_FLIGHT → SUCCEEDED — By a *different* attempt than the one holding the record. Two attempts each completing an effect is the exact duplicate this design exists to prevent; only the attempt that won the insert may write a result.
- SUCCEEDED → IN_FLIGHT — Reopening a completed command lets a late retry produce a second effect. A completed command is terminal; a new intent needs a new id.
- FAILED_PERMANENT → IN_FLIGHT — Retrying a permanent failure burns provider quota and, for a declined card, can trigger fraud scoring. Permanence is the whole point of distinguishing it from a transient failure (An Error Taxonomy That Survives Contact).
- IN_FLIGHT → ABSENT — On a bare timer, with no verification. A slow provider looks exactly like a crashed one from here, so expiring on time alone re-runs an effect that may be about to complete — the double charge you were preventing, delivered by the mitigation (Retries Are a Property of the Operation).
The last forbidden transition is the failure of the mitigation itself, and it is the one that produces the worst incidents: a lease expiry that is not backed by a check with the provider converts a slow dependency into duplicate effects at exactly the moment the system is already struggling.
Choosing the identity
What makes two requests the same request is a domain question with several defensible answers, and the choice determines what the system can and cannot express. Picking it deliberately is most of the work; the storage is mechanical.
Who knows, at the moment of the first attempt, that this is a new intent rather than a repeat?
when The client can mint and persist an id across its own retries — a browser, a mobile app, a job runner with durable job records.
cost The strongest option and the default. Cost: clients must cooperate, and a client that regenerates the id silently disables the protection with no server-side symptom.
when The domain genuinely forbids repetition: one enrolment per student per course, one invoice per period per customer.
cost No client cooperation needed and the constraint is a real domain invariant rather than a mechanism. Cost: it only works where repetition is genuinely illegal, and every attempt to stretch it to where repetition is merely unusual produces silent refusals (Enforcing Invariants).
when Processing a message from a queue or a webhook, where the producer already assigned an id.
cost Free — the id exists. Cost: you inherit the producer's id semantics, including whether it is stable across their retries, which is a contract you should verify rather than assume (Contract Tests).
when Almost never. It is the naive design in this lesson.
cost Wrong in both directions with no correct window, and both errors are silent. Acceptable only for low-consequence effects where a missed duplicate and a wrongly-suppressed action are both cheap.
when The effect can be expressed as setting a state rather than appending one: cancel(orderId) sets status to CANCELLED.
cost The cheapest correct answer when it is available, and it is available more often than people assume. Cost: it constrains the domain model to absolute rather than relative operations — setQuantity(5) is idempotent, addOne() is not (Explicit State).
How to build it
Most important first.
- Put the id in the signature. This is the whole lesson:
createPayment(commandId: CommandId, amount: Money)cannot be called without deciding what makes this call this call (Units in Names and Types). - Generate it at the point of intent. The client mints a UUID when the customer presses the button and reuses it for every retry of that intent — a new id per retry is the most common implementation error and it disables the entire mechanism.
- Store the id and the result together, in the same transaction as the effect. Then a repeat is a lookup that returns the original outcome (Where the Transaction Boundary Goes in Backend).
- Return the original result, not a conflict. The retrying client wants the answer it did not receive; an error forces it to invent a recovery path it does not need (An Error Taxonomy That Survives Contact).
- Handle the in-flight case: a second call arriving while the first is still running should wait or report
in progress, not create a second effect. This is the case everyone forgets and it is the one that actually happens under load (Concurrency by Design). - Prefer operations that are naturally idempotent where you have the choice. "Set status to CANCELLED" needs no id; "append a cancellation" does. Choosing the first shape is cheaper than making the second safe (Explicit State).
What the next change costs
The field this whole domain exists for. A structure is only better if it makes the change after this one cheaper — and it is worth saying which changes it does not help.
- Retrofitting: every caller must be found and given an id, ids must be threaded through queues and job records, existing rows have no id so the mechanism cannot be enforced until backfill, and there is a window where old and new callers coexist. Expect weeks, plus a migration. This is precisely what §170 means by not adding it after the fact.
- Designed in: the id is a parameter from the first commit. Adding a new caller costs one line. Adding a new idempotent operation costs a table and a constraint that already have a pattern to copy.
- The next change to retry policy — more attempts, a longer backoff, a queue in front — costs nothing at all, because retry safety is a property of the operation and not of any particular caller (Retries Are a Property of the Operation).
- The permanent cost: one durable write on every call, and one more parameter every caller must supply and every reader must understand.
- A durable write on the hot path costs latency on every request to protect against duplicates in a small fraction. For a high-volume, low-consequence operation that is a bad trade and dropping duplicates is fine.
- It pushes work onto clients, who must generate and persist ids. A client that gets this wrong silently disables the protection, and you cannot detect that from the server.
- The dedup table is a new operational object with growth, retention and hot-key characteristics of its own (Designing for Cost).
What can go wrong
- The id is generated inside the retry loop or by a middleware, so every attempt has a fresh id. The table fills, the mechanism reports success, and duplicates continue.
- The dedup record is written after the effect. A crash in between produces an effect with no record, and the retry duplicates it — the write must be in the same transaction.
- Concurrent duplicates: two retries arrive simultaneously, both find no record, both proceed. Without a unique constraint doing the arbitration, the mechanism is a race (Backend Races in Backend).
- The table grows without bound and eventually dominates the database. The retention policy is part of the design, not an operational afterthought.
- The id is scoped globally rather than per operation, so an id reused for a different operation returns the wrong cached result — a subtle and very confusing bug.
- The service depends on a durable, transactional store for command records. A Redis cache with an eviction policy is not that, and using one converts a correctness property into a probability.
- Callers depend on being able to generate and persist an id across their own retries, which for a job runner means the id lives in the job record (Explicit State).
- Nothing depends on request content for identity any more, which is the coupling this removes.
- "Idempotency means the same result." It means the same *effect*. The second call may legitimately return a different envelope — a
replayed: trueflag, different timing — as long as no second payment exists (Idempotency vs Deduplication in API Design). - "So make everything idempotent." Reads already are. Naturally idempotent writes need no id. The machinery is for operations with an effect that must not repeat (Over-Design and Under-Design).
- "The server should generate the id." Then a retry gets a new one and the mechanism does nothing. The id must come from the side that knows whether this is a new intent.
- "A unique constraint on business fields is equivalent." It conflates identity with content, so a customer legitimately repeating a purchase is refused, and the failure is silent and customer-visible.
- primitive-obsession
Testing it, and how it ages
- Call twice with the same id, assert one effect and identical results — the basic property, and the one most implementations pass (Property-Based Testing).
- Call twice concurrently with the same id and assert one effect. This is the test that finds real implementations wanting, because it exercises the constraint rather than the lookup (Concurrency by Design).
- Crash between effect and response, retry, assert one effect and the original result returned (Where a Test Must Be Real).
- Assert the negative: two different ids with identical content produce two payments. Someone will eventually "improve" the dedup to hash content, and this test is what stops them.
- Command records accumulate and become a genuinely useful audit trail — what was requested, by whom, and what came back — which is often worth more than the deduplication (Debuggability by Design).
- Retention pressure arrives around the first million rows and forces a decision about how long a retry may legitimately arrive after the original. That number is a business answer, not a technical one.
- The pattern spreads to every mutating operation at the boundary, and the ones that resist it are usually the ones with a modelling problem underneath (Explicit State).
Where this applies
This domain's advice is contested more than most. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view rather than a caricature.
- GENERALThat identity of intent must be supplied by whoever forms the intent is a consequence of the information available on each side, so it holds for HTTP APIs, message consumers, job runners and in-process command handlers alike.
- DOMAIN-SPECIFICFor payments, inventory reservation and anything with legal or financial consequence this is mandatory. For appending an analytics event, duplicates are cheaper than the machinery and the correct design is to accept them and deduplicate downstream at query time — which is the same reasoning reaching the opposite conclusion.
- CONTESTEDThe strongest opposing argument is that server-side deduplication on a natural business key is simpler and sufficient for most systems: it needs no client cooperation, cannot be broken by a client that regenerates ids, and for genuinely unique intents — one enrolment per student per course — the business key really is the identity. That is correct precisely where a natural key exists and is genuinely unique. The failure is at the edges, where the same customer legitimately repeats an identical action, and the disagreement is really about how often that happens in your domain rather than about the mechanism.
Where the depth lives
This domain teaches the codebase-level structure and hands the rest off.
- — Testing & Reliability Engineering — proving idempotency under concurrent load, and the fault-injection setup that exercises the crash-between-effect-and-record window, are that domain's techniques.