Job Idempotency
Delivery will repeat, so the effect must not. How to make a worker safe to run twice, including concurrently.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The same message is delivered twice. How do I guarantee the effect happens once?
A retried job must not send a second email, charge a second time, or add a second row — even if the two deliveries arrive at the same moment.
Check whether the work has already been done at the start of the handler: if the order is already paid, return early. A simple guard covers the retry case.
The guard is a read followed by a write with a gap in the middle. Two concurrent deliveries both read pending, both pass the guard, and both charge. This is a check-then-act race and it happens exactly when duplicates happen — under retry pressure (Backend Races).
- The guard is a read followed by a write with a gap in the middle. Two concurrent deliveries both read
pending, both pass the guard, and both charge. This is a check-then-act race and it happens exactly when duplicates happen — under retry pressure (Backend Races). - The guard only covers the last step. A job that charges, then updates status, then emails, crashing after the charge, will re-charge on retry because the status was never written.
- It cannot express partial completion. Three side effects and one status flag means the retry either redoes all three or skips all three, and neither is right.
- External side effects are outside your database. Marking the order paid does not stop the payment provider from accepting a second capture — only a key they honour does (Idempotency Keys).
- Nothing bounds the state. If the guard reads a table that is pruned after 24 hours, a duplicate on day two is not a duplicate as far as the guard is concerned.
What is actually happening
- Idempotent means: applying the effect N times leaves the same state as applying it once. Not "the handler does not crash on a second run" — the observable state must be identical.
- Some operations are naturally idempotent: setting a field to a fixed value, an upsert on a natural key, a delete by id, writing a computed document to a search index. If you can express the work this way, you need nothing else.
- Everything else needs a deduplication record: a row keyed by something stable, inserted atomically, whose presence means "this effect already happened or is happening". The atomicity has to come from the database — a unique constraint or an atomic insert-if-absent — because a read-then-write reintroduces the race you are trying to remove (Duplicate Detection).
- The key must be derived from the *effect*, not from the delivery. A broker message id changes when a producer re-publishes;
capture:order-123does not. Key on the business fact you are asserting (Idempotency Keys). - For work that is not naturally idempotent and cannot be deduplicated locally — a third-party charge — push the requirement outward: most payment and messaging APIs accept an idempotency key and will return the original result rather than performing the action twice.
- Multi-step jobs need per-step records, or a single step that is atomic. A job that does three things needs either one transaction covering all three, or three effect keys so that a retry resumes rather than restarts (Where the Transaction Boundary Goes).
Make the guard atomic, not sequential
The difference between a handler that is idempotent and one that merely looks idempotent is a single property: whether the "have I done this?" question and the "record that I am doing it" answer happen as one indivisible operation. A read followed by a write is two operations with a gap, and duplicates arrive precisely when that gap is busy.
The database already has the primitive: a unique constraint. Insert first, let the constraint reject the duplicate, and treat the rejection as the answer. The check and the claim become the same statement.
const order = await db.orders.byId(msg.orderId)
if (order.status === 'paid') return // <- both deliveries see 'pending'
await payments.capture(msg.orderId, msg.amountCents)
await db.orders.update(msg.orderId, { status: 'paid' })
// Two captures. The guard read stale state in the window
// before either write landed.// effects(key) has a UNIQUE constraint. The insert IS the check.
const claimed = await db.query(
`INSERT INTO effects (key, message_id, created_at)
VALUES ($1, $2, now())
ON CONFLICT (key) DO NOTHING
RETURNING key`,
[`capture:${msg.orderId}`, msg.id],
)
if (claimed.rowCount === 0) {
metrics.inc('job_duplicate_suppressed', { type: msg.type })
return // someone else owns this effect
}
await payments.capture(msg.orderId, msg.amountCents, {
idempotencyKey: `capture:${msg.orderId}`, // the provider dedupes too
})
await db.orders.update(msg.orderId, { status: 'paid' })The unique constraint serialises the two deliveries inside the database, so exactly one of them can hold the effect. There is no window between the check and the claim because they are the same statement. The provider-side key then closes the remaining window — the one between claiming the effect and completing it, where a crash would otherwise leave a claim with no capture.
Choose the cheapest form the operation allows
Not every job needs a deduplication table. Many operations can be reshaped so that repetition is harmless by construction, and that is always the better outcome: no extra write, no retention policy, no index.
The pattern to look for is assert a state rather than apply a delta. "Set status to shipped" is idempotent; "increment shipped count" is not. "Write this document to the index" is idempotent; "append this row" is not. Where the domain permits the first shape, take it.
| Operation | Naturally idempotent? | How to make repetition safe | Cost |
|---|---|---|---|
| Set a field to a fixed value | Yes | Nothing needed — the second write is a no-op | None |
| Upsert a row on a natural key | Yes | INSERT ... ON CONFLICT DO UPDATE | None beyond the unique index you already want |
| Write a document to a search index | Yes | Index by document id; a second write replaces (Keeping a Search Index in Sync) | None |
| Increment a counter | No | Record the contributing event id, or recompute from source rows | A dedupe row per event, or a scan to recompute |
| Insert an audit or ledger row | No | A unique constraint on (entity, effect, occurred_at) | One unique index |
| Send an email or push notification | No | An effect key claimed before sending; providers rarely dedupe (Email and Notifications) | A dedupe row, and a small window where a crash suppresses a send |
| Charge a payment method | No | An effect key locally plus the provider's idempotency key | A dedupe row and reconciliation against the provider |
| Call a third-party API with no key support | No | Claim locally, then reconcile; there is no way to be certain | Ongoing reconciliation — the genuinely hard case |
Multi-step jobs need per-step keys
A job that charges a card, marks an order paid and sends a receipt has three effects and one retry story. With a single status flag, a crash after the charge means the retry either re-charges or skips the receipt — there is no state that says "step one done, steps two and three not".
Two shapes fix it. Put everything that can be in one transaction into one transaction, so it is atomic by construction; and give each effect that cannot be — the external call, the email — its own key, so a retry resumes at the first unclaimed step. The pipeline below is what a resumable handler looks like.
- 1Claim capture:{orderId}
Atomic insert; if it already exists, skip to the next step.
fails by Claiming in a different transaction from the effect, leaving a claim with no charge.
- 2Capture the payment
External call with the same key as the idempotency key.
fails by Provider does not support keys — then reconciliation is the only answer.
- 3Mark the order paid
A fixed-value write; idempotent by construction.
fails by Nothing — repeating this is a no-op, which is why it needs no key.
- 4Claim receipt:{orderId}
Second atomic insert, independent of the first.
fails by Sharing one key across both effects, so a retry after the charge never sends the receipt.
- 5Send the receipt
Hands the email to the provider.
fails by A crash between claim and send suppresses the email permanently — the deliberate trade of at-most-once for this step.
- 6Ack
Tells the broker the message is complete.
fails by A crash here redelivers, and every step above is now a no-op. That is the design working.
The last line is the test: after the final step, a full redelivery of the message must change nothing. If any step would repeat, it needs a key it does not have.
How to build it
Most important first.
- Prefer naturally idempotent operations. An upsert keyed on the entity, or a write of the full computed state, removes the whole problem and needs no extra storage.
- Where you cannot, insert a deduplication row keyed on the effect, and rely on a unique constraint to make the check atomic. Catch the constraint violation as "already done" rather than checking first (Database Constraints).
- Pass an idempotency key to every external API that accepts one, derived from the same effect key, so the third party deduplicates its own side.
- Record the *outcome* alongside the key when a caller needs it — a duplicate delivery should be able to return the original result rather than merely doing nothing (Idempotency Storage).
- Give effect records a retention at least as long as your maximum possible retry window, including the time a message can sit in a dead-letter queue before being replayed.
- For multi-step jobs, make each step independently keyed so a retry resumes at the first incomplete step instead of repeating the completed ones.
- Test it directly: run the handler twice on the same message in an integration test, and once concurrently. Both cases belong in the suite, because the concurrent one is the one that fails in production (Test Against the Real Database).
What can go wrong
- A check-then-act guard that passes under concurrency, producing exactly the duplicate it was written to prevent.
- A deduplication key derived from the message id, so a re-published message is treated as new work.
- Effect records pruned before the maximum retry window, so a late replay is not recognised as a duplicate.
- The dedupe row committed in a different transaction from the effect: the row exists, the effect does not, and the retry skips work that never happened.
- Idempotency applied to the database write and not to the email, so retries are safe for the row and visible to the customer.
- A key that is too coarse — one per order rather than one per effect — so a legitimate second charge on the same order is silently swallowed.
- A "does the row exist" check on a table another process can also write to, so an unrelated write makes the job think it already ran.
- Two concurrent deliveries both passing a check-then-act guard — the defining race, and the reason the guard must be an atomic insert.
- The effect row committed in a separate transaction from the effect, so a crash between them leaves a claim with no work behind it.
- A duplicate delivery arriving while the first is still in flight: the second must not merely skip, it must either wait for or safely ignore an in-progress effect.
- Two workers keyed on the same entity racing on the underlying row, which idempotency does not solve — that needs versioning or a lock (Optimistic Concurrency).
- A prune job deleting an effect record while a retry for that key is in flight.
- A duplicate charge is a financial incident. Idempotency here is a control, and its absence is an auditable defect rather than a rough edge (Idempotency in Backends).
- Deduplication keys must be scoped per tenant. A key built from a customer-supplied id without a tenant prefix lets one tenant suppress another tenant's work, or read its stored outcome (Tenant Isolation).
- Do not accept a client-supplied idempotency key without binding it to the authenticated principal. An attacker who can guess or reuse a key can replay or block operations they do not own.
- Store outcomes, not payloads, in the deduplication table where you can. It is queried and dumped during incidents far more often than your primary tables (Secrets in Logs).
- "The queue handles duplicates." A broker may deduplicate its own deliveries within a bounded scope. It cannot deduplicate your effect on another system (Queue Semantics).
- "Idempotent means the code can run twice without erroring." It means the *state* is the same after two runs. A handler that cheerfully sends two emails without erroring is not idempotent.
- "Retryable means safe to retry." Retryable describes the transport: another attempt is permitted. Safe describes the effect: another attempt changes nothing. A charge is retryable and unsafe until you add a key.
- "We check before we act, so we are covered." Check-then-act is a race, and it fails precisely under the concurrency that produces duplicates (The Atomicity Illusion).
- "GET is idempotent, so read jobs are fine." Idempotency is about effects. A read job that writes a cache entry, an audit row or a rate-limit counter has effects.
Operating it
- Count duplicate suppressions by job type. A number that is not zero proves the mechanism is doing work; a number that is zero when redelivery rate is high proves it is not wired up.
- Alert on unique-constraint violations that are *not* being caught as duplicates — those are a bug in the guard, not a duplicate (An Error Taxonomy That Maps Cause to Response).
- Track the age gap between the first and second delivery of the same key. It tells you what retention your effect table actually needs, rather than a number someone guessed.
- Log the effect key on every job execution, so "did this run twice?" is a single query rather than an investigation (Structured Logging).
- For externally-keyed operations, reconcile periodically against the provider. Their duplicate detection and yours can disagree, and only a reconciliation finds it.
- At 10x, the effect table becomes a hot write path of its own: one insert per job execution, with a unique index. Size and index it deliberately (Should I Add an Index?).
- At 100x, retention becomes the dominant cost and needs partitioning or time-based pruning rather than a delete job (Idempotency Storage).
- More workers means more concurrent duplicate deliveries, so the *atomicity* of the guard matters more with scale, not less.
- Naturally idempotent designs scale best because they need no extra storage at all — another reason to prefer them where the operation permits.
- A deduplication table costs a write and an index on every job execution, and it is the cheapest correct answer for non-idempotent work.
- Naturally idempotent operations cost nothing and constrain how you model the work — an upsert of full state may write more than a targeted update would.
- Storing outcomes lets a duplicate return the original result and increases what you retain and for how long.
- Fine-grained per-step keys make retries resumable and multiply the number of records and the complexity of the handler.
- Long retention makes late replays safe and makes the table large.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALRequired with every broker, at every scale, in every language. It is the half of the exactly-once construction that no product supplies.
- DATABASE-SPECIFICThe atomic guard depends on the engine. PostgreSQL offers INSERT ... ON CONFLICT DO NOTHING with a RETURNING clause that distinguishes inserted from skipped; MySQL has INSERT IGNORE and ON DUPLICATE KEY UPDATE with different reporting of affected rows. On engines without an atomic upsert you need a unique constraint plus catching the violation. Isolation level also matters: at read-committed the constraint is what serialises you, not the transaction (Database Constraints).
- SIMPLIFIEDPresents one job as one effect. Real jobs often have several, and a genuinely correct multi-effect job needs either one transaction spanning all of them or one key per effect — the middle ground of a single status flag is where most duplicate bugs live.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.