Webhooksidempotencydeduplicationevent idexactly-onceconsumer

Consumer-Side Idempotency

The provider promised at-least-once, so duplicates are not a bug — they are scheduled. Exactly-once processing is an illusion the consumer manufactures locally: record the event_id, process each id exactly once, and make the recording atomic with the effects.

Follow the failure

Frame the contract

API design starts with a consumer, a design question and a guarantee — never with a URL.

Design question
When the same event arrives twice — and it will — how does the consumer make the second arrival a no-op instead of a second shipment?
Consumers
Any engineer writing a webhook receiver whose effects are not naturally idempotent: shipping orders, sending emails, crediting accounts, kicking off jobs.
The promise
A consumer that keys processing on `event_id` and records that key atomically with its side effects converts at-least-once delivery into exactly-once *processing* — the only place exactly-once can actually exist.
Requirement→Consumers→Resource Model→Style→Contract→Validation→Authorization→Errors→Idempotency→Pagination→Versioning→Observability→Evolution→Trade-offs

Duplicates are scheduled, not exceptional

Walk the ordinary failure: the provider POSTs order.paid, your handler processes it in 9 seconds, the provider's timeout was 8. From the provider's ledger the attempt failed, so 30 seconds later the same event arrives again — same event_id, new delivery_id. Nothing malfunctioned. Both sides followed their contract exactly, and you shipped the order twice.

Redelivery also arrives in bulk: a Webhook Delivery: States, Retries, Redrive redrive after your outage replays thousands of old events, some of which you *did* process before crashing. And it arrives concurrently: a retry can land while the original attempt is still executing, so two workers process the same event side by side. A dedup design has to survive all three shapes — the lone retry, the bulk replay, the concurrent double — not just the first.

The retry that follows a timed-out success: same event_id, new delivery_id
Request
POST /hooks/orders HTTP/1.1
Host: consumer.example
X-Delivery-Id: dlv_5c09e2   ← differs from the first attempt
X-Event-Id: evt_8f2c1a      ← identical to the first attempt
Content-Type: application/json

{
  "event_id": "evt_8f2c1a",
  "type": "order.paid",
  "occurred_at": "2026-08-25T09:14:03Z",
  "data": { "order_id": "ord_4211" }
}
Response
HTTP/1.1 200 OK

{ "received": true }

← the consumer returns 200 even for a duplicate:
  "I have durably accepted this event_id" is true
  whether processing ran now or last time. A 409
  here would just make the provider retry again.

Dedup that survives concurrency: one atomic claim

The broken version of dedup is check-then-act: look up event_id in a processed-set, and if absent, process and then record it. Two concurrent deliveries of the same event both pass the check, both process. The fix is to make the *claim* atomic: insert the event_id into a table with a unique constraint (or SET NX in Redis, with the durability caveat that a cache flush reopens the duplicate window). Exactly one worker wins the insert; the loser stops.

Claiming is half the job — the claim and the effects must not be separable by a crash. If you record the event as processed and crash before shipping, the order never ships and the retry is refused; if you ship and crash before recording, the retry ships again. When the effects live in your own database, put the insert and the effects in one transaction. When they cross a process boundary (an email API, a payment call), the transaction cannot cover them — record the event and the *intent* atomically, then execute the external effect from a worker with its own idempotency protection, ideally passing the provider's event_id onward as your Idempotency Keys: The Mechanism key for the downstream call.

Check-then-act: correct alone, wrong under concurrency
1def process(event):
2  if db.exists("processed", event.id):   # both workers: false
3    return
4  ship_order(event.data.order_id)        # both workers ship
5  db.insert("processed", event.id)       # second insert fails,
6                                          # too late to matter
Atomic claim inside the same transaction as the effects
1def process(event):
2  with db.transaction():
3    ok = db.try_insert("processed_events",
4                       id=event.id)      # unique constraint
5    if not ok:
6      return                             # someone else won
7    db.insert("shipments",
8              order_id=event.data.order_id)
9  # external effects go through the outbox/worker,
10  # keyed on event.id downstream

The unique constraint turns "have I seen this?" into a race-free claim: the database serializes the two workers, exactly one proceeds, and a crash rolls back claim and effects together. Check-then-act passes every single-threaded test and fails the first concurrent retry.

Scope, retention, and what to dedup on

Dedup on `event_id`, never on delivery_id (which changes per attempt, by design) and never on a payload hash (two legitimate distinct events can carry identical payloads — two order.paid events for the same amount — and would falsely collapse). The provider's event id is the identity of the business fact; that is the entire reason Webhooks: The Inverted Contract requires it in the envelope.

Retention must outlast every path by which an old event can reappear: retry horizon plus redrive window plus your own replay tooling. If the provider retries for 72 hours and offers 30-day redrive, keeping processed ids for 45–60 days is cheap insurance — at one row per event, ten million events a month costs you a few hundred MB, versus one duplicated shipment costing more than the table ever will. This is Idempotency vs Deduplication from the consumer's chair: you are deduplicating *messages* here; whether your business effect is idempotent is a separate property worth having anyway, as the second line of defense when the dedup store lets you down.

  • Key: event_id — stable across retries and redrives; that is its job.
  • Not delivery_id — unique per attempt; deduping on it dedups nothing.
  • Not payload hashes — distinct facts can look identical; collapsing them is silent data loss.
  • Retention: ≥ provider retry horizon + redrive window; document your own number.
  • Store: durable and transactional beats fast and flushable — a Redis-only dedup set reopens the window on every failover.

Key points

  • At-least-once delivery makes duplicates a scheduled event, arriving as lone retries, bulk redrives, and concurrent doubles.
  • Exactly-once exists only as consumer-side processing: claim the event_id atomically, exactly one claimant proceeds.
  • Check-then-act dedup fails under concurrency; a unique-constraint insert in the same transaction as the effects does not.
  • External side effects cannot ride in your transaction — record intent atomically, execute via a worker, and pass event_id downstream as the idempotency key.
  • Return 200 for duplicates: "durably accepted" is true either way, and any error status just provokes another retry.
  • Retention outlasts retry horizon plus redrive window; dedup on event_id, never delivery_id or payload hashes.

Follow the failure

How the contract fails or gets misused, hop by hop — and what it costs when it completes.

  1. 1
    Consumer → handler: processes events directly, no dedup — the demo never sees a duplicate, so the code assumes there are none.
  2. 2
    Provider → consumer: an 8s timeout fires on a 9s handler; the retry arrives while the original is still running.
  3. 3
    Both workers → database: check-then-act dedup passes twice; the order ships twice.
  4. 4
    Consumer → patch: adds a Redis processed-set with a 24h TTL; the provider's 72h retry horizon sails past it.
  5. 5
    Provider → redrive: an outage recovery replays 40,000 events; every one older than 24 hours processes again.
What breaks
  • Duplicate irreversible effects: double shipments, double emails, double credits — each a customer-facing incident with a refund path.
  • Reconciliation drift: consumer records disagree with provider records, and without recorded event ids nobody can prove which duplicates ran.
  • Redrive becomes unusable: the provider's recovery tool, which assumes consumer dedup, multiplies damage instead of repairing it.

Design, observe, evolve

A contract decision is incomplete until you know how you would notice it failing and how it changes later.

Design the contract
  • • Claim event_id via unique-constraint insert in the same transaction as local effects; treat the losing insert as success.
  • • Route external side effects through a worker that receives the event_id and uses it as the downstream idempotency key.
  • • Size dedup retention from the provider's published retry horizon and redrive window, and write your number down.
  • • Make business effects idempotent where the domain allows (upsert the shipment keyed by order_id) as a second layer under the dedup store.
Observe in production
  • • Count duplicate claims (losing inserts) per event type: a baseline trickle is health; a spike means your handler latency crossed the provider timeout or a redrive is running.
  • • Alert when dedup-store retention is shorter than the provider's configured retry horizon — the drift usually happens when someone "optimizes" the TTL.
  • • Sample-audit effects against events: shipments without a claimed event_id are the leak your dedup missed.
Evolve without breaking
  • • Dedup lives entirely on the consumer side, so it can be retrofitted without provider changes — but backfilling identities for already-processed events requires a reconciliation pass first.
  • • If the provider later shortens or lengthens its retry horizon, retention is one config value — provided you recorded which provider promise it was derived from.
What it costs
  • • A durable dedup store adds a write to every event and a table that grows with traffic; the cost is trivial next to one duplicated shipment, but it is not zero.
  • • Transactional claiming serializes concurrent deliveries of the same event, adding contention exactly during retry storms — which is the correct place to be slow.
  • • Idempotent business effects (upserts keyed on business ids) can mask provider bugs that dedup would have surfaced; keep the duplicate-claim metric even when the effect layer absorbs them.

Misconceptions

Claim
“The provider retries only on failure, so if my handler works, I will never see duplicates.”
Reality
A timeout after successful processing is recorded as failure on the provider side. Your handler being correct is exactly how you earn a duplicate: it processed, the 200 arrived late, the retry came anyway.
Claim
“Deduplication and idempotency are the same thing.”
Reality
Dedup detects a repeated *message* (same event_id) and skips it; idempotency makes a repeated *operation* produce the same effect. You want both: dedup as the gate, idempotent effects as the backstop when the gate's store fails you. See Idempotency vs Deduplication.
Claim
“Returning 409 for a duplicate tells the provider what happened.”
Reality
Most providers treat non-2xx as delivery failure and retry — a 409 for duplicates schedules the next duplicate. Acknowledge with 200: from the delivery contract's perspective, "already durably accepted" is success.

Apply it

⚡ Break: Webhook Delivered Twice
← Webhook Delivery: States, Retries, RedriveWebhook Ordering: Assume None →