Webhook Delivery: States, Retries, Redrive
Every event delivery is a little state machine: queued → attempting → delivered, or failed → retrying → dead. The retry schedule, the definition of "delivered", and the dead-letter escape hatch are contract clauses both sides build against.
Frame the contract
API design starts with a consumer, a design question and a guarantee — never with a URL.
The delivery state machine
Treat each (event, endpoint) pair as a state machine, not a fire-and-forget POST. The event is queued when the business fact commits, attempting while a POST is in flight, delivered when the consumer returns 2xx within the timeout, and retrying with a next-attempt-at timestamp otherwise. After the schedule is exhausted it becomes dead — parked, not discarded, with every attempt's status code and latency recorded.
This machine is contract surface because consumers make real engineering decisions against it. Their dedup retention must outlast your retry horizon (Consumer-Side Idempotency); their outage recovery plan depends on whether a 4-hour downtime means "events arrive late" or "events are gone". A provider that cannot state its own retry schedule has one — implicitly, in code — and consumers will discover it during an outage.
One consumer endpoint must never degrade delivery to others. Serialize or bound concurrency *per endpoint* (a slow consumer gets slow deliveries, not a bigger share of your workers), and consider automatic disablement with notification after days of sustained failure — an abandoned endpoint retrying forever is pure waste and a nice amplification target.
What counts as delivered, and when to retry
"Delivered" needs a definition sharp enough to code against: the consumer returned a 2xx status within your attempt timeout — typically 5–10 seconds. Not "we sent the bytes", not "they returned *something*". A 3xx is a failure (following redirects re-sends signed bodies to locations nobody vetted); a timeout is a failure *even if the consumer actually processed it* — which is precisely why duplicates are structural and not a bug.
Response codes should steer the retry decision the way Retryability: Telling Clients What To Do Next steers clients, with the roles reversed: 5xx and 429 mean try again later; most 4xx mean the request itself is defective — a 401 from a rotated secret or a 410 from a decommissioned endpoint will not improve with repetition. Retrying hard 4xx for a few attempts is defensible (consumers misconfigure things transiently); retrying them for 72 hours is denial-of-service against your own worker pool.
The schedule itself should be exponential with jitter, published as a table, and finite. A common shape: a quick first retry to paper over blips, then widening gaps over roughly three days. Anything shorter strands consumers with weekend outages; anything much longer means your dedup and storage obligations grow without buying consumers much.
attempt 1 immediately
attempt 2 +30 seconds
attempt 3 +5 minutes
attempt 4 +30 minutes
attempt 5 +2 hours
attempt 6 +6 hours
attempts 7+ every 12 hours, up to 72h total
success: any 2xx within 8s
retried: timeout, connection error, 5xx, 429
not retried after 3 tries: 400, 401, 403, 404, 410
after horizon: event → dead letter, endpoint flagged,
consumer notified, redrive available 30 daysThe consumer's side: ack fast, process later
The single most important consumer-side rule: the webhook handler acknowledges, it does not process. Verify the signature, persist the event, enqueue the work, return 200 — tens of milliseconds. A handler that does the real work inline (calls its own database, a third-party API, sends an email) will eventually exceed the provider's 8-second timeout, get marked failed, and receive a retry *while the first attempt is still running* — manufacturing the exact duplicate-under-concurrency scenario that is hardest to dedup.
Dead letters and redrive close the loop. The consumer had a bad deploy, returned 500 for six hours, and 40,000 events died: the contract answer is a redrive — re-queue dead deliveries, by endpoint and time range, triggered by the consumer from a dashboard or API rather than by filing a ticket. Redriven events are re-deliveries of the same event_id, which is why consumer dedup retention must exceed retry horizon *plus* redrive window.
1def handle_webhook(req):2 event = parse(req.body)3 order = db.load(event.data.order_id)4 warehouse_api.create_shipment(order) # 2–40s, third party5 email.send_confirmation(order) # 1–5s6 return 2007 # provider timed out at 8s during create_shipment8 # → marked failed → retried → second shipment1def handle_webhook(req):2 verify_signature(req) # reject forgeries first3 db.insert_event(event_id, req.body) # idempotent insert4 queue.enqueue(process_event, event_id)5 return 200 # ~20ms, always6 7def process_event(event_id): # worker, own retries8 ...The good handler makes "delivered" mean "durably accepted", which is the only promise a consumer can keep in 8 seconds. Processing moves behind the consumer's own queue, with its own retry policy, where a 40-second warehouse call is normal instead of fatal.
Key points
- Model delivery as a state machine — queued, attempting, delivered, retrying, dead — with every attempt recorded; fire-and-forget POSTs cannot honor any contract.
- Define "delivered" precisely: 2xx within a stated timeout (5–10s). Timeouts count as failures even when the consumer processed the event — duplicates are structural.
- Publish the retry schedule and horizon (exponential with jitter, ~72h is common); consumers size dedup retention and outage recovery against it.
- Steer retries by status code: 5xx/429 retry, hard 4xx stop after a few attempts; isolate slow endpoints so one consumer cannot starve the rest.
- Dead-letter exhausted events and offer self-serve redrive; consumers ack fast and process from their own queue.
Webhook Delivery Simulator
Change the contract and observe which guarantee moves.
—
Follow the failure
How the contract fails or gets misused, hop by hop — and what it costs when it completes.
- 1Provider → dispatcher: implements delivery as an inline POST with one retry, no persisted state, no published schedule.
- 2Consumer → handler: processes events inline; p95 handling time is 6 seconds, occasionally 30.
- 3Traffic spike → both: slow handlers exceed the provider timeout, get retried, and now run concurrently with themselves.
- 4Consumer outage → events: a six-hour deploy failure exhausts the single retry; events are silently gone.
- 5Reconciliation → weeks later: the consumer's nightly report disagrees with the provider's; neither side has a delivery log to arbitrate with.
- Events lost forever after consumer outages shorter than a weekend, with no dead letter to recover from.
- Duplicate-under-concurrency processing on the consumer side, triggered by the provider's own timeout policy.
- Provider worker pools consumed by dead endpoints retrying 4xx responses for days.
Design, observe, evolve
A contract decision is incomplete until you know how you would notice it failing and how it changes later.
- • Persist delivery state per (event, endpoint) and publish the schedule, timeout, horizon and dead-letter policy as documentation consumers can engineer against.
- • Bound per-endpoint concurrency and auto-disable endpoints after sustained failure, with notification before and after.
- • Provide a delivery log API and self-serve redrive scoped by endpoint and time range.
- • Document the consumer pattern explicitly: verify, persist, enqueue, 200 — and say that inline processing will cause duplicates.
- • Per-endpoint success rate, attempt latency and current backoff stage; a consumer sliding toward dead-letter is visible days in advance.
- • Dead-letter inflow rate and age; a spike is either your bug or a consumer incident, and the delivery log says which.
- • Distribution of attempts-per-delivered-event: creeping upward means consumer fleets are slowing down or your timeout is too tight.
- • Retry schedules can lengthen and dead-letter retention can grow without breaking consumers; shortening either is a breaking change to their recovery plans and needs notice.
- • Adding delivery-log and redrive APIs later is additive and pays down years of support tickets; start with at least the data model so the history exists.
- • Persisted per-delivery state is a real storage and complexity cost — a busy platform stores billions of attempt records to keep a promise most events never need.
- • Long retry horizons help consumers with outages but stretch the duplicate window and the dedup retention consumers must fund.
- • Auto-disabling failing endpoints protects your fleet but converts "late events" into "no events" for a consumer who was mid-incident; the notification path matters as much as the mechanism.