Dead Letter Queues Are an Operation
A DLQ needs an alert, an inspection path, a replay strategy and an owner. Without those four it is a place failures go to be forgotten.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What happens to a message after it fails for the last time — and who finds out?
Dead-lettering is configured as a safety net and then treated as a destination. Work that a customer is waiting for ends up in a queue nobody looks at, with no alert, no tooling and no owner.
Configure a dead letter queue so failing messages stop retrying and stop poisoning the main queue. Good — failures are contained.
Containment is not resolution. The failed work still matters: an unprocessed payment, an unsent notification, an unfulfilled order.
- Containment is not resolution. The failed work still matters: an unprocessed payment, an unsent notification, an unfulfilled order.
- With no alert, the count grows silently. Teams routinely discover months of dead letters during an unrelated investigation.
- By the time someone looks, message retention may have expired and the work is gone with no record of what it was.
- Nobody owns it. The queue belongs to the platform team, the messages belong to the product team, and the failures belong to neither.
- Replayed in bulk without thought, the messages re-fail identically, or succeed and produce duplicate side effects because the original attempt had partially completed.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A dead letter is a message that exhausted its retry policy. That is a statement about attempts, not about the message being invalid — most dead letters are perfectly valid work that failed during a dependency outage.
- This is why classification comes before replay. The four common classes — poison message, transient dependency failure, bad deploy, schema or contract change — need completely different responses, and only one of them is fixed by replaying.
- Replay is a *new* delivery of old work. Everything that made at-least-once processing safe originally has to still hold: idempotency keys, dedupe windows, and side effects that tolerate repetition (Idempotency in Backends in Backend Engineering).
- Replay also has a rate. Dumping an accumulated backlog back into the main queue at full speed re-creates the load spike that may have caused the failure, and starves live traffic behind it.
- Retention is the deadline. A DLQ with a retention window is a countdown on recoverable work, and the alert has to fire with enough lead time to act within it.
- Ordering is usually lost. A replayed message arrives after messages that were produced later, so consumers must not assume the sequence they were originally sent in (Operating Queues and Scheduled Work).
The four things a DLQ needs to be an operation
A dead letter queue with none of these is storage. With all four it is a recovery path. Most systems have the first zero or one of them.
- The alert is the cheapest of the four and the one most often missing.
- The owner is the one that cannot be automated, and the one that decides whether the other three ever get built.
- Retention turns all of this into a deadline: recovery is possible only inside the window (Partial and Logical Data Recovery).
| Requirement | What it means concretely | What its absence looks like |
|---|---|---|
| Alert | A page or ticket on the first message, not on a threshold | Thousands of messages found by accident, months later |
| Inspection | View body, failure reason, attempt count and trace without a production console | Nobody can tell what the messages are, so nobody acts |
| Replay strategy | Classify, fix, replay in rate-limited batches to a watched target | A single bulk redrive that re-fails or duplicates side effects |
| Owner | A named team responsible for the business work in the queue | Platform owns the queue, product owns the meaning, neither acts |
Classify before you replay
Replaying is the reflex and is right for exactly one of these classes. The classification takes minutes and prevents re-running a known failure across an entire backlog.
What class of failure produced these dead letters, and what does each class need?
when The consumer is fine and something it called was down or throttled during a known window.
cost Safe to replay once the dependency is healthy — at a controlled rate, because the same volume that failed is now arriving at once.
when A specific message the consumer cannot process — malformed, missing a reference, violating an invariant.
cost Replay will fail identically. Needs a code fix or a data fix first, then targeted replay; sometimes the honest answer is a recorded discard.
when Failures start at a deploy boundary and stop at the rollback (Change Correlation).
cost Fix or roll back first, then replay the whole affected window. The window boundaries come from the deploy timeline, which is why deploy annotations matter (Deploys on the Same Timeline as the Symptom).
when Producers emit a shape the consumer cannot parse.
cost Old messages may never be replayable by the new consumer. Either add compatibility for the old shape or transform the messages during replay — and treat it as the rolling-compatibility failure it is (Version Coexistence: N and N+1, in Both Directions).
The replay procedure
Replay is a production change with a blast radius, and it deserves the same discipline as a deploy: know what you are changing, do it in a bounded batch, watch the result, and be able to stop.
- 1Alert
Notify the owning team on the first message.
fails by Threshold-based alerting, or a route nobody reads.
evidence The alert has fired for a real event and reached a human (An Alert Should Demand Action).
- 2Preserve
Copy the dead letters to durable storage before touching them.
fails by Messages expire, or a failed redrive loses them.
evidence A stored copy with a count that matches the queue.
- 3Classify
Sample messages and their failure context; assign a class.
fails by Skipping straight to replay, re-running a known failure at scale.
evidence A stated cause and a count per class.
- 4Fix the cause
Deploy the code fix, repair the data, or wait for the dependency to recover.
fails by Replaying into an unfixed consumer, doubling the DLQ.
evidence A new message of the same shape processes successfully.
- 5Verify idempotency
Confirm reprocessing these specific messages cannot duplicate external side effects.
fails by Duplicate emails, charges or webhooks after a successful replay.
evidence A single message replayed end to end, with the side effect confirmed to happen once.
- 6Replay a batch
Send a small batch at a controlled rate to a watched target.
fails by A full redrive that spikes load and starves live traffic.
evidence Batch processed with no new dead letters and no downstream degradation.
- 7Drain
Continue in batches, watching queue age, failure rate and the dependency.
fails by Unattended drains that fail halfway with no record of position.
evidence Dead letter count falling; main queue age stable.
- 8Close
Record what happened, what was discarded, and what changed to prevent it.
fails by The queue is empty and the cause ships again next month.
evidence A postmortem note or action item with an owner (Action Items That Change the System).
Steps 5 and 6 are the ones under time pressure to skip, and are the two that separate a recovery from a second incident.
How to do it properly
Most important first.
- Alert on the first dead letter, not on a threshold. The correct steady-state count is zero, so any non-zero value is information (Alert on Symptoms, Not on Causes).
- Give every DLQ a named owning team, recorded next to the queue and in the service catalogue (The Ownership Record).
- Build an inspection path before you need one: view a message with its metadata, its failure reason and its attempt history, without a console session against production.
- Classify before replaying. Fix the cause first for anything but a transient failure, or you are re-running a known failure at scale.
- Replay at a controlled rate, in batches, to a target you are watching, with the ability to stop after the first batch.
- Verify that replay is safe end to end: the consumer must be idempotent for these messages specifically, including any partially applied side effects.
- Write a runbook per DLQ: what these messages are, what business impact a stuck one has, how to inspect, how to replay, and when to discard (Runbooks).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A per-queue DLQ with an alert and an owner contains the damage to the affected stream. A blind bulk replay does not: it can duplicate side effects across every tenant in the backlog at once.
What can go wrong
- Bulk replay re-fails everything and doubles the dead letter count, along with the noise.
- Bulk replay succeeds and produces duplicate side effects, because the original attempt had completed the external call and failed after it.
- The DLQ alert exists and routes to a channel nobody watches, which is the same as no alert with more paperwork (Alert Fatigue).
- Messages are discarded to clear the alert, destroying work and evidence at once.
- The DLQ itself fills or expires, and the failure of the failure path is unmonitored.
- "Messages in the DLQ have been handled." They have been *removed from the retry loop*. Nothing about them is resolved.
- "Dead letters mean bad messages." Most are valid work that failed while a dependency was down.
- "We can just replay everything." Only if the consumer is idempotent, the cause is fixed, and the rate is controlled. Otherwise you are amplifying the incident.
- "Zero dead letters means the system is healthy." It often means dead-lettering was never configured and failures are being retried forever or dropped (Operating Queues and Scheduled Work).
- "The platform team owns the DLQ." They own the infrastructure. The unprocessed business work belongs to whoever owns the messages.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A dashboard per DLQ showing count, age of the oldest message, and arrival rate — with zero as the expected steady state.
- An owning team recorded for each DLQ, and an alert route that has been tested end to end.
- A recorded instance of inspection and replay, showing what was found and what happened, so the procedure is known to work.
- Consumer idempotency demonstrated for the replay path, not assumed.
- A replay cannot be undone once the messages are consumed and side effects have happened. The batch-and-watch discipline is the rollback: stop after the first batch if it is behaving unexpectedly.
- Before any bulk operation, copy the dead letters somewhere durable. That copy is what lets you retry the recovery itself.
- Discarding messages is final. It is sometimes correct — work that is no longer meaningful — but it should be a recorded decision with a count, not a cleanup action.
- Automate: alerting on the first message, age and count metrics, capture of failure context, an inspection interface, and rate-limited batch replay tooling.
- Automate replay for a narrowly defined class only — messages that failed on a dependency that has since recovered, where the consumer is verified idempotent.
- Keep human: classification, the decision to replay after a bad deploy, and the decision to discard. Automatic blind replay is a way to re-run a failure at scale (The Automation Trap).
- Alerting on the first message is noisier than a threshold and is the point: a threshold means agreeing in advance to lose some work silently.
- Rate-limited replay takes longer to clear a backlog and protects the system that failed in the first place.
- Long DLQ retention keeps recovery possible and keeps failed customer data around longer, which is a privacy consideration as well as a storage one.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- TOOL-SPECIFICManaged queues typically provide a real dead letter queue with a configurable maximum receive count; log-based systems such as Kafka have no built-in equivalent, so the consumer must produce failed records to a separate topic itself, and the retention and replay semantics are then whatever you built. Verify which of the two you have.
- GENERALThe four requirements — alert, inspection, replay strategy, owner — apply to any terminal failure store, including a failed-jobs table in a database or a bucket of rejected files.
- ORG-SPECIFICWho owns dead letters is an organisational decision that determines whether they are ever looked at. Platform-owned queues with product-owned messages is the arrangement in which nobody acts.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — why a message can be both delivered and unprocessed, and why the consumer, not the broker, is where exactly-once processing is achieved.