The question this answers
This message fails every single time. How long should I keep trying, and what stops it from taking the pipeline down?
A bounded retry policy guarantees that a message is attempted at most N times before being routed elsewhere. It does not guarantee that N attempts are enough for a transient failure, nor that a message failing N times is genuinely unprocessable. Both misclassifications are possible and neither can be eliminated.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The consumer knows this attempt threw, and — if the broker supplies a receive count — roughly how many attempts preceded it. It does not know whether the cause is permanent (malformed payload, deleted referent) or transient (dependency restarting, lock contention). The exception type is a hint chosen by whoever wrote the code, not a fact about the world, and treating it as authoritative is how transient failures get dead-lettered during an outage.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Two failures that look identical and need opposite responses
A handler throws. There are exactly two useful categories. Transient: the message is fine, the world was briefly wrong — a database failover, a rate limit, a network blip. The correct response is to retry, ideally with Without Jitter, Every Client That Failed Together Retries Together, and it will succeed. Permanent: the message itself cannot be processed — a required field is missing, the referenced entity was deleted, the payload is a schema version this consumer never learned. No number of retries will help.
The failure surfaces identically: an exception and a nack. If you treat everything as transient you build an infinite loop; if you treat everything as permanent you dead-letter your entire stream the first time a dependency hiccups. Neither default is safe, and the correct behaviour differs per message on the same queue at the same moment.
The workable posture is: classify what you confidently can, and let a retry budget decide the rest. Deserialisation failures and validation errors are permanent with high confidence — fail them fast, on the first attempt, straight to the DLQ. A 503 from a dependency is transient with high confidence — retry generously. Everything else is unclassified, gets a bounded number of attempts, and ends up in the DLQ if it exhausts them. The DLQ is where uncertainty goes, which is exactly why it needs a human.
| Symptom | Class | Right response | Cost of the wrong response |
|---|---|---|---|
| Payload fails to deserialiseprotocol | Permanent | DLQ on attempt 1 | Infinite loop burning a consumer slot |
| Schema validation errorprotocol | Permanent | DLQ on attempt 1 | Same, plus log spam that hides real errors |
| Referenced entity not foundassumption | Ambiguous — often a race with a slower upstream | Bounded retry with backoff, then DLQ | Dead-lettering messages that would have worked in 2 seconds |
| Dependency 503 / timeouttypical | Transient | Retry with backoff and jitter, generous budget | Dead-lettering the whole stream during a dependency outage |
| Rate limited (429)typical | Transient with a stated delay | Honour Retry-After; slow the consumer, do not just retry | A retry storm that extends the rate limit |
| Handler OOM / process killedassumption | Looks transient, often permanent | Bounded attempts; a payload-size guard | A single huge message repeatedly killing workers — the crash loop |
The crash loop is the dangerous one
The worst poison message does not throw an exception — it kills the process. A payload that triggers an out-of-memory condition, an infinite loop, or a stack overflow takes the worker down before any handler-level retry counter is incremented. The broker sees no ack, the lease expires, the message is redelivered, and it kills the next worker.
This escalates in a way ordinary poison does not. Every worker in the pool dies in turn, the orchestrator restarts them, they pull the same message, and they die again. Throughput goes to zero, the queue backs up, and the restart loop looks like an infrastructure problem rather than a data problem. Teams routinely spend an hour on the wrong hypothesis here.
The defences are cheap and are almost never in place before the first incident: cap payload size at the producer and reject oversized messages at the consumer before parsing; run the handler with a hard wall-clock deadline so a hang becomes a catchable failure rather than a hang; and rely on the broker’s receive count rather than an in-process attempt counter, because an in-process counter dies with the process. The receive count is the only counter that survives the thing you are defending against.
14:02:11 worker-3 started, subscribing to orders-q 14:02:12 worker-3 received msg 8f21c (receiveCount=1, 42 MB) 14:02:19 worker-3 <no further output> 14:02:19 k8s worker-3 OOMKilled, restarting 14:02:41 worker-1 received msg 8f21c (receiveCount=2, 42 MB) 14:02:48 k8s worker-1 OOMKilled, restarting 14:03:10 worker-2 received msg 8f21c (receiveCount=3, 42 MB) ... Symptom on dashboards: pod restart rate spiking, throughput 0, error rate 0. There is no exception anywhere, because nothing lived long enough to log one.
Head-of-line blocking: why a log suffers more than a queue
In a work queue, a poison message occupies one consumer at a time and the rest of the stream flows past it. It is a slow leak: wasted capacity, noisy logs, a rising receive count. Unpleasant, bounded, and visible.
In a A Topic Is Not One Log: Ordering Lives Inside a Partition the same message is a wall. Position is a per-partition offset, and an offset can only move forward past a message, never around it. A message that will not process therefore halts every message behind it in that partition — which, if you partitioned by user_id, means every event for a large slice of your users stops. Aggregate lag looks fine because the other partitions are healthy, so the alert does not fire.
This is why log-based pipelines need an explicit escape hatch built before it is needed: on failure, publish the message to a retry topic (often several, with increasing delays) or straight to a DLQ topic, commit the offset, and let the partition continue. You are deliberately giving up ordering for that one message in order to preserve liveness for everything else — a trade worth making consciously, and one that is much harder to make at 3am.
Budgets, not counters
A fixed maxAttempts = 5 is a reasonable default and a poor policy on its own, because attempts are not the resource that matters. Five attempts in 200 milliseconds is nothing; five attempts over four hours is a very different statement about how hard you tried. What you actually want to bound is time and cost, and to spread attempts across a window long enough for a plausible transient cause to resolve.
The shape that works: exponential backoff with jitter, an absolute deadline (give up after 30 minutes regardless of attempt count), and an immediate DLQ path for confidently permanent errors that skips the budget entirely. Add a fleet-level retry budget so that during a dependency outage the consumer pool does not spend 100% of its capacity retrying — see Cap Retries as a Fraction of Traffic, Not as a Count per Request, and Performance’s retry-storms for what happens when it does.
One caution that surprises people: a message can hit its attempt limit without ever failing, because Visibility Timeout: The Message Is Hidden, Not Yours expiry also increments the receive count. If your DLQ is full of messages with no logged exception, the retry policy is measuring slowness, not failure, and the fix is in the timeout rather than in the policy.
Key points
- Transient and permanent failures surface identically; the right response is opposite, and no default is safe for both.
- Classify confidently where you can (deserialisation and validation are permanent; 503 and 429 are transient) and let a bounded budget handle the rest.
- The dangerous poison message kills the process before any in-process counter increments — only the broker’s receive count survives that.
- In a work queue a poison message costs one worker; in a log partition it blocks every message behind it, and aggregate lag hides it.
- Bound retries by elapsed time and cost, not attempt count alone, and remember that timeout expiry also increments the attempt counter.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A handler fails. The consumer either nacks explicitly or lets the lease expire.
- • The broker increments the receive count and, after any configured delay, makes the message available again.
- • The consumer classifies the failure: permanent errors are routed immediately to the dead-letter destination; transient errors are retried with backoff.
- • Each retry is spaced by an exponentially growing, jittered delay, bounded by an absolute deadline.
- • When the receive count exceeds the maximum, or the deadline passes, the message is moved to the dead-letter destination and the main stream continues.
- • In a log, this final step is an explicit publish to a retry or DLQ topic followed by an offset commit, because the offset cannot skip in place.
- • A transient failure is misclassified as permanent and dead-lettered during an outage that would have resolved in a minute.
- • A permanent failure is retried forever because no attempt limit was configured.
- • The message crashes the process, so no handler-level classification or counting ever runs.
- • The retry policy consumes the entire consumer pool during a dependency outage, turning degradation into a full stop.
- • The receive count is inflated by lease expiry, and slow-but-valid messages are dead-lettered as poison.
- • Infinite retry loop: the operator sees the same message id in the logs thousands of times per hour, a receive count in the tens of thousands, and a consumer at full CPU delivering zero completed work. Log volume and bill both spike.
- • Consumer crash loop: pod restart rate spikes, throughput is zero, and the error rate is zero because no process survives long enough to report an error. Every dashboard points at infrastructure; the cause is one 42 MB payload.
- • Silent partition stall: one partition’s lag climbs linearly for hours while aggregate lag stays flat. Users whose keys hash to that partition see stale data; nobody is paged.
- • Mass dead-lettering during a dependency outage: the DLQ receives 200,000 valid messages in ten minutes because the handler classified timeouts as permanent. Replay afterwards is possible but the ordering is gone.
- • Poisoned-by-slowness DLQ: the dead-letter queue fills with messages that have no associated exception anywhere, because visibility-timeout expiry drove the receive count to its limit.
- • None is required to retry — which is precisely why an unbounded retry policy is so easy to write and so damaging.
- • A fleet-wide retry budget is genuine coordination: consumers must share a view of how much retry capacity is being spent, usually approximated with a per-consumer token bucket rather than actual agreement.
- • Deciding to skip a message in a log means giving up an ordering guarantee, which is a correctness decision that should be made at design time, not improvised during an incident.
- • With a bounded policy, the pipeline continues to make progress on every message except the poisoned one — liveness is preserved at the cost of that message’s ordering.
- • With an unbounded policy, a single message removes liveness for a whole partition or a share of the pool, indefinitely.
- • A dead-lettered message is not lost — but it is only "not lost" if the DLQ has an owner. See A Dead-Letter Queue Is a Workflow, Not a Bin.
- • Detect: alert on receive-count outliers and on per-partition lag; both fire before the message reaches the DLQ.
- • Contain: get the message out of the hot path immediately — DLQ it, or skip the offset — before diagnosing. Liveness first, forensics second.
- • Recover: fix the handler or the upstream producer, then replay from the DLQ. Expect the replay to arrive out of order relative to the original stream.
- • Reconcile: for a partition that stalled, verify downstream views caught up on the whole backlog and not merely on the newest messages.
- • Verify: DLQ arrivals should correlate with real handler errors. If they correlate with latency instead, the timeout is the bug and the retry policy is innocent.
- • Receive-count distribution per queue, with an alert on the maximum rather than the mean.
- • Per-partition lag for logs, never aggregate — the aggregate is designed to hide exactly this.
- • DLQ arrival rate, split by classified cause: deserialisation, validation, dependency error, attempt exhaustion, timeout exhaustion.
- • Consumer restart rate correlated with message ids — the only cheap way to identify a crash-loop payload.
- • Retry traffic as a fraction of total consumer work, which is the early warning for a retry-driven capacity collapse.
- • Any consumer that will ever receive input it did not produce — which is all of them, eventually.
- • Pipelines where a schema change may reach consumers before they are updated, so old and new payloads coexist.
- • Log-based pipelines, where the absence of a policy is not a slow leak but a hard stop.
- • Aggressive permanent-classification in a system with flaky dependencies dead-letters valid work at scale during exactly the moments you least want to be doing manual recovery.
- • Very high attempt limits with short backoff, which is an infinite loop with extra steps.
- • Elaborate per-error-type classification in a small system, where a single bounded budget plus a monitored DLQ would have been sufficient and far easier to reason about.
- • Validate at the producer so malformed messages never enter the system. The cheapest poison message is the one that was never published.
- • A schema registry with compatibility enforcement, which turns a class of permanent consumer failures into a publish-time error with a stack trace and an owner.
- • Retry topics with tiered delays (5s, 1m, 15m) instead of in-place retry, so the main partition is never blocked and the delay is achieved by scheduling rather than by holding a lease.
- • Park-and-continue: write the failing message to a durable side table and commit the offset immediately, deciding what to do about it out of band. This is a DLQ you built yourself, and it needs the same operational workflow.
The one that fails every time, forever
What people believe, and what is true
Retrying is always safe.
Retrying a permanent failure is an infinite loop, and in an ordered stream it is an outage for everything behind it.
The exception type tells me if it is permanent.
It tells you what the library author decided to throw. A wrapped timeout can arrive as a generic application error, and a validation error can wrap a dependency failure.
maxAttempts = 5 handles poison messages.
It bounds them, if attempts are actually spaced out and if failures actually increment the counter. A process-killing payload increments nothing you control, and timeout expiry increments it without any failure at all.
A poison message only affects itself.
True in a work queue. In a log partition it blocks every message behind it, and the aggregate lag metric will not show you.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Some messages can never succeed. Retrying them forever burns capacity and, in an ordered stream, stops everything behind them. Bound the retries and route the failures somewhere a human will look.
Practical
Fail fast on deserialisation and validation errors. Retry dependency errors with exponential backoff, jitter and an absolute deadline. Cap payload size to prevent crash loops. Use the broker’s receive count rather than an in-process counter. In a log, publish failures to a retry or DLQ topic and commit the offset so the partition keeps moving.
Advanced
The classification problem is undecidable in the general case: whether a failure is permanent depends on the future state of systems you cannot observe. So every policy is a bet, and the design question is which way to be wrong. Erring toward transient costs capacity and blocks ordered streams; erring toward permanent costs correctness in bulk during exactly the outages when manual recovery is hardest. Practical systems bias toward transient with a hard time bound, and then invest in making the DLQ a real workflow — because the DLQ is where an undecidable question gets handed to a human, which is the only oracle available.
Apply it
- 🔧 Publish a message that OOM-kills the handler and observe the pool-wide crash loop. Then add a size guard before parsing and confirm the message reaches the DLQ instead.
- 🔧 Build tiered retry topics with 5s / 1m / 15m delays and show that a failing message never blocks the main partition.
- ⚡ During a 10-minute database failover your consumer dead-letters 200,000 messages. What was wrong with the classification, and how do you replay safely?
- ⚡ One partition has been stalled for three hours and no alert fired. Design the alert, and explain why the one you had did not work.
- 💬 How do you distinguish a poison message from a transient failure? What happens when you are wrong in each direction?
- 💬 A single message is killing every worker in your pool. How do you detect it, and what stops it?
- 💬 Why is a poison message worse in Kafka than in SQS?