The question this answers
My handler sometimes takes longer than expected. What does the broker do about it, and what does my code have to survive?
A claimed message is invisible to other consumers for the duration of the visibility timeout, and no longer. That is a time-bounded lease, not a lock: exclusivity holds only while the timer runs, and the broker will not check whether the original consumer is still working before offering the message to someone else.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The worker knows it was handed the message and, if it looks, roughly when its lease expires by its own clock. It does not know whether the broker still considers the lease valid, whether the message has already been redelivered to a peer, or whether its eventual ack will be accepted or silently discarded. The broker, in turn, knows only that no ack arrived — it cannot distinguish a dead worker from a slow one, which is Crashed or Just Slow: The Distinction You Cannot Make in its most operationally expensive form.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
A lease is not a lock
The broker hands out a message with an expiry attached. Until that expiry, no other consumer will be offered it. After that expiry, the message is available again — and the broker does no check of any kind on the original consumer first. It cannot: a worker that is slow and a worker that is dead look identical from outside, and waiting to find out would mean waiting forever.
So the message can be worked on by two consumers simultaneously. Both read the same payload, both perform the side effect, and both may eventually try to ack. This is not the same as the ordinary at-least-once duplicate, where two attempts are sequential and the first one finished. Here they overlap, which means a handler that would be perfectly safe when run twice in sequence — read balance, add 10, write balance — can still corrupt state, because the two runs interleave.
The practical consequence is uncomfortable and worth stating plainly: your handler must be safe under concurrent execution with itself, not merely under repetition. Idempotence via a dedup row helps only if the dedup insert and the effect are in the same transaction, so that one of the two racing transactions loses. A dedup check followed by a separate write is exactly the check-then-act race that Distributed Locks: What They Are Actually For exists to discuss.
Choosing the timeout: both directions are bad
Set the timeout too short and you get the picture above on every slow message — spurious concurrency, wasted work, and a redelivery count that climbs while throughput falls. Set it too long and every genuine crash costs you that entire duration before anyone else can pick up the work, which turns a routine restart into a multi-minute stall in drain rate.
The starting rule is: timeout comfortably above your p99 processing time, then measure the redelivery rate and adjust. p99, not the mean — the mean is irrelevant here, because the timeout only ever interacts with the slow tail. A handler averaging 200 ms with a p99 of 45 seconds needs a timeout sized for 45 seconds, and if that seems absurd it is telling you something true about the workload.
The better answer for genuinely long work is to stop relying on a static number. Most brokers let a consumer extend the lease while it works — a heartbeat that says "still alive, give me another 30 seconds". This converts a guess about worst-case duration into a liveness signal, and it is the right shape: the timeout should reflect how long you are willing to wait after a crash, not how long the work takes.
| Setting | On a slow message | On a crashed worker | Operator sees |
|---|---|---|---|
| Timeout << p99 processingprotocol | Concurrent duplicate execution | Fast recovery | Rising redelivery count, falling throughput, rising CPU |
| Timeout ≈ p99 processingtypical | Occasional duplicate on the tail | Recovery in ~p99 | A small, stable redelivery rate |
| Timeout >> p99 processingprotocol | No spurious duplicates | Long stall before rework | Drain rate drops to zero after each deploy for the timeout duration |
| Heartbeat extension while workingtypical | No spurious duplicates | Recovery in one heartbeat interval | Low redelivery rate and fast recovery — the good quadrant |
| Extension without a hard capassumption | Safe | Safe | A hung handler holds the message forever — a stuck message with no DLQ path |
The fencing question: can you tell you lost the lease?
A worker whose lease expired is doing work it no longer has the right to do. The clean solution to this class of problem is a a fencing token (Fencing Tokens: Making the Stale Actor Safe, Not Just Unlikely): the lease carries a monotonically increasing number, the worker presents it when writing, and the store rejects writes carrying a stale token. That is how a lease is made safe against a slow holder.
Message brokers generally do not give you this. Some hand back a receipt handle that becomes invalid after expiry, which lets the *ack* fail — but the ack failing happens after the side effect has already been committed, so it detects the problem rather than preventing it. That is worth having for observability and worth nothing for correctness.
So in practice the fence must be built in the destination store: a unique constraint on the message identity, a conditional update on an expected version, or a state machine that rejects a transition it has already made. All three make the second writer lose deterministically, which is what the fencing token would have done. Without one of them, the visibility timeout is a lease with no enforcement, and The Stale Lock Holder: A Paused Process Does Not Know It Was Paused describes exactly what you have built.
1handle(msg):2 deadline = now() + msg.leaseDuration3 attempts = 04 5 start heartbeat every leaseDuration/3:6 if attempts++ > MAX_EXTENSIONS: // hard cap: a hung handler must7 stop heartbeat // eventually reach the DLQ8 else9 broker.extendLease(msg.receipt, leaseDuration) // may fail if expired10 11 result = doWork(msg)12 13 // The fence lives in the STORE, not the broker: a second worker's write14 // must lose deterministically even if both handlers ran to completion.15 db.transaction:16 inserted = insert into processed(message_id) values (msg.id) // unique17 if not inserted: return // someone else already committed18 apply(result)19 20 broker.ack(msg.receipt) // may be rejected; the transaction already decidedSecond-order effects on the pool
Spurious redelivery is not merely duplicated work; it is duplicated *load*. When the timeout is too short, every slow message becomes two slow messages, then four. Throughput drops, which makes each message slower, which causes more expiries. This is a positive feedback loop and it is the messaging-layer twin of One Retry per Tier Is Not One Retry — It Multiplies.
It also breaks the attempt counter that your A Dead-Letter Queue Is a Workflow, Not a Bin policy depends on. A message redelivered five times because it kept outliving the timeout hits maxReceiveCount and is dead-lettered, even though it never failed — it was simply slow. Operators then find a DLQ full of perfectly valid messages and, reasonably but wrongly, conclude the DLQ is noise.
Finally, in-flight messages held by a stalled worker are invisible until their leases expire. A deadlocked consumer with a large prefetch can hide thousands of messages from the rest of the pool for the full timeout. The metric that reveals this is in-flight count against ack rate, and it belongs on the same dashboard as backlog age.
Key points
- A visibility timeout is a time-bounded lease, not a lock. Exclusivity ends on a timer, with no check on whether the holder is still working.
- Expiry mid-processing produces *concurrent* duplicate execution, which is strictly harder to survive than sequential redelivery.
- Size the timeout against p99 processing time, not the mean; better still, heartbeat to extend the lease while working, with a hard cap.
- The broker gives you no fencing token. The fence has to live in the destination store as a unique constraint or conditional write.
- Too-short timeouts feed back on themselves: duplicated work slows processing, which causes more expiries.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The broker delivers a message, marks it in-flight, and starts a timer of length
visibilityTimeout. - • The message is not offered to any other consumer while the timer runs.
- • The consumer processes. It may optionally request an extension, resetting the timer.
- • If an ack arrives before expiry, the message is removed and the timer is discarded.
- • If the timer expires first, the message returns to available and its receive count increments — the original consumer is not notified.
- • Once the receive count exceeds the configured maximum, the message is routed to the dead-letter destination on its next expiry.
- • Processing legitimately outlives the timeout and a second worker begins the same message.
- • The heartbeat thread is starved (event loop blocked, GC pause) and the lease expires despite the worker being alive.
- • The extension call itself fails or arrives after expiry, and the worker continues believing it holds the lease.
- • An unbounded extension loop lets a hung handler hold a message indefinitely, so it never reaches the DLQ.
- • A stalled consumer holds a large prefetch of leased messages, hiding them from an otherwise idle pool.
- • Concurrent duplicate execution: the operator sees two overlapping runs of the same job in the logs on different hosts, with interleaved timestamps rather than sequential ones, and a corrupted counter that neither run explains.
- • DLQ full of valid messages: the dead-letter queue fills with messages that never threw an exception. Their receive count reached the maximum purely through timeout expiry, and the team learns to ignore the DLQ.
- • Post-deploy drain stall: after every restart, throughput is zero for exactly the visibility-timeout duration, then recovers. The graph is a perfect square wave and nobody connects it to the setting.
- • Redelivery feedback loop: redelivery rate and CPU climb together while completed-messages-per-minute falls. Adding workers makes it worse, because the extra load lengthens processing and causes more expiries.
- • Invisible backlog: queue depth is 0, in-flight is 8,000, ack rate is 0. The pool appears idle and the queue appears empty while nothing at all is being completed.
- • The lease is the coordination primitive, and it is the weakest useful one: mutual exclusion that expires on a clock rather than on a fact. See Leases: Authority With an Expiry Date for the general treatment.
- • Its safety depends on an assumption about the *maximum* time processing can take — an assumption no asynchronous system can enforce, which is why the guarantee is availability-flavoured rather than safety-flavoured.
- • Genuine mutual exclusion would require fencing at the resource, which moves the coordination out of the broker and into the store where the effect lands.
- • No message is lost by expiry: the guarantee preserved is availability of the work, not exclusivity of the worker.
- • Exclusivity is guaranteed only for the timeout duration, and is therefore not a safety property under asynchrony — a paused process can always exceed it.
- • Once two workers hold the message, correctness depends entirely on the destination store, because the broker has no further say.
- • Detect: redelivery-rate and receive-count distribution, plotted against ack-latency p99. Redeliveries with no handler errors is the signature.
- • Contain: raise the timeout or enable heartbeat extension immediately; both stop the feedback loop faster than adding capacity.
- • Recover: expect and absorb the duplicates already created; do not attempt to purge them from the queue, since you cannot tell which are duplicates.
- • Reconcile: check the destination store for the effects of concurrent runs — interleaved increments, duplicate rows, out-of-order state transitions.
- • Verify: redelivery rate back to a low stable baseline, and DLQ arrivals correlating with real handler errors rather than with slow messages.
- • Ack latency p99 versus the configured timeout — the ratio is the real health metric, and it should not be near 1.
- • Receive-count distribution: a growing shoulder at 2 and 3 means the timeout is too tight before it becomes a DLQ problem.
- • In-flight count against ack rate, which reveals leases held by processes that are not making progress.
- • Lease-extension success and failure counts, if the broker exposes them; failures mean workers are continuing without a lease.
- • DLQ arrivals split by cause — handler exception versus receive-count exhaustion. Conflating them makes the DLQ uninterpretable.
- • Any pull-based queue: it is what makes crash recovery automatic without a supervisor watching workers.
- • Workloads with predictable, bounded processing time, where a single sensible number covers the tail comfortably.
- • With heartbeat extension, workloads with a long and unpredictable tail — the extension turns duration into liveness.
- • Handlers whose duration varies by orders of magnitude. No single value is right, and every value is wrong in one direction.
- • Non-idempotent side effects with no fence in the destination store — the timeout will eventually manufacture the concurrency that breaks them.
- • Very long tasks without extension support, where the required timeout makes crash recovery unacceptably slow.
- • Heartbeat lease extension with a hard cap — the same mechanism, driven by liveness instead of a guess.
- • Split the long task into short steps with a state machine, so no individual message needs a long lease. This is the general fix and it also improves observability.
- • A claim row in your own database with an explicit
claimed_untiland a fencing version, when you need the fence and the broker will not give you one. - • A A Topic Is Not One Log: Ordering Lives Inside a Partition, where there is no per-message lease at all — the exclusivity unit is the partition assignment, which shifts the problem to Rebalancing: Everyone Stops So the Partitions Can Move rather than removing it.
The message is hidden, not yours
What people believe, and what is true
The message belongs to my worker until I ack it.
It belongs to your worker until the timer expires. After that it belongs to whoever asks next, and nobody tells you.
Redelivery means the first attempt failed.
It means the first attempt did not ack in time. It may be running right now, about to succeed.
Idempotent handlers make the timeout harmless.
Idempotent under *repetition* is not the same as safe under *concurrency*. Read-modify-write is idempotent in sequence and wrong when interleaved.
Just set the timeout very high.
Then every crash costs that duration before the work is retried, and a hung handler holds its message for the same period. You moved the cost onto recovery time.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
The broker hides a delivered message for a fixed time instead of removing it. If your handler is still running when that time runs out, someone else gets the same message and both of you are working on it.
Practical
Set the timeout above p99 handler duration, extend the lease by heartbeat for long work with a hard cap, and put a unique constraint or conditional write in the destination store so the second writer loses. Alert on receive-count distribution and on ack-latency-to-timeout ratio.
Advanced
A visibility timeout is a lease, and a lease in an asynchronous system provides no safety guarantee at all — only a liveness one. There is no bound on how long a process can be paused, so there is no timeout value for which "the holder has stopped" is implied by "the timer expired". The literature’s answer is fencing: make the resource reject operations from an expired lease, so safety is enforced where the effect lands rather than where the lease was granted. Brokers do not implement this, so the fence is yours to build, and a messaging system without one has exactly the correctness properties of a distributed lock without one.
Apply it
- 🔧 Force a lease expiry mid-handler and demonstrate two workers committing to the same row. Then add a unique constraint and show one of them losing.
- 🔧 Instrument ack latency and receive count, then lower the timeout until the feedback loop starts. Note the throughput curve on the way down.
- ⚡ Throughput collapses and CPU rises after a downstream dependency slows by 3x. No code changed. Explain the mechanism and the two-line fix.
- ⚡ A hung handler holds a message forever because lease extension has no cap. Design the cap and decide what should happen at the limit.
- 💬 A message takes 90 seconds and the visibility timeout is 60. Walk me through the timeline.
- 💬 Why is a visibility timeout not a distributed lock?
- 💬 Your DLQ is full of messages that never raised an error. What happened?