The question this answers
I have more work than one worker can do. What does a queue actually give me, and what does it take away?
Each enqueued task is delivered to at least one consumer, and in the absence of failure is processed by exactly one. Under failure it is processed at least once. No ordering is guaranteed across tasks once more than one consumer is active, and none across retries even with a single consumer.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A worker knows the contents of the task it holds and that the broker granted it that task. It does not know whether another worker also holds it (it may, after a timeout expiry), how many tasks remain, whether it is the fastest or slowest worker, or whether the task it is about to complete was already completed by a predecessor that crashed after doing the work but before acknowledging.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Competing consumers is the whole pattern
The producer does not address a worker. It puts a task in a queue, and N workers all pull from that same queue. The broker hands each task to exactly one of them. Adding capacity is therefore purely a matter of starting more workers, with no configuration change on either side — that property is the entire reason the pattern is popular.
Note what this requires: the workers must be interchangeable. Any worker must be able to handle any task. The moment a task needs a specific worker (because that worker holds session state, or a local file, or a warm cache for a particular customer), you no longer have a work queue — you have a routing problem wearing a queue’s clothes, and the queue will fight you.
The throughput ceiling is set by the slowest of three things: the producer’s enqueue rate, the broker’s dispatch rate, and the aggregate worker rate. Only the last is trivially scalable, which is why "add workers" stops helping abruptly once one of the others binds.
The claim lifecycle: why a task is not simply "sent"
A naive queue would delete a task when it hands it out. Then a worker crash loses the task permanently. So real queues do not hand out messages — they lease them. The task is marked as claimed and made invisible to other consumers for a bounded time, and only an explicit acknowledgement removes it for good. See Acknowledgement: The Two-Line Protocol That Decides Your Delivery Semantics for the ack half and Visibility Timeout: The Message Is Hidden, Not Yours for the lease half.
This is the crucial structural difference from a log. A work queue’s storage shrinks as work completes; a message that has been acked is gone and cannot be re-read. You cannot replay yesterday’s tasks, because there is no yesterday — the queue is a mutable collection with a claim protocol, not a record of what happened. The Log Is Not a Queue makes the opposite choice and everything downstream differs.
Because completion is destructive, a work queue also cannot support two independent consumers of the same task. If billing and analytics both want to see every order, a work queue gives one of them each order at random. That is the Queue or Pub/Sub: Answer the Question in One Sentence decision, and getting it wrong produces a bug that looks like "we are losing about half our events".
t=0 available visible to all consumers t=1 claimed leased to worker-7, invisible, deadline t=31 t=4 claimed worker-7 still processing t=31 available lease expired — worker-7 said nothing, task is offered again t=32 claimed leased to worker-2, deadline t=62 <-- worker-7 may STILL be running t=40 deleted worker-2 acked. worker-7's later ack, if it comes, is a no-op or an error.
What scaling out costs you
One worker gives you FIFO processing for free. Two workers destroy it, and no configuration flag brings it back — concurrency and global ordering are mutually exclusive over a shared queue. If a downstream state machine depends on Created being processed before Updated, a two-worker pool will eventually process them in the wrong order and produce an update to a row that does not exist yet.
The usual fixes are all forms of narrowing the ordering requirement rather than restoring global order: make handlers commutative so order stops mattering; make them idempotent and self-healing so an out-of-order arrival is retried until it fits; or route related tasks to the same consumer so you buy ordering per key rather than globally. The third is exactly what a A Topic Is Not One Log: Ordering Lives Inside a Partition does structurally, and it is why log-based brokers won so much ground from classic queues.
Scaling out also converts a per-task failure into a fleet-wide one if the tasks are not independent. Ten workers all hammering the same downstream database turn a queue drain into an outage of the thing you were draining into — see Backpressure Is a Signal That Has to Travel — and Reach Someone Who Can Slow Down and Performance’s queueing for what happens next.
| Lever | Buys | Costs |
|---|---|---|
| More worker processesassumption | Linear throughput until a downstream binds | Global ordering; downstream load multiplies |
| Higher prefetch per workertypical | Fewer broker round trips, better throughput | Head-of-line blocking inside one worker; worse rebalancing on crash |
| Concurrency inside one workertypical | Throughput without more processes | Ack bookkeeping gets subtle; one hung task can hold a lease |
| Separate queue per task typetypical | Isolation — a slow type cannot starve a fast one | More queues to operate; capacity is no longer pooled |
| Priority queuestypical | Urgent work jumps ahead | Starvation of low priority; the broker’s priority support is often weak |
Head-of-line blocking and the shared-queue trap
A single queue mixing a 50 ms task type with a 90-second task type behaves badly in a way that is invisible in averages. When a burst of slow tasks arrives, every worker is occupied for 90 seconds, and the fast tasks — which the product considers latency-sensitive — sit behind them. Queue depth may be small; queue *age* for the fast type is catastrophic.
Prefetch makes it worse and is a common accidental self-inflicted wound. If each worker prefetches 50 messages, a worker that grabs 50 slow tasks holds them all, invisible to its idle peers. You have re-created head-of-line blocking inside a single consumer, and the broker cannot help because from its point of view those messages are being worked on.
The fix is almost always separation rather than tuning: distinct queues per workload class, with distinct worker pools and distinct alerts. This is Bulkheads: Buying Independence by Giving Up Utilisation applied to messaging, and it is the difference between one workload degrading and all of them degrading together.
Key points
- A work queue distributes each task to exactly one of N interchangeable workers; capacity scales by starting more workers.
- Tasks are leased, not sent: claimed and hidden, then destroyed on acknowledgement. Completion is destructive, so replay is impossible.
- One worker gives FIFO; two workers do not, and no setting restores it. You must narrow the ordering requirement instead.
- A single queue for heterogeneous workloads produces head-of-line blocking that averages hide; prefetch amplifies it inside one consumer.
- A work queue cannot serve two independent consumers of the same message — that is what pub/sub or consumer groups over a log are for.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The producer enqueues a task; the broker persists it and marks it available.
- • An idle worker requests work (pull) or the broker pushes to a worker with spare prefetch capacity.
- • The broker marks the task claimed and starts a lease timer, making it invisible to every other consumer.
- • The worker processes the task, performing whatever side effects it entails.
- • The worker acknowledges. The broker deletes the task; it is now unrecoverable.
- • If the lease expires without an acknowledgement, the broker returns the task to available and it may be claimed by anyone, including while the original worker still runs.
- • The worker crashes after doing the side effect and before acknowledging — the task is redelivered and the side effect happens twice.
- • Processing outlives the lease and a second worker starts the same task concurrently.
- • The acknowledgement itself is lost in the network; the broker never sees it and redelivers.
- • A worker with a large prefetch is killed, and all its held messages become invisible until their leases expire.
- • A poisoned task fails on every worker in turn and consumes the entire pool’s attention — see Poison Messages: The One That Fails Every Time, Forever.
- • Duplicate side effects at scale: the operator sees two charges, two emails or two shipment records, correlated with a worker restart or a deploy. The application logs show one successful run per worker, both correct in isolation.
- • Head-of-line stall: p50 processing time is flat and healthy, p99 end-to-end latency for a specific task type has grown to minutes. Queue depth looks fine. Only per-type queue age reveals it.
- • Uneven worker utilisation from prefetch: half the pool is idle while the queue is deep, because a few workers hold large prefetch buffers of slow tasks. CPU across the fleet averages 30% during an apparent capacity crisis.
- • Silent capacity loss on deploy: rolling restarts kill workers mid-task, and every in-flight message waits a full visibility timeout before anyone picks it up. Drain rate drops to near zero for the timeout duration and then recovers, producing a sawtooth nobody can explain.
- • Downstream collapse on drain: after an outage the pool scales up to clear a backlog and takes the shared database down with it, converting a queue incident into a site incident.
- • Between workers: none. That is the design. Workers never talk to each other, which is why the pattern scales so cleanly.
- • Between worker and broker: the claim is a lease, and a lease is the weakest useful form of distributed coordination — see Leases: Authority With an Expiry Date. It buys mutual exclusion only for as long as the clock and the timeout are both trusted.
- • The broker is a single logical arbiter for who holds what. That centralisation is what makes "exactly one worker" achievable at all, and it is also the component whose availability now bounds the whole pipeline.
- • A task acknowledged by the broker is gone; a task not acknowledged will be retried. There is no third state, so no task is silently lost by the queue itself.
- • The at-most-once property degrades to at-least-once the moment anything fails. Design for at-least-once always; the failure-free case is not the one that matters.
- • Ordering guarantees, already weak, vanish entirely across a retry: a redelivered task will be processed after tasks that were enqueued after it.
- • Detect: alert on per-queue backlog age, and separately on consumer count reaching zero. Depth alone will mislead you in both directions.
- • Contain: isolate the failing workload class into its own queue rather than scaling the shared pool, so recovery does not require the whole fleet to behave.
- • Recover: scale workers to drain, with a rate limit toward shared downstreams so the drain does not become the next incident.
- • Reconcile: for duplicate-sensitive effects, run the deduplication check after any incident involving worker restarts — duplicates are expected, not exceptional.
- • Verify: backlog age back to baseline, DLQ empty or triaged, and no residual invisible messages waiting on expired leases.
- • Backlog age per queue, split by task type if types share a queue (and let that split motivate splitting the queue).
- • In-flight / invisible message count — a persistently high number with low throughput means leases are being held by dead workers.
- • Redelivery count distribution. A rising tail means tasks are outliving their visibility timeout, not that the system is failing.
- • Worker utilisation variance across the pool. High variance with a deep queue is the prefetch signature.
- • Ack latency: time from delivery to acknowledgement, per task type. This is the number your visibility timeout must exceed.
- • Homogeneous, independent tasks where any worker can do any task and order does not matter: image resizing, sending notifications, generating exports.
- • Bursty arrival with expensive-to-scale processing, where a buffer converts a rate spike into a latency spike.
- • Work that must survive a worker crash and be retried without the producer being involved.
- • Tasks with a strict global ordering requirement — you will either run one worker (no scaling) or be subtly wrong.
- • Multiple independent consumers needing the same event; a work queue will give each event to exactly one of them.
- • Tasks that need replay or audit after completion. The queue deletes on ack; the history you want does not exist.
- • Very short tasks at very high rate, where the per-message broker round trip dominates the actual work.
- • A database-backed job table with
SELECT ... FOR UPDATE SKIP LOCKED. You get the same claim semantics inside your existing transaction boundary — which removes the dual-write problem entirely — at the cost of database load and a lower ceiling. - • A A Topic Is Not One Log: Ordering Lives Inside a Partition with consumer groups: per-key ordering, replay, and multiple independent consumers, at the cost of a fixed parallelism ceiling and heavier operations.
- • Direct synchronous invocation with a bounded worker pool inside the caller, when the work is short and losing it on restart is acceptable.
- • A scheduled batch scan, when latency tolerance is measured in minutes and you would rather have zero new failure modes than low latency.
One task, one worker, competing consumers
What people believe, and what is true
A FIFO queue means my tasks are processed in order.
It means they are *dequeued* in order. With N concurrent workers, completion order is a function of task duration, and a retry moves a task arbitrarily far back.
Adding workers always increases throughput.
Only until the broker, the producer, or a shared downstream binds. Past that point extra workers add contention and can reduce throughput.
If the worker crashed, the task was not done.
The crash may have occurred after the side effect and before the ack. That is the single most common source of duplicates in production.
The queue is empty, so all the work is finished.
Empty means nothing is *available*. Messages claimed by workers are invisible, and messages in the DLQ are not in the queue at all.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Many workers pull from one queue; each task goes to one of them. Scale by adding workers. Lose ordering when you do, and expect every task to be processed at least once.
Practical
Set the visibility timeout above your p99 processing time, keep prefetch low for heterogeneous work, separate workload classes into separate queues, and make every handler idempotent before you scale past one worker. Alert on backlog age and on in-flight count, never on depth alone.
Advanced
A work queue is a distributed mutual-exclusion protocol over a mutable set, implemented with leases. Every one of its interesting properties follows from that: exclusivity holds only until a lease expires, so exclusivity is time-bounded and therefore not a safety property under asynchrony; destruction on ack means the structure has no history, so no replay and no second consumer; and because the broker is the single arbiter of the claim, the pattern inherits that arbiter’s availability as a hard ceiling. Reaching for a log instead is not a performance decision — it is choosing a different data structure with different algebra.
Apply it
- 🔧 Build a queue consumer, then kill it with SIGKILL between side effect and ack. Observe the duplicate. Then make the handler idempotent and repeat.
- 🔧 Mix a 20 ms and a 30 s task type in one queue with prefetch 100, and measure the p99 latency of the fast type as the slow type’s rate rises.
- ⚡ A rolling deploy makes throughput collapse for exactly the length of your visibility timeout, every deploy. Explain the mechanism and fix it without changing the timeout.
- ⚡ Your pool scales from 5 to 200 workers to drain a backlog and the primary database falls over. Design the drain so this cannot happen.
- 💬 You have a FIFO queue and three workers. Is processing ordered? Why not, and what would you change if a downstream required order?
- 💬 Half your workers are idle and the queue is 50,000 deep. Give me three hypotheses in order of likelihood.
- 💬 A task takes 5 minutes and your visibility timeout is 30 seconds. Describe exactly what happens.