The question this answers
When one part of a system creates work faster than another can run it, what exactly must the two sides agree on?
An upload handler enqueues image-resize jobs. Four worker threads dequeue and run them. Peak arrival is about 900 jobs per minute; the pool finishes about 600 per minute.
The queue: its buffer, its item count, and its head and tail positions. The jobs themselves are deliberately *not* shared — ownership transfers on dequeue, and after that exactly one worker touches a given job.
Every enqueued job is dequeued exactly once by exactly one consumer; no consumer ever returns an item from an empty buffer, and no producer ever overwrites an item a consumer has not taken.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The pattern is a contract, not a data structure
Producer/consumer is usually introduced as "a queue between two threads", which hides the actual content. The queue is a data structure; the pattern is a *contract*. It says: the producer never touches a job after handing it over, the consumer never touches a job before receiving it, and the queue guarantees that the handover is indivisible. Ownership transfer is the point — it is what lets you stop reasoning about the job as shared mutable state at all. See Message Passing for the general form of that trade.
The moment you write the queue yourself, the contract stops being free. A ring buffer with items[tail] = job; tail = (tail + 1) % cap is two operations, and two producers can interleave them. The schedule below is the classic loss: both producers read the same tail, both write into the same slot, and one job is gone with no exception, no log line and no dropped-message metric — because from the producer's side both enqueues returned normally.
What makes this a *race condition* rather than merely a data race is that the bug is logical: the ordering of two well-formed operations produces a wrong result. Whether it is *also* a data race depends on the language memory model — in C++ two unsynchronized writes to tail are undefined behaviour, in CPython the interpreter's own locking makes each bytecode indivisible but the two-step sequence is still not. Reasoning About Races: A Method, Not an Instinct and Data Race Is Not Race Condition separate the two ideas properly; Race Conditions in Operating Systems defines them.
| # | Producer 1 | Producer 2 | Consumer | State |
|---|---|---|---|---|
| 1 | read tail (= 7) | · | · | tail=7 slot[7]=empty |
| 2 | · | read tail (= 7) | · | tail=7 slot[7]=empty |
| 3 | write slot[7] = job-A | · | · | tail=7 slot[7]=job-A |
| 4 | · | write slot[7] = job-B | · | tail=7 slot[7]=job-B ✕ job-A is overwritten before any consumer took it — an enqueued job will never be dequeued. |
| 5 | write tail = 8 | · | · | tail=8 slot[7]=job-B |
| 6 | · | write tail = 9 | · | tail=9 slot[7]=job-B slot[8]=stale |
| 7 | · | · | dequeue slot[7] → job-B | head=8 tail=9 |
| 8 | · | · | dequeue slot[8] → whatever was there last cycle | head=9 tail=9 ✕ A consumer returned an item that was never enqueued in this cycle. |
Many producers, a pool of consumers, and where the waiting goes
Scale the shape up and the interesting behaviour is not the fast path, it is the two stalls. When the queue is empty, consumers wait — that is the cheap stall, and it is what you want, because a consumer waiting on an empty queue costs nothing but a parked thread (Condition Variables: Waiting Until a Predicate Is True covers the predicate loop that makes the wait correct, and Lost Wakeups: The Notify That Arrived Before the Wait covers getting it wrong). When the queue is full, producers wait — and that stall propagates backwards into whatever called the producer, which is usually a request handler holding a socket. That is Backpressure, and it is the stall people design around instead of designing *for*.
The timeline shows one busy second at 4 workers against a producer running faster than the pool. Workers are running almost continuously; the producer spends most of it blocked on a full queue. That picture is the healthy one. The unhealthy version of the same load is a queue with no bound, where the producer lane shows no blocked segments at all and the memory graph is the only thing that moves — see Bounded vs Unbounded Queues.
Note what the pool size does and does not buy. Four workers gives you at most four jobs in flight; if the jobs are I/O-bound the CPUs are mostly idle and more workers may help, and if they are CPU-bound more workers than cores mostly buys context switches (Oversubscription, More Threads Is Not More Speed). There is no formula. The signal is queue depth and queue *age* together: depth alone says how much is waiting, age says how long the oldest item has waited, which is the one users feel.
The four decisions the pattern forces on you
Every producer/consumer implementation answers four questions, and the default answers are usually wrong for your system. Capacity: how many items may wait? Full-queue policy: block, reject, drop the oldest, or drop the newest? Ordering: is FIFO required, or would a priority or LIFO order serve users better under backlog? Shutdown: when the producer stops, how does a blocked consumer learn there is nothing more coming, and what happens to items still in the buffer?
The last one is the one that gets shipped broken most often, because it never shows up until the process is asked to stop. A consumer blocked in take() on an empty queue does not return when the producer's thread exits — nothing told it. It sits there until the process is killed, and the deploy shows a 30-second termination-grace timeout every rollout. Draining a Pipeline is that lesson in full; Channels shows the close-the-channel version of the same signal.
The full-queue policy is a product decision wearing an engineering hat. Blocking is right when the producer is a request handler that *should* slow down. Rejecting is right when the caller can retry or degrade — a 429 is more honest than a 30-second hang (Rate Limiting in Architecture, Quotas vs Rate Limits in API Design). Dropping oldest is right for telemetry and live metrics where the newest sample is the valuable one. Dropping newest is right almost nowhere, and is nonetheless the default of several popular executor configurations.
| Policy | Producer sees | Good for | Failure it invites |
|---|---|---|---|
| Block until space | A long, silent pause | Internal pipelines where slowing the producer is the correct answer | Blocked request threads pile up; the stall moves to the connection pool |
| Reject immediately | An explicit error or 429 | Public endpoints, anything with a retrying client | Retry storms if clients retry without jitter |
| Drop oldest | Success, always | Metrics, live dashboards, sensor samples | Silent data loss with no counter unless you add one |
| Drop newest | Success, always | Almost nothing | Newest and most relevant work is discarded first; usually a default, rarely a choice |
| Grow without bound | Success, always | Nothing in production | Backpressure delivered as an OOM kill — see Bounded vs Unbounded Queues |
Key points
- Producer/consumer is an ownership-transfer contract; the queue is just the mechanism that makes the handover indivisible.
- A hand-rolled enqueue is two steps (write slot, advance tail) and interleaving them loses jobs silently — both callers still see success.
- Consumers waiting on an empty queue is the healthy stall; producers waiting on a full one is backpressure, and it propagates into the caller.
- Capacity, full-queue policy, ordering and shutdown are four separate decisions, and the library default answers all four for you badly.
- Queue depth tells you how much is waiting; queue age tells you how long the oldest item waited, which is the number users actually feel.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • The producer constructs a job and hands it to the queue; after the handover it holds no reference and must not mutate it.
- • The queue takes a lock (or uses a lock-free protocol), checks capacity, stores the item, advances the tail, and signals any consumer waiting on "not empty".
- • If the queue is at capacity, the configured policy runs: block on a "not full" condition, reject, or evict.
- • A consumer takes the lock, waits in a predicate loop while the buffer is empty, removes one item, advances the head, and signals "not full".
- • The consumer now solely owns the job and runs it outside the lock — holding a queue lock across job execution serialises the whole pool.
- • On shutdown the producer stops accepting, then signals end-of-stream so blocked consumers wake and observe that no more items are coming.
- • P1 reads tail (7); P2 reads tail (7); P1 writes slot[7] = A; P2 writes slot[7] = B; both advance tail — job A vanished and slot 8 holds a stale entry.
- • C1 checks isEmpty (false); C2 checks isEmpty (false); C1 removes the last item; C2 removes from an empty buffer — a check-then-act race on the emptiness test.
- • Queue is full; P blocks on "not full"; C removes an item and signals "not full" *before* P has entered the wait — the signal is missed and P sleeps until the next removal. That is a lost wakeup (Lost Wakeups: The Notify That Arrived Before the Wait), and it is why the wait must be a loop over the predicate, not an
if. - • Producer finishes and exits; three consumers are parked in
take()on an empty queue; nothing signals them and the process never terminates cleanly. - • The safe schedule: every enqueue and dequeue takes the same lock, so the two-step update is indivisible and the interleavings above cannot be constructed.
- • A correctly implemented bounded blocking queue guarantees: each item is delivered to exactly one consumer, no consumer observes a partially written item, and FIFO order *of enqueue* is preserved.
- • It does NOT guarantee FIFO order of *completion* — four workers finish in whatever order their jobs take, so downstream ordering assumptions break immediately.
- • It does NOT guarantee the job ran. Dequeue is a transfer of custody; if the worker crashes mid-job the item is gone from the queue and not done. Durable delivery is a different problem — see Message Queues in Architecture.
- • It does NOT guarantee fairness between producers or between consumers unless the underlying lock is fair, and most are not by default (Fairness, Starvation).
- • It does NOT bound latency. A bounded queue bounds *memory*; the oldest item can still be arbitrarily old if the consumers are slow enough.
- • Every enqueue and every dequeue takes the same lock, so the queue is a single serialisation point for the whole pipeline — with enough producers, that lock is the bottleneck (What Contention Actually Costs).
- • Under heavy contention a lock convoy forms: workers spend more time acquiring and releasing than working, and throughput falls as you add them (Lock Convoys).
- • Holding the queue lock while running the job, rather than only while transferring it, turns a four-worker pool into a one-worker pool. This is the most common review finding in this pattern.
- • Head and tail on the same cache line means producers and consumers invalidate each other's lines on every operation even when the buffer is nearly full — False Sharing: Different Variables, Same Cache Line.
- • Lost update on the tail index: two producers write the same slot, one job silently disappears.
- • Check-then-act race on
isEmpty(): two consumers both pass the check and one dequeues from an empty buffer. - • Lost wakeup: a signal fires before the waiter is registered and a consumer sleeps forever on a non-empty queue.
- • Deadlock if a consumer enqueues into the same bounded queue it consumes from and the queue fills — a self-feeding pipeline with capacity is a circular wait (The Four Conditions).
- • Starvation of one producer by a stream of others when the queue lock is unfair.
- • Unbounded growth if capacity is left at "no limit", which is the memory-exhaustion path rather than a queue policy.
- • When the producing and consuming rates differ and the difference is *bursty* — the queue absorbs the burst instead of forcing the producer to run at the consumer's speed.
- • When the work can be decoupled from the request that created it: the handler returns in 5 ms and the resize happens behind it (Background Jobs and Workers in Architecture).
- • When you want to bound concurrency explicitly: a queue plus N consumers *is* a concurrency limit, expressed as a number you can see (Bounding Concurrency).
- • When the alternative is many threads mutating one structure — moving the data removes the shared mutable state instead of protecting it.
- • When the consumer is *persistently* slower than the producer. A queue smooths bursts; it cannot fix a rate mismatch, and pretending otherwise just relocates the failure.
- • When the caller needs the result. A queue plus a wait-for-result mechanism is a synchronous call with extra latency, extra failure modes and worse error reporting.
- • When jobs are tiny. Handoff cost — lock, signal, context switch, cache miss — can exceed the work, and a direct call is faster and simpler (Parallel Overhead).
- • When ordering across the whole stream matters end to end, because a consumer pool destroys completion order by construction.
- • Queue depth over time, and queue *age* of the head item — depth without age hides a slow leak that only affects the oldest work (Depth Is Not an Emergency; Age Is in Performance).
- • Producer block time: total seconds per minute that producer threads spent waiting for space. If this is non-zero on a request path, requests are being held.
- • Consumer idle ratio. Consumers idle while depth is high means they are blocked on something downstream, not on the queue.
- • Enqueue rejection or drop counters — if the policy can lose work, the loss must be a metric, not an inference.
- • Lock wait time on the queue lock itself, which is the signal that the queue has become the bottleneck rather than the buffer (Low CPU, High Latency: Lock Contention in Performance).
- • You now have two lifecycles to manage instead of one: the producer's and the pool's, plus a shutdown protocol that ties them together.
- • Errors move. A job that throws inside a worker has no caller to propagate to, so you need a failure destination — a retry, a dead-letter list, a log with enough context to act on.
- • Testing gets harder: the interesting behaviour is at capacity and at shutdown, and neither shows up in a unit test that enqueues three items.
- • Observability is no longer optional. Without depth, age and drop metrics the pipeline is invisible between "it works" and "the disk is full".
- • Do the work inline. If the work is 3 ms and the arrival rate is 10/s, a queue and a pool are pure overhead and one more thing to page someone about.
- • A semaphore around direct execution: bounds concurrency without introducing a buffer, so there is no backlog to reason about and no shutdown drain (Semaphores: Counting Permits as a Resource Limit, Bounding Concurrency).
- • A durable broker when losing in-flight work on a crash is unacceptable — an in-process queue is a memory structure and dies with the process (Message Queues in Architecture).
- • A channel, when the language gives you one with close semantics built in; it is the same pattern with the end-of-stream signal already solved (Channels).
Producers, a bounded queue, consumers
The producer is faster than the consumer
What people believe, and what is true
A queue decouples producer from consumer, so the producer never has to care how fast the consumer is.
A bounded queue decouples them for the length of a burst and then couples them tightly again. An unbounded one decouples them until memory runs out. The rates still have to match on average.
Adding workers will drain the backlog.
Only if the workers were the bottleneck. If they are all blocked on one database connection pool, more workers deepen the wait and change nothing about throughput.
FIFO in means FIFO out.
FIFO dequeue with N concurrent consumers gives no ordering on completion. If downstream needs order, the pool is the wrong shape or the ordering key must be partitioned across workers.