The question this answers
Can different items occupy different stages at the same time, and what does that actually buy — throughput, latency, or neither?
A 10-million-row import: for each row, Read from disk, Parse the record, Process it (validate and enrich), and Write it to the database.
The queues between stages — and only those, if the pipeline is built correctly. Each stage owns its own working memory and hands ownership of an item to the next stage through the queue, so nothing is concurrently mutated. The queues themselves are shared and must be concurrent structures (Concurrent Queues).
Every input row passes through all four stages exactly once, in stage order, and no row is in two stages at once. Whether the *output* preserves input order is a separate promise that costs extra — see section three.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The staircase: four stages, four items in flight
Sequentially, each row costs Read(2) + Parse(1) + Process(4) + Write(3) = 10 units, and rows are processed one at a time: 10 million rows take 100 million units. Give each stage its own worker with a queue between them and, once the pipeline is full, all four stages are busy on four different rows simultaneously. A new row completes every 4 units — the duration of the slowest stage — so 10 million rows take roughly 40 million units plus the fill and drain.
Read the timeline below as a staircase. The first row still takes 10 units end to end; it did not get faster and it cannot, because it must still visit four stages in order. What changed is that rows 2, 3 and 4 no longer wait for row 1 to finish. Pipeline parallelism buys throughput, not latency. If your complaint is "one import takes ten seconds", this is the wrong tool; if it is "we can only do a hundred a second", it is exactly the right one.
The rate is set entirely by the slowest stage. Speeding up Parse from 1 unit to 0.5 changes nothing, because Process still takes 4 and every row must pass through it. This is the single most useful property of a pipeline: it converts a diffuse performance question into a specific one — *which stage is the bottleneck* — and it makes the answer measurable rather than arguable.
The arithmetic: rate, fill, and where to spend your next hour
A pipeline's steady-state throughput is 1 / max(stage durations). Its per-item latency is at least the sum of the stage durations, plus whatever time the item spent queued between stages. Both formulas are worth memorizing, because together they tell you the two things engineers most often get wrong: that adding stages does not reduce latency, and that improving a non-bottleneck stage does not increase throughput.
The read-out below is what to do with those formulas. Notice that replicating the bottleneck stage — running three Process workers pulling from one queue — is the move that raises the rate, and that it converts the pipeline into a hybrid: task parallelism across stages, data parallelism within the bottleneck stage. That combination is the standard shape of a real import pipeline, an ETL job, a media transcoder and a compiler backend.
Bounded queues between stages are not an optimization, they are the design. An unbounded queue in front of a slow stage converts a throughput mismatch into unbounded memory growth: the Read stage happily reads all ten million rows into RAM while Process is still on row 900. The blocked: queue full segments in the timeline above are the system working — that is Backpressure propagating a rate limit backwards through the pipeline. See Bounded vs Unbounded Queues.
- Throughput = 1 / slowest stage. Latency >= sum of stages. Both are true simultaneously and they point in opposite directions.
- Replicating only the bottleneck stage is the cheapest capacity increase available and needs no change to the other stages.
- Every bottleneck fix promotes a new bottleneck. Decide the target rate in advance so you know when to stop.
stage duration utilization at steady state ------------------------------------------------- Read 2u 50% (idle half the time) Parse 1u 25% Process 4u 100% <- BOTTLENECK, sets the rate Write 3u 75% sequential per row = 2+1+4+3 = 10u pipelined throughput = 1 row / 4u (2.5x) pipelined latency per row = still >= 10u (1.0x) fill time (first row out) = 10u what changes the rate: Parse 1u -> 0.5u .... no change (not the bottleneck) Read 2u -> 1u .... no change (not the bottleneck) Process 4u -> 3u .... rate 1/4 -> 1/3 (+33%) Process x3 workers .... rate 1/4 -> 1/3 (Write becomes the bottleneck) then Write x2 .... rate 1/3 -> 1/2 (Read becomes the bottleneck) the pattern: fixing a bottleneck reveals the next one. Stop when the rate is good enough, not when the graph is flat.
What replication costs: output ordering
A single-worker stage preserves order for free: it takes items from its input queue in order and pushes them to its output queue in order. Replicate that stage across three workers and the guarantee evaporates instantly — worker B can finish row 8 before worker A finishes row 7, and row 8 reaches the Write stage first. Nothing raced, nothing is corrupt, and if your import must apply rows in file order the result is wrong anyway.
The schedule below shows exactly that, and it is worth stepping through because the bug is invisible in every stage's own code. Each Process worker is correct in isolation. The invariant that broke — "rows are written in input order" — was never written down anywhere, and was being provided incidentally by the fact that there used to be one worker.
The fixes all cost something. Sequence-number reordering at the Write stage restores total order but needs a buffer that grows with the worst-case skew between workers, and stalls if one worker is slow. Partitioning by key gives you per-key order for free — route all rows for account 42 to the same worker — which is usually the guarantee you actually need and is the same idea as a partitioned log (Ordering Guarantees: Four Levels, Four Prices). Or you decide the pipeline genuinely does not need ordering, write that down as a contract, and make the Write stage idempotent so replays and reorders are harmless.
| # | Process worker A | Process worker B | Write stage | State |
|---|---|---|---|---|
| 1 | take row 7 (account 42, balance -> 100) | · | · | A holds=row 7 db[42]=null |
| 2 | · | take row 8 (account 42, balance -> 250) | · | A holds=row 7 B holds=row 8 db[42]=null |
| 3 | enrich row 7 — cache miss, fetches customer record (slow) | · | · | A holds=row 7 (in flight) |
| 4 | · | enrich row 8 — cache hit, returns immediately | · | B holds=row 8 (ready) |
| 5 | · | push row 8 to write queue | · | write queue=[8] |
| 6 | · | · | write row 8: db[42] = 250 | db[42]=250 |
| 7 | push row 7 to write queue | · | · | write queue=[7] |
| 8 | · | · | write row 7: db[42] = 100 | db[42]=100 ✕ Row 7 preceded row 8 in the input, so the final balance must be 250. The later row was overwritten by the earlier one. |
Key points
- Pipeline parallelism improves throughput and does not improve per-item latency — it usually adds a little, from queueing between stages.
- Steady-state rate is 1 / slowest stage; optimizing any other stage changes nothing measurable.
- Replicating only the bottleneck stage is the cheapest capacity increase, and it turns the pipeline into task parallelism across stages plus data parallelism within one.
- Bounded queues between stages are the design, not a tuning detail: they propagate backpressure and stop a fast stage from consuming all memory.
- A single-worker stage provides ordering incidentally. Replicating it removes that guarantee silently, and the fix costs either a reorder buffer or a partitioning scheme.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Decompose the work into stages with a clear handoff — each stage takes an item, transforms it, and passes ownership onward, never sharing mutable state with the next stage.
- • Connect consecutive stages with bounded concurrent queues; each stage is a Producer / Consumer pair with its neighbours.
- • Run each stage in its own worker (thread, task or process) so all stages can be busy on different items simultaneously.
- • Measure per-stage service time under load; the maximum is the pipeline rate and the sum is the floor on per-item latency.
- • Replicate the bottleneck stage across N workers pulling from one queue, then re-measure — the bottleneck will have moved.
- • Shut down in stage order: close the head, drain each queue, and only then stop the next stage, or in-flight items are lost (Draining a Pipeline).
- • The intended one: Read(r4) || Parse(r3) || Process(r2) || Write(r1). Four items, four stages, no shared mutable state, invariant preserved under every relative ordering because ownership moves with the item.
- • Backpressure: Read finishes r5 and blocks pushing to a full queue while Process is still on r2. The Read worker is idle by design — that is the rate limit propagating backwards, not a bug.
- • Ordering broken by replication: A takes row 7, B takes row 8, B finishes first, Write applies 8 then 7, and the earlier row overwrites the later one for the same key.
- • Unbounded queue: Read pushes ten million rows in seconds while Process is on row 900; resident memory grows to the whole input and the process is killed by the OOM killer with no error from any stage.
- • Shutdown race: the Read stage signals completion and the process exits while three items are still queued for Write — the run reports success and silently drops them.
- • Poison item: Process throws on row 4,000,001, its worker dies, no one drains its input queue, and the whole pipeline stalls with every upstream stage blocked on a full queue. Nothing crashes; throughput just becomes zero.
- • Guarantees each item visits stages in order, and that a stage sees an item only after the previous stage finished with it — the queue handoff establishes happens-before (Happens-Before: The Edge That Makes a Write Visible).
- • Guarantees throughput of 1 / slowest-stage in steady state, given bounded queues large enough to absorb per-item jitter.
- • With one worker per stage, guarantees FIFO ordering end to end. With replicated stages, guarantees nothing about output order.
- • Does NOT reduce per-item latency. Ever. Adding stages increases it.
- • Does NOT guarantee liveness on its own: a stage that dies or blocks forever stalls everything upstream through the bounded queues, and the symptom is silence rather than an error.
- • Does NOT guarantee that in-flight items survive shutdown unless you drain the queues explicitly.
- • The queues are the shared structures; a single-producer/single-consumer queue between two stages has almost no contention, while a replicated stage pulling from one shared queue has real contention on its head.
- • A full queue means the upstream stage is blocked — intended, but if it is blocked most of the time you are paying for a worker that mostly waits, and could merge the stages.
- • An empty queue means the downstream stage is starved, which points at an upstream bottleneck rather than a downstream one. Queue occupancy is the diagnostic.
- • Handoffs cost a context switch or a task wake per item, so very fine-grained stages spend more on coordination than on work — batch items through the queues when the per-item work is tiny.
- • Unbounded queue growth in front of a slow stage: memory exhaustion, or a latency that grows without limit while throughput looks fine (Depth Is Not an Emergency; Age Is in Performance is the signal).
- • Silent reordering after replicating a stage — the schedule above.
- • Stalled pipeline after a worker dies: no error, all upstream stages blocked on full queues, throughput zero.
- • Lost items at shutdown when queues are not drained in stage order.
- • A tiny per-item work unit swamped by handoff overhead, so the pipelined version is slower than the sequential loop.
- • Head-of-line blocking: one pathological item occupying the bottleneck stage for a hundred times the normal duration stalls everything behind it.
- • Streaming work with a natural stage structure and far more items than stages: imports, ETL, log processing, media transcoding, compilation.
- • Workloads where stages use different resources — Read is I/O-bound, Process is CPU-bound, Write is database-bound — so overlapping them uses the whole machine instead of one part at a time.
- • When throughput is the requirement and per-item latency has slack, which describes most batch work.
- • When you want the bottleneck to be identifiable: a pipeline makes "which stage" a measurable question.
- • When latency is the requirement. A pipeline makes a single item slightly slower, never faster.
- • When one stage dominates completely — 95% of the time in Process means the ceiling is about 1.05x, and you want to parallelize inside that stage instead.
- • When per-item work is smaller than the handoff cost, so the queues cost more than the stages save.
- • When the stages cannot be made to hand off ownership cleanly and end up sharing mutable state — you have taken on all the coordination cost of the pipeline and all the race risk of shared memory.
- • For a small number of items, where fill and drain dominate the whole run.
- • Per-stage busy fraction: the stage at ~100% while others idle is the bottleneck, and it is unambiguous.
- • Queue depth and queue *age* between each pair of stages — persistently full means the downstream stage is the constraint; persistently empty means it is upstream.
- • Steady-state throughput against 1 / slowest-stage. A large gap means handoff overhead or jitter, not a missing stage.
- • End-to-end item latency distribution alongside throughput, so a throughput win that quietly tripled p99 does not go unnoticed.
- • Fill and drain time separately from steady state, because for short runs they are most of the elapsed time.
- • You now own queues: their bounds, their backpressure behaviour, their metrics and their shutdown protocol.
- • Error handling becomes per-stage — what happens to an item that fails in Process, and who notices that its worker died — and getting it wrong produces a silent stall rather than an exception.
- • Shutdown must be ordered and draining, which is more code than it sounds and is almost always written after the first data-loss incident.
- • Replicating a stage adds an ordering decision you did not previously have to make, and the reorder buffer or partitioning scheme that implements it.
- • Debugging spans stages: a bad item is produced in one worker and observed in another, so items need a correlation id from the start.
- • Data-parallel batch processing: split the ten million rows into eight chunks and run the whole four-step sequence per chunk. Simpler, no queues, no ordering surprises — the right default when items are independent and order does not matter.
- • Just make the bottleneck stage faster. A 4x faster Process beats any pipeline arrangement of the original and adds no infrastructure.
- • An existing streaming framework or a message broker between stages, when the stages want to be separate processes or services anyway.
- • Async I/O within a single loop, when the stages are all waiting rather than computing — overlap without workers or queues (Async Is Not Parallelism).
Pipeline visualizer
Producers, a bounded queue, consumers
The producer is faster than the consumer
What people believe, and what is true
Pipelining makes each item faster.
It makes items *complete more often*. Item latency is at least the sum of stage times and typically increases slightly from queueing between stages.
More stages means more speedup.
Throughput is capped by the slowest stage regardless of stage count. More stages adds handoff cost and latency; only rebalancing or replicating the bottleneck raises the rate.
Optimizing any stage helps.
Only the bottleneck stage changes the rate. Time spent on the other three is time spent making idle workers idle sooner.
An unbounded queue between stages avoids blocking, so it is faster.
It converts a throughput mismatch into unbounded memory growth and unbounded latency. The blocking is the rate limit doing its job (Backpressure).