Throughput Improved, Latency Did Not
Nearly every technique modern CPUs use — pipelining, superscalar issue, out-of-order execution, speculation — increases the number of operations completed per unit time without reducing, and sometimes while increasing, the time any single operation takes. This is why decades of architectural progress leave a dependent chain almost exactly as slow as it was.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Pipelining buys throughput by adding latency
Splitting instruction execution into stages lets several instructions occupy different stages simultaneously, so one completes every cycle rather than one every several cycles. That is a large throughput gain. But an individual instruction now traverses every stage, and the pipeline registers between stages add real delay — so a single instruction executed in isolation takes *longer* on a pipelined machine than on an unpipelined one with the same logic.
This is the fundamental trade in one sentence: pipelining improves the rate at which work completes and slightly worsens the time any single unit of work takes. Nobody minds, because programs contain many instructions and the throughput gain dominates. The trade only becomes visible when the program cannot supply independent work — which is exactly what a dependency chain is.
The same shape recurs at every level of the machine. Superscalar issue widens throughput without shortening any single operation. Out-of-order execution finds independent work to overlap, which does nothing when there is none. Speculation hides latency by guessing ahead, and a chain of dependent operations gives it nothing to guess about. Every one of these mechanisms converts available parallelism into speed, and returns nothing when parallelism is absent.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | |
|---|---|---|---|---|---|---|---|---|---|
| i1 add r1, r2, r3 | F | D | X | M | W | ||||
| i2 add r4, r5, r6 | F | D | X | M | W | ||||
| i3 add r7, r8, r9 | F | D | X | M | W | ||||
| i4 add r10, r11, r12 | F | D | X | M | W | ||||
| i5 add r13, r14, r15 | F | D | X | M | W |
A dependent chain sees none of it
Now make each instruction depend on the previous one. The second cannot enter its execute stage until the first has produced its result, so the overlap that made the previous diagram fast is impossible. Five dependent instructions take roughly five times the latency of one, and every mechanism the machine has for finding parallelism sits idle, because the program has supplied none.
This is why the practical advice throughout this domain is structural rather than local. Breaking a chain into several independent chains lets the machine do what it is built for; shaving a cycle off one link does almost nothing by comparison. It is the same argument Misses That Overlap Are Nearly Free makes about cache misses, Dependency Graphs: The Real Shape of Your Code makes about instruction scheduling, and Pointer Chasing: The Address You Do Not Have Yet makes about data structures — three faces of one constraint.
It also explains a persistent observation about hardware generations: workloads rich in independent work — media processing, dense linear algebra, streaming analytics — benefit substantially from new processors, while workloads dominated by dependent chains, such as pointer-heavy traversals and deeply serial interpreters, improve far less. The machine got wider, not fundamentally quicker per step.
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | |
|---|---|---|---|---|---|---|---|---|---|
| i1 add r1, r2, r3 | F | D | X | M | W | ||||
| i2 add r4, r1, r6 | F | D | X | M | W | ||||
| i3 add r7, r4, r9 | F | D | X | M |
Choosing which one you are optimising
Because the two quantities respond to different changes, the first question in any optimisation is which one actually matters for the workload. A batch job cares about throughput almost exclusively: total work per unit time is the whole objective, and individual item latency is irrelevant. An interactive request cares about latency, and improving aggregate throughput while lengthening the critical path makes the user experience worse.
The distinction has teeth because several standard techniques trade one for the other. Batching improves throughput and increases the latency of anything waiting in the batch. Deeper pipelines improve throughput and raise misprediction cost, which shows up as latency. Adding threads improves aggregate throughput and can worsen individual latency through contention. In each case the right answer depends entirely on which quantity the workload is judged by — the argument the Observability & Performance domain develops in Latency Is a Distribution, Not a Number and Throughput: Requests, Packets and Bytes per Second.
For hardware specifically, the useful mental model is that a modern core is a throughput machine that is willing to work hard to hide latency. It will speculate, reorder, prefetch and overlap in order to keep its units busy. Give it independent work and it will reward you generously; give it a serial chain and it will hand back a machine barely faster than one from several generations ago.
| Technique | Throughput | Single-operation latency | Wins when |
|---|---|---|---|
| Pipelining | Large improvement | Slightly worse | Many independent instructions available |
| Superscalar issue | Large improvement | Unchanged | Instruction mix has parallelism to exploit |
| Out-of-Order Execution | Improvement | Unchanged for the chain itself | Independent work exists to overlap |
| Speculative Execution: Doing Work Before You Know You Need It | Improvement | Hides latency when correct | Branches are predictable |
| SIMD: One Instruction, Many Elements | Large improvement | Per-instruction latency often higher | Regular data-parallel work |
| Batching | Improvement | Worse for items waiting | Throughput is the objective |
| More cores | Improvement if work is parallel | Unchanged or worse under contention | Work divides cleanly |
| Breaking a dependency chain | Improvement | Improvement | Almost always — the rare change that helps both |
Key points
- Pipelining raises throughput and slightly increases the latency of a single isolated instruction.
- Superscalar issue, out-of-order execution and speculation all convert available parallelism into speed and return nothing when none exists.
- A dependent chain defeats every latency-hiding mechanism the machine has, which is why such workloads improve little across hardware generations.
- Batching, deeper pipelines and additional threads all trade individual latency for aggregate throughput.
- Breaking a dependency chain is the unusual change that improves both quantities at once.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Instruction → stages: execution is divided so that several instructions occupy different stages at once.
- 2Steady state → one per cycle: throughput approaches one completion per cycle while each instruction still traverses every stage.
- 3Dependency → stall: a dependent instruction cannot proceed until its input is produced, and the overlap disappears.
- 4No independent work → idle mechanisms: reordering, speculation and prefetch have nothing to work with.
- 5Chain length → runtime: total time becomes roughly the chain length multiplied by per-link latency, regardless of machine width.
- • Expecting a newer processor to speed up a serial dependent workload proportionally to its throughput gains.
- • Reading peak throughput specifications as achievable for a workload with a serial critical path.
- • Adding threads to reduce the latency of a single request rather than to raise aggregate throughput.
- • Optimising one link of a chain instead of restructuring to remove the chain.
Consequences, controls and cost
- • Workloads rich in independent work gain substantially from newer hardware; serial workloads gain far less.
- • A change that improves aggregate throughput can degrade the latency of an individual request.
- • Optimising the cost of one link in a dependency chain produces disappointingly small gains.
- • Peak throughput figures on a datasheet are unreachable for any workload with a serial critical path.
- • Break dependency chains into independent streams — the change that improves both latency and throughput.
- • Decide explicitly which quantity the workload is judged on before choosing a technique.
- • Avoid batching on latency-sensitive paths, however attractive the throughput number looks.
- • Give the machine independent work: unroll, interleave, and prefer structures that allow address computation ahead of loads.
- • Latency of a single operation in isolation, alongside aggregate throughput under load — never one alone.
- • A scaling experiment: if throughput rises with concurrency while single-operation latency holds, the machine has parallelism to exploit.
- • Dependency chain length in the hot loop, from the disassembly or a dependency-aware analyser.
- • [[cpi]] together with execution port utilisation, to see whether the units are starved or saturated.
- • Breaking dependency chains costs code complexity, registers and often extra memory traffic.
- • Optimising for throughput can make interactive latency worse in ways aggregate metrics conceal.
- • Deeper pipelines raise throughput and increase misprediction penalties, which hurts branchy code.
Scope
§224 — what these claims are specific to.
- SIMPLIFIEDThe five-stage pipeline is a teaching device; real cores have many more stages, multiple issue ports and out-of-order scheduling, which changes the numbers substantially while leaving the throughput-versus-latency trade intact.
- MICROARCH-SPECIFICPipeline depth, issue width, and the latency of individual operations differ by design, so the ratio between throughput gain and latency cost is a per-core property.