Execute: Not All Operations Cost the Same
The execute phase is where the work happens, in a set of specialised functional units. Two things surprise people: different operations take very different numbers of cycles, and an operation's latency and its throughput are separate numbers that can differ by an order of magnitude.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Latency and throughput are different numbers
Latency is how many cycles pass between an operation starting and its result being available to a dependent instruction. Throughput is how often a new operation can be started — often expressed as its reciprocal, the number of cycles before the unit can accept another.
For a pipelined multiplier, latency might be several cycles while throughput is one per cycle: the unit is working on several multiplies simultaneously at different stages. For a divider, both numbers are large and roughly equal, because dividers are typically not pipelined — the unit is occupied for the whole operation.
This distinction decides whether an expensive operation actually costs you. A loop performing independent multiplies runs at throughput, so multiply latency barely matters. A loop where each multiply feeds the next runs at latency, and the difference between those two loops can be several-fold with identical instruction counts. That is the same throughput-versus-latency structure as Throughput Improved, Latency Did Not, visible at the scale of a single functional unit.
Units are specialised, and there are only so many
Functional units are not interchangeable. A core has some number of integer units, some number capable of floating-point work, some number of load units and some number of store units, and these counts are asymmetric by design because typical code uses them asymmetrically.
The consequence is a second kind of ceiling. If a loop performs two loads per iteration and the core has two load units, the loop cannot exceed one iteration per cycle no matter how simple the arithmetic is. This is a *structural* limit — the structural hazard from Pipeline Hazards: The Three Ways Overlap Fails — and it is why some loops refuse to speed up when you remove arithmetic from them.
This is also the mechanism underneath Superscalar Execution: issuing several instructions per cycle is only possible when they need *different* units. Four consecutive multiplies cannot issue together on a core with one multiplier, however independent they are.
Where this changes code
The reliable win is removing division from inner loops. Because dividers are typically not pipelined, a division on the critical path of a loop can dominate everything else in it. Replacing repeated division by the same value with one reciprocal and a series of multiplies is the classic transformation — and a compiler will often do it for integers, but for floating point it usually cannot without permission, because the reciprocal-multiply result is not bit-identical to the division.
The second win is breaking dependency chains so that latency stops mattering. Summing an array into one accumulator creates a chain of dependent adds, each waiting for the previous. Using several accumulators and combining them at the end makes the adds independent, so the loop runs at the unit's throughput instead of its latency. This is the single most common hand transformation that actually works on modern cores, and it is the same insight Instruction-Level Parallelism generalises.
The honest caveat: on both counts the compiler frequently gets there first, and on both counts the win depends on the operation actually being on the critical path. Measure the loop before and after; a transformation that helps a floating-point reduction may do nothing at all for an integer one that was already limited by memory.
1sum = 0.0;2for (i = 0; i < n; i++) {3 sum += a[i]; // each add waits for the previous add4}5// Runs at FP-add LATENCY per element.1s0 = s1 = s2 = s3 = 0.0;2for (i = 0; i + 3 < n; i += 4) {3 s0 += a[i]; // these four are mutually independent4 s1 += a[i+1];5 s2 += a[i+2];6 s3 += a[i+3];7}8sum = (s0 + s1) + (s2 + s3);9// Runs closer to FP-add THROUGHPUT per element.Identical arithmetic, identical instruction count, different dependency structure. The first version can only start an add when the previous one finishes; the second keeps several in flight, so the pipelined adder is kept busy. For floating point this changes the result slightly because addition is not associative, which is exactly why a compiler will not do it for you without explicit permission.
Key points
- Operations differ substantially in cost: bitwise and add are cheap, multiply moderate, divide and square root expensive.
- Latency (result availability) and throughput (issue rate) are separate numbers and can differ by an order of magnitude.
- Dependent chains run at latency; independent work runs at throughput — the same code can hit either regime.
- Functional units are specialised and finite, so instruction mix determines how much can issue per cycle.
- Breaking dependency chains with multiple accumulators is one of the few hand transformations that reliably helps.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Scheduler → functional unit: an operation is issued once its operands are available and a unit capable of performing it is free.
- 2Functional unit → pipeline stages: pipelined units accept a new operation each cycle while previous ones are still in progress.
- 3Divider → occupancy: non-pipelined units hold the operation for its full duration, blocking other operations of that class.
- 4Result → forwarding network: the value is delivered to waiting dependents, often before it is architecturally written back (Forwarding and Stalls: Paying for Dependencies).
- 5Dependency chain → issue rate: if each operation feeds the next, issue rate collapses to one per latency rather than one per cycle.
- • "An operation is expensive, so I should avoid it" — only if it is on the critical path; an expensive operation with plenty of independent work around it is often free.
- • "Instruction count went down so it will be faster" — replacing four cheap independent operations with one expensive dependent one usually is not.
- • "Latency numbers from a table tell me the loop's speed" — only for dependent chains; independent work runs at throughput.
Consequences, controls and cost
- • A loop with a division on its critical path can be several times slower than the same loop with a multiply, at identical instruction count.
- • Reduction loops written with a single accumulator run at operation latency and leave most of the machine idle.
- • Loops limited by the number of load units do not speed up when arithmetic is removed.
- • Remove division from inner loops where possible — compute a reciprocal once and multiply, accepting the floating-point rounding difference deliberately.
- • Break long dependency chains with multiple accumulators so independent operations can overlap.
- • Check the instruction mix of a hot loop against the units available; a loop that is load-limited needs fewer loads, not cheaper arithmetic.
- • Prefer letting the compiler do this: it knows the target's unit counts and latencies, and hand transformations decay as the code changes.
- • Compare cycles against instructions retired for the loop; a high cycles-per-instruction with no cache misses suggests a latency-bound dependency chain.
- • Test the hypothesis directly: add a second accumulator and see whether time changes. If it does, the loop was latency-bound.
- • Consult the CPU vendor's optimisation manual for latency and throughput of specific instructions on the specific target — these are published and are the authoritative numbers.
- • Multiple accumulators change floating-point results because addition is not associative; if bit-exact reproducibility matters, this is not available.
- • Reciprocal multiplication trades exactness for speed and can be unacceptable in financial or scientific contexts.
- • Unrolling for accumulators increases code size, which pushes back on the front end ([[instruction-cache]]).
Scope
§224 — what these claims are specific to.
- MICROARCH-SPECIFICUnit counts, operation latencies and which units are pipelined differ per microarchitecture. Vendor optimisation manuals publish exact tables per model; treat any single number as specific to one design.
- GENERALThat operations differ in cost, that latency and throughput are distinct, and that dependency chains run at latency, are true of every pipelined processor.