CPI and IPC: The Number Everyone Misreads
Cycles per instruction, and its reciprocal instructions per cycle, describe how smoothly work is flowing through the machine. Neither is a measure of performance. A change that halves CPI while tripling instruction count made the program slower, and a vectorized loop that runs twice as fast often shows a worse CPI than the scalar loop it replaced.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
One number, defined by two others
CPI is cycles divided by instructions retired. IPC is instructions retired divided by cycles. They are the same measurement written two ways, and both are ratios — which is the entire source of the confusion, because a ratio can improve when either the numerator falls or the denominator rises, and only one of those is good news.
What CPI genuinely measures is smoothness of flow. A superscalar core can retire several instructions per cycle when everything is ready; when it is waiting on a cache miss, a mispredicted branch or a long dependency chain, it retires none. So a high CPI is a reliable signal that the core is stalling, and the next question is which of the small set of possible stalls it is. That is a real and useful diagnosis, and it is what the metric is for.
What CPI does not measure is work done. The quantity that maps to elapsed time is cycles, and cycles equals instructions multiplied by CPI. Improving CPI while inflating instruction count is a common way to move the ratio in the pleasing direction while moving the runtime in the other one.
1cycles = instructions x CPI2time = cycles / frequency # and frequency is not constant either3 4# So a "20% CPI improvement" is only a win if instruction count held:5# before: 10.0e9 insn x 1.50 CPI = 15.0e9 cycles6# after: 9.0e9 insn x 1.20 CPI = 10.8e9 cycles -> genuinely 28% fewer cycles7# after': 14.0e9 insn x 1.20 CPI = 16.8e9 cycles -> same CPI win, 12% SLOWERThe vectorized loop that looks worse and is faster
The cleanest demonstration of why CPI is not a score comes from vectorization. A scalar loop summing an array executes many cheap, highly predictable instructions and can sustain a very good IPC. Replace it with a vector loop and the instruction count collapses — each instruction now does eight elements of work — but each of those instructions has longer latency and the loop is far more likely to be waiting on memory bandwidth rather than issuing. IPC drops. Runtime drops much further.
The same inversion appears whenever you replace many cheap operations with fewer expensive ones: a lookup table replacing arithmetic, a wider load replacing several narrow ones, a specialised instruction replacing a sequence. In each case the ratio worsens and the program improves, because the ratio was never counting the thing you cared about.
The discipline that avoids all of this is simple and rarely followed: always report cycles alongside CPI, and treat CPI as a diagnostic for *why* rather than a verdict on *whether*. The Observability & Performance domain makes the same argument one level up in Benchmark Fallacies: Confident Numbers That Are Wrong — a metric that improves while the user-visible number does not has told you nothing.
1for (i = 0; i < n; ++i)2 sum += a[i];3 4# ~4 instructions per element, highly predictable5# instructions: 4.0e9 CPI: 0.55 cycles: 2.2e91for (i = 0; i < n; i += 8)2 vsum = vadd(vsum, vload(&a[i]));3 4# ~4 instructions per EIGHT elements, memory-bandwidth limited5# instructions: 0.5e9 CPI: 1.60 cycles: 0.8e9Instruction count fell 8x while CPI rose 3x, so cycles fell roughly 2.75x. Anyone optimising for IPC would have rejected this change. Numbers are illustrative of the shape, not measured from a specific machine — see SIMD: One Instruction, Many Elements and Vectorization: Turning a Loop Into Vector Work for the mechanism.
Reading CPI as a diagnosis
Used properly, CPI is the first branch in a decision tree. A low CPI with a disappointing runtime says the machine is flowing smoothly and simply has too much work to do — go and look at the algorithm, the instruction count, and whether the compiler vectorized (Algorithmic Cost in a Request Handler is the higher-level version of this conversation). A high CPI says the core is stalling, and the counters narrow down which stall it is.
The interpretation is genuinely microarchitecture-dependent, because the peak issue width differs between designs. A CPI of 1.0 on a core that can retire four instructions per cycle means the machine is running at a quarter of its capability; on a narrower core the same number is closer to par. This is why absolute CPI thresholds copied from a blog post are worthless and why the useful comparison is always against the same code on the same machine.
It is also worth knowing that CPI can be *too* good. Very low CPI on a workload you expected to be memory-heavy sometimes means the compiler eliminated the work entirely — a case Every Way a CPU Microbenchmark Lies treats at length, and one that has embarrassed many benchmark results.
| CPI | Other evidence | Likely diagnosis | Where to go next |
|---|---|---|---|
| Low | Runtime still disappointing | Machine flowing well; simply too many instructions | CPI and IPC: The Number Everyone Misreads is done — look at algorithm and Vectorization: Turning a Loop Into Vector Work |
| Low | Suspiciously low, workload should be heavy | Work may have been optimised away entirely | Every Way a CPU Microbenchmark Lies |
| High | High LLC miss rate | Waiting on memory | Busy Is Not the Same as Working, Misses That Overlap Are Nearly Free |
| High | High branch-miss rate | Pipeline being flushed and refilled | Misprediction: What a Wrong Guess Costs |
| High | High dTLB miss / page-walk cycles | Address translation, not data, is the constraint | When Translation Itself Is the Bottleneck |
| High | Front-end stall cycles high, data misses low | Code, not data, failing to arrive | Your Code Is Data Too |
| High | None of the above; long dependent chains | Latency-bound on the dependency graph itself | Dependency Graphs: The Real Shape of Your Code, Throughput Improved, Latency Did Not |
Key points
- CPI is cycles over instructions retired; IPC is its reciprocal. Both are ratios, and a ratio improves when either term moves.
- Cycles, not CPI, is the quantity that corresponds to elapsed time — always report both.
- A change that reduces instruction count often raises CPI while lowering runtime; vectorization is the canonical example.
- High CPI is a reliable signal that the core is stalling, and the accompanying counters say which stall it is.
- Good and bad CPI values are relative to the core's issue width, so absolute thresholds copied from elsewhere are meaningless.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Front end → back end: instructions are fetched, decoded and issued into the out-of-order engine.
- 2Back end → retirement: instructions complete and retire in program order; the retired counter increments.
- 3Stall → cycles: when no instruction can retire — waiting on a miss, a flush or a dependency — cycles accumulate with no retirement.
- 4Counters → ratio: cycles divided by retired instructions produces CPI, mixing "how much work" and "how smoothly" into one number.
- 5Ratio → misreading: because the denominator is under the programmer's control, the ratio can be improved by doing more work.
- • Treating higher IPC as strictly better and rejecting a change that reduced total work.
- • Comparing CPI between two different microarchitectures with different issue widths.
- • Reading a low CPI as "the code is efficient" when it may mean "the code was deleted".
- • Assuming a CPI near 1.0 is respectable without knowing how many instructions the core can actually retire per cycle.
Consequences, controls and cost
- • Optimisation efforts steered by IPC can select changes that increase runtime.
- • A vectorized or otherwise work-reducing change looks like a regression on the metric while being an improvement in fact.
- • Teams that compare CPI across different machines reach conclusions that do not hold on either.
- • An unexpectedly excellent CPI is often the first sign that a benchmark measured nothing at all.
- • Report cycles and instructions alongside every CPI figure so the ratio can never be read in isolation.
- • Use CPI to choose *which* stall to investigate, never to decide whether a change was good.
- • Compare only against the same workload on the same machine; never against a published threshold.
- • When instruction count changes substantially, evaluate the change on cycles and stop looking at the ratio.
- • Cycles and instructions-retired from a counting run; compute both CPI and total cycles.
- • The same pair before and after any change, with instruction count reported explicitly.
- • Stall-cycle events to attribute a high CPI to front end, back end, or memory.
- • A sanity check that instruction count moved in the direction the source change implies — if it did not, the compiler did something you did not expect.
- • CPI is cheap to obtain and hard to interpret; the interpretation cost is where teams get it wrong.
- • Optimising for smoothness of flow can conflict with optimising for total work, and the two require different measurements.
- • The metric is microarchitecture-relative, so knowledge built on one fleet transfers imperfectly to the next.
Scope
§224 — what these claims are specific to.
- MICROARCH-SPECIFICWhat counts as a good CPI depends on the core's peak retirement width, which differs between designs and generations; a value that indicates stalling on a wide core can be near-peak on a narrow one.
- SIMPLIFIEDThe worked figures illustrate the arithmetic relationship between instruction count, CPI and cycles; they are not measurements from any particular machine.