Busy Is Not the Same as Working

"CPU utilisation" is measured by the operating system, and it means one thing only: what fraction of the interval the core was not idle. It says nothing whatsoever about what the core was doing while not idle. A core that issues one instruction and then stalls for two hundred cycles waiting on DRAM is 100% utilised for that entire stall, because it never went to sleep.

This is why an entire genre of performance surprise exists: adding cores to a memory-bound workload produces almost no speedup, and sometimes a slowdown, while every dashboard insists the CPUs were saturated and therefore more of them should help. The cores were saturated; the *memory system* was the constraint, and adding consumers to a saturated shared resource makes contention worse, not better. The Observability & Performance domain reaches the same conclusion from the outside in Computing or Waiting?; this is the mechanism underneath it.

The honest reformulation is that a core has two ways to spend a cycle: retiring work, or waiting. Compute-bound means the first dominates and the execution ports are the constraint. Memory-bound means the second dominates and the data supply is the constraint. Every practical distinction follows from which of those two is true.

The diagnosis is mechanical once you have the numbers. Start with CPI. If it is close to the core's peak — meaning the machine is retiring near its maximum instructions per cycle — the machine is flowing and you are compute-bound; the lever is doing less work. If CPI is poor, the core is stalling, and the next counters say why.

For memory-bound confirmation, the pair that matters is last-level misses per instruction together with stall cycles attributable to memory. A high LLC miss rate on its own is suggestive but not conclusive, because independent misses overlap — the point Misses That Overlap Are Nearly Free makes. What settles it is the combination of poor CPI, high LLC misses, and low execution-port utilisation: the units are idle, the data is not there, and the misses are not overlapping enough to hide it.

One further check separates the two flavours of memory-bound, and it changes the fix. If memory *bandwidth* is near the platform's achievable ceiling, you are bandwidth-bound and the answer is to move fewer bytes — compression, smaller types, better layout. If bandwidth is nowhere near the ceiling but misses are still stalling you, you are latency-bound on a dependency chain, and the answer is to break the chain or increase the number of independent accesses in flight. The two look identical in a wall-clock profile.

This is the part that makes the diagnosis worth the effort. For a compute-bound loop the levers are all about doing less: a better algorithm, vectorization so each instruction does more work, strength reduction, removing redundant computation, and only then more cores. Improving cache locality in such a loop changes essentially nothing, because the data was already arriving in time.

For a memory-bound loop the levers are all about touching less memory or touching it in a friendlier order: contiguous layout instead of pointer-linked structures, struct-of-arrays when you only read some fields, blocking so the working set fits a cache level, smaller data types, and eliminating the indirections that force dependent loads. Meanwhile making the arithmetic cheaper is nearly free of effect, because the arithmetic was never the constraint — the units were idle.

The cost of getting this backwards is not merely wasted time; it is a plausible-looking change that ships, does nothing, and consumes the team's belief that performance work is worthwhile. That is a large part of why Measure Before You Optimize exists as a discipline rather than a slogan.