Hits, Misses and What a Miss Actually Costs
A miss is not an error; it is a cost, and it is the normal way data arrives. What matters is where the miss is satisfied — one level out, three levels out, or in DRAM — because those outcomes differ by more than an order of magnitude and imply completely different fixes.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
The escalation path
A load begins at L1. On a hit, the value is available within a handful of cycles and the pipeline barely notices. On a miss, the request is forwarded to the next level, and the same question is asked again. Each level that fails to answer adds its latency to the total, and the final fallback is a trip to the memory controller and DRAM.
Crucially, the core does not necessarily stop. An out-of-order core will keep executing instructions that do not depend on the pending load, and can have several misses outstanding at once (Misses That Overlap Are Nearly Free). That is why a workload with many independent misses can be far cheaper per miss than one where each miss depends on the previous — which is precisely the difference between streaming and Pointer Chasing: The Address You Do Not Have Yet.
The line is installed on the way back, typically at each level it passed through. That is what makes the *next* access to any byte in that line cheap, and it is why the first iteration of a loop is unrepresentative of the rest.
Where the miss lands is the whole story
Two programs can have identical L1 miss rates and completely different performance, because one has its misses caught by L2 and the other goes all the way out. This is why per-level counters matter and why a single "cache misses" number is a poor diagnostic — it usually counts last-level misses, or L1 misses, depending on the tool, and the two mean very different things.
The scale below shows why. Each escalation costs several times the previous stop, and the final step off-chip is the largest. A miss caught at L2 is a minor inconvenience; a miss that reaches DRAM on the critical path of a dependency chain is a stall measured in hundreds of cycles, during which the core may find nothing useful to do.
This is also the reason the *fix* depends on where misses land. Frequent L1 misses caught by L2 often mean a slightly-too-large working set or a conflict problem, and respond to layout or blocking changes. Frequent last-level misses mean the working set genuinely exceeds on-chip capacity, and respond to algorithmic change, streaming-friendly access or accepting bandwidth limits (When the Memory Bus Is the Bottleneck).
Reading miss numbers without fooling yourself
A miss *count* scales with how much work you did, so it is nearly useless for comparison. A miss *rate* — misses per load, or misses per instruction — is comparable across runs and sizes, and is what you should be computing. Misses per instruction is particularly useful because it connects directly to the CPI and IPC: The Number Everyone Misreads story: a high miss rate with a high CPI is the signature of a memory-bound loop.
Beware two systematic distortions. First, prefetching converts what would have been demand misses into prefetch traffic, so a low demand-miss rate can coexist with high memory traffic — the data is still moving, just earlier. Second, counters are often sampled and can be attributed to the instruction after the one that actually stalled, so precise blame requires precise-event sampling rather than raw counts.
The pragmatic loop: get a per-level miss rate, decide which level is the problem, form a hypothesis about *why* from Three Kinds of Miss, Three Different Fixes, change one thing, and re-measure. Skipping the "which level" step is the most common way to spend a day optimising for the wrong constraint.
Loads retired 1.00e9 L1-dcache-load-misses 8.10e7 -> 8.1% of loads L2 misses 6.40e7 -> 79% of L1 misses escalate LLC-load-misses 5.90e7 -> 92% of L2 misses reach DRAM Reading: L1 misses are moderate, but almost all of them are going all the way out. The problem is not L1 sizing; the working set exceeds on-chip capacity entirely.
Key points
- A miss is a routine cost, not an error — every line is a miss the first time it is touched.
- Where a miss is satisfied matters far more than that it occurred; L2 and DRAM differ by more than an order of magnitude.
- Out-of-order cores continue with independent work during a miss, so independent misses are much cheaper than dependent ones.
- Use miss rates per load or per instruction, never raw counts, which scale with the amount of work done.
- Prefetching hides demand misses without reducing traffic, so a low demand-miss rate does not prove low memory pressure.
Cache Simulator
Change an input and watch which number moves — and which one refuses to.
Almost all hits. Either the working set is resident or the pattern has enough spatial locality that each line pays for many accesses.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Load → L1 tag check: the set is indexed and tags compared; a match returns data in a few cycles.
- 2L1 miss → L2: the request escalates and a fill buffer is allocated to track it, limiting how many misses can be outstanding.
- 3L2 miss → L3: the shared level is consulted, where other cores' traffic competes for capacity and bandwidth.
- 4L3 miss → memory controller: the request leaves the chip and is queued against DRAM timing constraints (How DRAM Is Organised).
- 5Return → install and resume: the line is installed on the way back and any dependent instruction can finally issue.
- • Reporting "cache misses" as a single number without knowing which level the tool counted.
- • Comparing miss counts between runs of different sizes and concluding the larger one is worse behaved.
- • Treating a low demand-miss rate as proof of good memory behaviour when prefetchers are doing the work.
- • Assuming a stall is compute because CPU utilisation is high — see Busy Is Not the Same as Working.
Consequences, controls and cost
- • A loop can stall on memory while the CPU reports full utilisation, because a stalled core still counts as busy.
- • Small increases in working set produce step changes in runtime as misses start escalating a level further.
- • Dependent-load chains are punished far more than independent ones at the same miss rate.
- • Adding cores can reduce per-core performance by increasing contention for the shared last level.
- • Determine which level is missing before changing anything; the fix differs completely per level.
- • For L1-level problems, reduce footprint or fix conflicts through layout and alignment ([[cache-mapping]]).
- • For last-level problems, restructure to reduce total traffic, or restructure to increase independence so misses overlap.
- • Break dependency chains where possible so several misses can be in flight at once ([[memory-level-parallelism]]).
- • Accept and design for bandwidth limits when the traffic is irreducible ([[bandwidth-bound-workloads]]).
- • Collect per-level counters and convert to rates: L1 misses per load, and the fraction of those that escalate further.
- • Compute misses per instruction alongside [[cpi]]; high on both is the memory-bound signature.
- • Use precise-event sampling to attribute stalls to the right instruction rather than the following one.
- • Compare demand misses against total memory traffic to see how much the prefetcher is contributing.
- • Restructuring to overlap misses adds complexity and can hurt readability for a benefit that only appears on latency-bound code.
- • Reducing footprint to keep misses at L2 often means denser encodings that cost extra instructions to unpack.
- • Optimising for one level can move pressure to another; the constraint relocates rather than disappearing.
Scope
§224 — what these claims are specific to.
- MICROARCH-SPECIFICLevel count, inclusivity, fill-buffer depth and which events the counters expose differ by vendor and generation; event names differ even between tools on the same machine
- SIMPLIFIEDThe escalation diagram omits write handling, prefetch traffic, coherence lookups and the fact that several requests are usually in flight concurrently