Past the Last-Level Cache
When every cache misses, the request leaves the CPU entirely. It goes to a memory controller that queues it, reorders it against other pending requests, and drives a DRAM device that is nothing like the flat byte array your program believes in.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
The path a miss actually takes
A load starts as an address in a register. If it hits in L1 the core barely notices. If it misses all the way through the last-level cache, the request is handed to the memory controller — a unit that on modern parts lives on the CPU die itself rather than on a separate chipset, which is one reason memory latency improved substantially in the 2000s.
The controller is not a passive wire. It holds a queue of outstanding requests from every core, and it is free to service them out of order. It will happily answer a request that arrived later if that request targets a DRAM row that is already open, because doing so is dramatically cheaper than closing one row and activating another. Your load's position in that queue depends on what every other core is doing.
The return trip fills a whole cache line, not the bytes you asked for. That is why Spatial Locality pays: the expensive part of the transaction was getting the memory system to answer at all, so it answers generously. Reading one byte and reading sixty-four consecutive bytes from a cold line cost almost the same.
How far away is "far away"
The useful mental model is not nanoseconds — those are wrong on every machine but the one they were measured on, and they change with frequency scaling, memory speed and how loaded the controller is. The useful model is the ratio, which has been stable in shape for decades even as the absolute numbers moved: each level down the hierarchy costs roughly an order of magnitude more than the one above it.
The scale below is deliberately unitless. What it says is that if a register access is your unit of cost, a DRAM access is in the hundreds. That is the number that should govern design decisions: an algorithm doing ten times the arithmetic to avoid one DRAM access is usually winning, which is a conclusion that looks insane from a pure instruction-count view and is exactly why Cache-Aware Algorithms exists.
Note also that the last row is *bandwidth*-shaped rather than latency-shaped, and this matters more than it looks. See Latency and Bandwidth Are Different Resources.
Why the controller reorders, and what it costs you
The memory controller optimises for aggregate throughput across all cores, not for your thread's latency. Given a queue of requests it will group ones targeting the same open row and issue them together, because activating a row is the expensive operation and reading additional columns from an already-open row is comparatively cheap.
The consequence is that memory latency under load is not memory latency on an idle machine. A benchmark that measures a single-threaded pointer chase on a quiet box will report a number your production service never sees, because in production the queue has entries from every other core. This is one of the most common ways a microbenchmark misleads, and it has its own lesson in Every Way a CPU Microbenchmark Lies.
It also means memory access cost is genuinely non-deterministic in a way arithmetic is not. Two runs of the same loop can differ measurably purely because of what the rest of the machine was doing to the memory controller's queue.
| Behaviour | Optimises for | What it costs your thread |
|---|---|---|
| Queues requests from all cores | Utilisation of the DRAM bus | Your latency now depends on other cores' behaviour |
| Reorders to group same-row accesses | Aggregate throughput | Your request may be overtaken by a later one |
| Fills a whole cache line per miss | Amortising the expensive activation | Nothing — this is why sequential access is cheap |
| Interleaves across channels and banks | Parallelism across DRAM devices | Pathological strides can land everything on one bank |
Key points
- A last-level cache miss leaves the core entirely and is serviced by an on-die memory controller shared with every other core.
- The controller queues and reorders requests to favour DRAM efficiency, so your latency depends on the whole machine's behaviour, not just yours.
- A miss returns a full cache line, which is why reading one byte and reading a whole line from a cold line cost nearly the same.
- Think in ratios rather than nanoseconds: absolute latencies are machine-specific and go stale, the ordering does not.
- Memory latency measured on an idle machine systematically understates what a loaded production system sees.
Where the Data Is
Change an input and watch which number moves — and which one refuses to.
The exact ratios vary by machine and the absolute times vary far more, which is why none are shown. What is stable enough to build intuition on is the shape: each level is several times the one above, and the gap between the last cache level and memory is the one that decides most program performance.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Core → L1: the load address is looked up and misses.
- 2L1 → L2 → LLC: each level is checked in turn, each miss adding cost before the request even leaves the CPU.
- 3LLC → memory controller: the request for a full cache line is queued alongside requests from every other core.
- 4Memory controller → DRAM: commands are issued, possibly out of arrival order, favouring rows that are already activated.
- 5DRAM → core: a burst returns, the cache line is filled at every level, and the waiting instruction finally gets its operand.
- • "RAM access time is constant." It varies with row state, controller queue depth, channel conflicts and what other cores are doing.
- • "The miss cost is the number in the table I read online." That number came from one machine at one frequency under one load; treat it as a ratio, not a measurement.
- • "My benchmark says memory is fast." It was probably measured on an idle machine with a hot row buffer and a working set that fit in cache.
Consequences, controls and cost
- • Loops whose working set exceeds the last-level cache degrade sharply rather than gradually, because every iteration now pays a DRAM round trip.
- • The same code shows different memory latency on a busy machine than on an idle one, with no code change involved.
- • Reading data you did not need is nearly free if it shares a line with data you did, and ruinously expensive if it does not.
- • Shrink the working set so it fits a cache level — the single highest-leverage change, and the subject of [[working-set]] and [[matrix-tiling]].
- • Access memory in patterns the prefetcher can predict, so the round trip overlaps with useful work rather than stalling on it.
- • Increase memory-level parallelism by removing dependencies between loads, so several misses are outstanding at once rather than serialised.
- • Accept it and measure: for genuinely random access over a large dataset, DRAM latency is the floor and no amount of code tuning removes it.
- • Count last-level cache misses with hardware counters and divide by instructions retired — a rising ratio localises the problem to the memory system.
- • Compare a run with the working set sized to fit the last-level cache against one just above it; a discontinuity confirms you are crossing into DRAM.
- • Measure on a loaded machine, not an idle one, if you care about the production number.
- • Restructuring data to fit a cache level usually costs abstraction — flatter, less expressive structures that are harder to change.
- • Increasing memory-level parallelism often means unrolling or restructuring loops, which costs readability and can hurt instruction cache pressure.
Scope
§224 — what these claims are specific to.
- SIMPLIFIEDPresents one controller and one DRAM device. Real systems have multiple channels, ranks and banks with interleaving policies that vary by platform and firmware configuration.
- PLATFORM-SPECIFICOn-die memory controllers are standard on modern x86-64 and AArch64 server and desktop parts; older systems placed the controller in a separate northbridge, with materially higher latency.