When the Memory Bus Is the Bottleneck
Streaming code that touches each byte once cannot be helped by caches, cannot be helped by more cores, and cannot be helped by faster arithmetic. It is limited by how fast bytes arrive, and the only real lever is moving fewer of them.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
The shape of a bandwidth-bound loop
The defining property is low arithmetic intensity: few operations per byte loaded. Summing an array performs one add per element read. Scaling a vector performs one multiply. Copying performs none. In all of these the execution units finish their work long before the next cache line arrives, so they idle.
Caches do not help, because there is no reuse to exploit. Temporal Locality is what makes a cache valuable, and streaming code has none by construction — each byte is touched once and never again. The cache still helps with Spatial Locality, in that a whole line arrives per miss and all of it gets used, but it cannot reduce the total bytes that must cross the bus.
Once the bus saturates, the workload has hit a hard ceiling. This is the regime where the roofline mental model applies: below a certain arithmetic intensity, performance is bounded by bandwidth rather than by compute, no matter what the CPU is capable of.
1// copy: 0 arithmetic ops per 8 bytes moved2for (i = 0; i < N; i++) dst[i] = src[i]3 4// scale: 1 multiply per 8 bytes read + 8 written5for (i = 0; i < N; i++) dst[i] = a * src[i]6 7// sum: 1 add per 8 bytes read8for (i = 0; i < N; i++) total += src[i]9 10// By contrast, a dense matrix multiply performs O(n) operations11// per element loaded once tiled -- high arithmetic intensity,12// and therefore compute-bound rather than bandwidth-bound.13// See [[matrix-tiling]].Why adding cores stops helping
Memory bandwidth is a shared resource. Cores have private L1 and usually private L2, but they share the last-level cache, the memory controller and the DRAM bus. A single core running a streaming kernel can often consume a large fraction of the available bandwidth on its own; two or three can saturate it entirely.
Past that point, adding cores adds queueing rather than throughput. Each additional thread issues requests that wait longer in the controller queue, so per-thread performance falls roughly in proportion to the number of threads, and aggregate throughput stays flat. Worse, more concurrent streams mean more row conflicts in DRAM (see How DRAM Is Organised), so aggregate throughput can actually *decline*.
This is a genuinely different scaling curve from a compute-bound workload, and recognising it saves a great deal of wasted effort. It is also why "more cores means linear speedup" is a red flag in an interview.
| Observation | Bandwidth-bound | Parallelisation bug (contention, false sharing) |
|---|---|---|
| Speedup at 2 threads | Roughly linear | Often already poor |
| Speedup at 8 threads | Flat — a hard ceiling | Flat or negative, and erratic |
| Achieved memory bandwidth | Near the machine's streaming ceiling | Low |
| CPU utilisation | High, but mostly stall cycles | High, spinning or coherence traffic |
| Coherence / cross-core traffic | Low | High — see False Sharing: Independent Data, Shared Line |
| Fix | Move fewer bytes | Fix sharing, padding or lock granularity |
The only lever that works: fewer bytes
When bandwidth is the limit, the only thing that helps is reducing the bytes that cross the bus. Every effective optimisation for this regime is a variant of that idea: use narrower types where precision allows, pack structures tightly (see Padding: Why Your Struct Is Bigger Than Its Fields), read only the fields you need rather than whole records (see Array of Structs, or Struct of Arrays?), or compress data and spend otherwise-idle CPU cycles decompressing it.
The compression case is worth dwelling on because it is counter-intuitive. Adding work — decompression — makes a bandwidth-bound loop faster, because the execution units were idle anyway and the bus was the constraint. This is precisely the trade columnar databases make, and it is why Cache-Aware Algorithms sometimes recommends doing *more* arithmetic.
The other genuine lever is increasing reuse so that bytes crossing the bus once get used many times. That is what tiling does, and it converts a bandwidth-bound kernel into a compute-bound one — the subject of Matrix Tiling: Same Arithmetic, Ten Times Faster.
- Narrower types: 32-bit floats instead of 64-bit halves the bytes if precision permits.
- Tighter layout: remove padding, reorder fields, split hot fields from cold ones.
- Field selectivity: struct-of-arrays so a scan reads only the columns it uses.
- Compression: spend idle CPU to reduce bus traffic — a win only in this regime.
- Increase reuse: tiling and blocking so each loaded byte serves many operations.
Key points
- Bandwidth-bound means low arithmetic intensity: few operations per byte, so the bus saturates before the execution units do.
- Caches cannot rescue streaming code because there is no temporal reuse to exploit.
- Memory bandwidth is shared, so a couple of cores can saturate it and further cores add only queueing.
- Flat scaling with near-peak bandwidth is bandwidth saturation; flat scaling with low bandwidth is a different bug entirely.
- The only effective fix is moving fewer bytes — narrower types, tighter layout, field selectivity, compression, or more reuse.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Loop → cache: each iteration requests the next element; the line arrives and every byte of it is used exactly once.
- 2Cache → controller: because there is no reuse, essentially every line is a compulsory miss that must come from DRAM.
- 3Controller → bus: requests from all active cores queue for the same finite bytes-per-second.
- 4Bus saturation → stall: execution units sit idle waiting for lines, so instructions retire slowly despite high apparent CPU utilisation.
- 5More cores → more queueing: aggregate throughput plateaus and per-thread latency rises proportionally.
- • "CPU utilisation is high, so we are compute-bound." Utilisation counts cycles where a thread is scheduled, including cycles spent stalled on memory.
- • "Scaling is flat, so our parallel code is broken." Check achieved bandwidth first; a saturated bus produces exactly this signature with correct code.
- • "Vectorising the loop will help." SIMD makes arithmetic wider, but the loop was never limited by arithmetic.
Consequences, controls and cost
- • Parallel speedup plateaus at a small number of threads, regardless of how many cores the machine has.
- • Micro-optimising the arithmetic inside the loop produces no measurable improvement.
- • The same code runs at very different speeds on machines with different memory configurations even at identical clock speeds.
- • Reduce bytes moved: narrower types, packed layouts, and reading only the fields actually needed.
- • Increase arithmetic intensity through tiling or fusion so each byte loaded does more work before being discarded.
- • Compress data and decompress on the fly, trading idle CPU cycles for scarce bandwidth.
- • Stop adding threads once bandwidth saturates — measure the ceiling and size the thread pool to it.
- • Measure achieved bytes-per-second during the hot loop and compare against a streaming benchmark ceiling on the same hardware.
- • Sweep thread count and plot aggregate throughput; a plateau with high bandwidth confirms saturation.
- • Compute arithmetic intensity — operations per byte loaded — directly from the loop body as a first-pass classification.
- • Compression and narrower types cost precision, CPU cycles or both, and are wins only while bandwidth is genuinely the constraint.
- • Tiling improves reuse but complicates code substantially and introduces parameters that must be retuned per machine.
Scope
§224 — what these claims are specific to.
- GENERALThe saturation mechanism applies to any shared memory system. How many cores are required to saturate is PLATFORM-SPECIFIC and depends on channel count, memory speed and core capability.
Misconceptions
Apply it
Where the rest of this lives
Bandwidth saturation is a shared-resource limit rather than a synchronisation limit, but it produces the same flat scaling curve — worth distinguishing when diagnosing why a parallel speedup stalled.