Executionfetchfront endinstruction cachefetch widthprogram counter

Instruction Fetch: Code Is Data Too

Before a CPU can do anything with an instruction it has to load it from memory, through a cache, at an address it may have had to guess. The front end is a supply chain, and a starved front end leaves the most sophisticated execution engine in the world with nothing to do.

Follow the mechanism

Software view, hardware view

The gap between what you wrote and what the machine does is where this whole domain lives.

The question
Where do instruction bytes come from, and what happens to a program when the CPU cannot get them fast enough?
What you wrote
Instructions feel free. You think about the cost of the *work* an instruction does — the multiply, the memory access — and never about the cost of the CPU obtaining the instruction in the first place.
What the hardware does
The fetch unit issues a read to the instruction cache for a block of bytes at the current program counter, every cycle, in parallel with everything else the machine is doing. That read can miss, the address can be a guess, and the block can contain fewer useful instructions than the machine can consume.
A program whose hot code does not fit in the instruction cache, or whose control flow jumps unpredictably, is limited by instruction supply rather than by the work the instructions do. No amount of arithmetic optimisation helps, because the execution units are already idle waiting for something to execute.
Source→Compiler→Instructions→Front End→Execution→Registers→Caches→Memory→I/O→Behavior

The front end is a supply chain

Fetch reads a *block* of bytes — a fixed width per cycle, sized by the microarchitecture — not one instruction. The block is decoded into as many instructions as it contains. This is the machine's supply rate, and it sets a hard ceiling: a core that fetches enough bytes for four instructions per cycle can never sustain more than four instructions per cycle, no matter how many execution units it has.

That block comes from the instruction cache, which is a separate cache from the data cache at the first level on essentially all designs. Separate, but not free: both are backed by the same unified caches further out, so a program with a large data working set can evict the code it is about to run, and vice versa. This is the mechanism behind Your Code Is Data Too pressure being a real and frequently missed cost.

The address for the fetch comes from the program counter — and for anything other than straight-line code, the CPU does not yet know what the next address is. It predicts it, fetches from the predicted address, and continues. That is not an optimisation bolted on the side; it is the only way a pipelined front end can operate at all, which is why Branch Prediction: Guessing Well Enough to Matter belongs in the fetch story rather than as an advanced topic.

predicted next addressread blockhitmissfillBranch PredictorProgram CounterL1 Instruction CacheUnified L2Fetch BufferDecoders
UserLLMAgentToolDataDecisionHumanGuardrail

Ways the supply breaks down

MICROARCH-SPECIFICFetch width, instruction cache capacity, and whether a decoded-instruction cache exists all differ per microarchitecture. Some x86 designs cache decoded micro-operations and can bypass fetch and decode entirely for tight loops; most in-order embedded cores have none of this machinery.

An instruction cache miss is the expensive one. The core has nothing to execute and must wait for the line to arrive from an outer cache or from memory — the same latency any data miss would pay, but with the entire pipeline idle behind it. Large binaries, deep call chains through cold code, and aggressive inlining that bloats hot loops all push in this direction.

A misprediction is the common one. The predictor guessed wrong, so everything fetched after the branch is discarded and fetch restarts at the correct address. The cost is the time to refill the pipeline, which scales with pipeline depth (Misprediction: What a Wrong Guess Costs).

A fetch bandwidth limit is the subtle one. Even with every access hitting, a fetch block that straddles a cache line boundary or contains a taken branch early may yield only one or two useful instructions instead of a full block. Code laid out so that hot paths are contiguous fetches better than code where the hot path jumps over cold error-handling — which is one concrete reason profile-guided layout produces measurable wins on large binaries.

Three front-end problems that look identical from source code
ProblemCauseSignalWhat helps
I-cache missHot code too large or scatteredFront-end stalls, instruction-fetch miss countersSmaller hot path, hot/cold splitting, less aggressive inlining
MispredictionData-dependent branch the predictor cannot learnBranch-miss counter, high stalls after branchesMake the branch predictable, or remove it (Branchless Code: A Trade, Not an Upgrade)
Fetch bandwidthTaken branches early in a block; poor layoutLow IPC with few misses of any kindProfile-guided layout, straight-line hot paths

Why this is invisible from source

Nothing in a function's text tells you how large its compiled form is, where the compiler placed it, or how far it sits from the function it calls in a loop. Two implementations with identical logic and identical data access can differ substantially in front-end behaviour purely because of code size and layout.

The example below is the shape that catches people: an "optimisation" that adds a rarely-taken special case. The logic is strictly better — the special case is genuinely faster when it hits. But the extra code lands in the middle of the hot loop body, and the loop no longer fits as neatly in the instruction cache. Whether this is a net win depends entirely on the hit rate of the special case and the size of the loop, which is a measurement question, not a reasoning question.

The practical stance: treat code size as a resource that hot loops spend, exactly like registers or cache capacity. It is the one front-end factor an application programmer can influence without leaving the source language, and it is why Your Code Is Data Too and inlining decisions belong together.

Special case inlined into the hot loop
1for (i = 0; i < n; i++) {
2    if (rare_condition(a[i])) {
3        // 200 instructions of specialised handling,
4        // taken on ~0.1% of iterations
5        handle_special_inline(a[i]);
6    } else {
7        sum += a[i];
8    }
9}
Special case moved out of line
1for (i = 0; i < n; i++) {
2    if (unlikely(rare_condition(a[i]))) {
3        handle_special(a[i]);   // not inlined; lives elsewhere
4    } else {
5        sum += a[i];
6    }
7}

Both versions execute the same instructions on the 99.9% path. The difference is that in the first, those 200 rarely-executed instructions occupy cache lines *inside* the loop body, so every iteration fetches around them. Moving the cold path out of line leaves the hot path contiguous. Whether this matters depends on loop size and cache capacity — measure before assuming either direction.

Key points

  • Fetching instructions is a memory access: code occupies cache, and code and data compete for it in the shared levels.
  • Fetch width sets a hard ceiling on instructions per cycle regardless of how many execution units the core has.
  • The fetch address for non-straight-line code is predicted, not known — prediction is intrinsic to fetching, not an optimisation.
  • Three distinct front-end problems (i-cache miss, misprediction, fetch bandwidth) look identical in source and need different fixes.
  • Code size is a resource hot loops spend; inlining trades call overhead for front-end pressure.

Follow the mechanism

The path through the machine, hop by hop — and the conclusions it invites that are wrong.

  1. 1
    Predictor → PC: for anything but straight-line code, the branch predictor supplies the address to fetch from before the branch has resolved.
  2. 2
    PC → L1 instruction cache: the fetch unit requests a fixed-width block of bytes at that address.
  3. 3
    L1 I-cache → fetch buffer: on a hit, bytes are delivered in a cycle or two; on a miss, the request goes to the unified L2 and the front end stalls.
  4. 4
    Fetch buffer → decoders: the block is split into instruction boundaries and handed to the decoders, yielding between one and fetch-width instructions.
  5. 5
    Decoders → back end: if the front end supplies fewer instructions than the back end can consume, the machine is front-end bound and execution units idle.
What people conclude from this — wrongly
  • • "The loop is slow so the arithmetic must be expensive" — the execution units may be idle waiting for instructions.
  • • "Inlining is an optimisation" — it removes call overhead and adds code size; which dominates is a property of the specific loop.
  • • "Only data has cache behaviour" — instruction supply has its own cache, its own misses, and its own bandwidth limit.

Consequences, controls and cost

What it causes
  • • Large binaries and deep, cold call chains run slower than their instruction count suggests, with no algorithmic explanation.
  • • Aggressive inlining can make code slower by inflating the hot path beyond the instruction cache.
  • • A program can show low IPC and low cache miss rates simultaneously — the tell for a fetch bandwidth or layout problem.
What you can do
  • • Keep the hot path contiguous: move cold error handling and rare special cases out of line so the loop body stays compact.
  • • Use profile-guided optimisation on large binaries — its main benefit is code layout, and layout is exactly what the front end is sensitive to.
  • • Treat inlining as a trade rather than a win, and measure both directions on the actual hot loop.
  • • For most application code the honest answer is that the compiler already does this better than you can; the value is in recognising the symptom rather than hand-tuning.
How to see it
  • • Read front-end stall counters if the vendor exposes them; a top-down breakdown attributing stalls to the front end is the direct signal.
  • • Read the instruction-fetch miss counter alongside the data miss counter — they are separate events and confusing them misdirects the whole investigation.
  • • Compare IPC against branch-miss and cache-miss rates: low IPC with low misses of both kinds points at fetch bandwidth or layout.
  • • Measure binary and hot-function size across builds; large jumps in hot-path size are worth correlating with performance changes.
What it costs
  • • Optimising layout by hand is fragile: it depends on compiler version, flags and the profile used, and it decays as the code changes.
  • • Moving cold paths out of line makes control flow less obvious to a reader and can complicate debugging.
  • • Profile-guided optimisation requires representative profiles and a more complex build; an unrepresentative profile can make things worse.

Scope

§224 — what these claims are specific to.

What these claims are specific to
  • MICROARCH-SPECIFICFetch width, instruction cache size and the presence of a decoded micro-operation cache vary per design. Some x86 cores can serve tight loops entirely from a micro-op cache, bypassing fetch; most embedded in-order cores have neither.
  • GENERALThat instructions are fetched from memory through a cache, and that the front end can starve the back end, is true of every cached processor.

Misconceptions

Claim
“Instructions are free; only the work they do costs anything.”
Reality
Instruction bytes are read from a cache that can miss and has finite bandwidth. A front-end-bound program has idle execution units, and no amount of cheaper arithmetic will help it.
Claim
“The instruction cache and data cache are the same cache.”
Reality
At the first level they are separate on essentially all designs, which is why code and data do not evict each other in L1. They do share the unified outer levels, so the competition is real, just one level removed.
Claim
“Inlining is always faster because it removes a call.”
Reality
It removes call overhead and increases code size. In a hot loop that already fits in cache, growing the body can cost more than the call ever did. It is a measurement, not a rule.

Where the rest of this lives

Programming Languages & Runtime Internals
Inlining and code layout decisions

The compiler or JIT decides how large your hot loop is and where the cold paths live. Those decisions are made before the CPU ever fetches a byte, and they are the main determinant of front-end behaviour.

← Fetch, Decode, Execute — and Why That Story Is IncompleteDecode: Turning Bytes Into Intent →