Speculative Execution: Doing Work Before You Know You Need It
Modern CPUs execute instructions they may have to discard, on the bet that the guess was right. It works because the guess usually is, and because discarded results never become architecturally visible. What they do leave behind — microarchitectural traces — turned out to matter enormously.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Speculate, then commit or discard
The machine executes down the predicted path immediately, producing results into internal, non-architectural storage. Registers written by speculative instructions are physical registers, not yet mapped to the architectural names the program can observe (Register Renaming). Stores go into a buffer, not to memory.
When the branch resolves correctly, those instructions retire in program order, and their results become architecturally visible (The Reorder Buffer and Precise State). When it resolves wrongly, they are squashed and their physical registers reclaimed. From the program's point of view, they never happened.
That two-phase structure — execute eagerly, commit conservatively — is what makes speculation safe for correctness. It is also what allows the machine to speculate deeply: it can be wrong about a great deal of work without the program ever being able to tell, which means the design can afford aggressive guesses.
What speculation buys
Without speculation, a core would stall at every unresolved branch — several cycles, every handful of instructions. Measured against that baseline, speculation is not a marginal optimisation; it is most of the single-thread performance of a modern core.
It also enables speculation on *memory*: a load can be issued before it is certain that an earlier store does not write the same address. If it turns out they aliased, the load is squashed and replayed. Since most loads and stores do not alias, guessing pays overwhelmingly.
And it is what allows the memory hierarchy to be hidden. Running ahead means multiple cache misses can be outstanding at once (Misses That Overlap Are Nearly Free) rather than serialised. A machine that could not speculate would pay full memory latency for every miss in sequence, which is precisely why Pointer Chasing: The Address You Do Not Have Yet — where speculation cannot help — is so much slower than array traversal.
| Speculation | The guess | If wrong |
|---|---|---|
| Branch direction | Taken or not taken | Squash and refill (Misprediction: What a Wrong Guess Costs) |
| Branch target | Destination of an indirect transfer | Squash and refill |
| Memory disambiguation | This load does not alias an earlier pending store | Squash and replay the load |
| Data prefetch | This address will be needed soon | Wasted bandwidth and a possibly evicted useful line |
Architecturally invisible is not invisible
Squashed instructions leave no architectural trace: no register the program can read, no memory the program can load. But they were really executed, and execution has effects the architecture does not describe. A speculative load that missed brought a line into cache, and that line is still there. A speculative branch trained the predictor, and that training persists.
Those residues are measurable by *timing*. Code that can time its own memory accesses can determine which lines are cached, and therefore infer something about addresses touched by speculative execution that was supposed to be discarded. That is the whole mechanism, and it is why Side Channels: When Performance Optimisations Leak and Spectre and Meltdown: When Speculation Crossed a Boundary belong in a hardware curriculum rather than only a security one.
The lesson to carry forward is a general one about abstraction. The architectural specification is a contract about *results*, not about *resources*. Anything observable through timing, power or contention was never covered by that contract — and a great deal is observable through timing. This is the sharpest example in the whole domain of an abstraction being correct and still leaking.
- Squashed work is architecturally erased: no register, no memory location, nothing the ISA defines is affected.
- Squashed work is not microarchitecturally erased: cache occupancy, predictor state and buffer contents persist.
- Timing turns residue into information: measuring access latency reveals what is cached.
- Mitigations cost performance: because the mechanism being restricted is the same one delivering the speed.
- This is not a bug in one CPU: it is a consequence of speculating, and it required architectural rethinking rather than a patch.
Key points
- Speculation executes past unresolved branches and commits results only at retirement, so wrong guesses are architecturally invisible.
- It provides much of modern single-thread performance, and enables multiple outstanding cache misses.
- The machine speculates about branch direction, branch targets, and whether loads alias pending stores.
- Discarded work still perturbs caches and predictors, and those perturbations are measurable through timing.
- The ISA contract covers results, not resource usage — which is exactly the gap side channels exploit.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Predictor → speculative path: the front end fetches and issues instructions past an unresolved branch.
- 2Execution → physical registers: results are written to renamed physical registers, invisible to the architectural state.
- 3Reorder buffer → retirement: when the branch resolves correctly, instructions commit in program order and become visible.
- 4Wrong prediction → squash: speculative instructions are invalidated and their resources reclaimed.
- 5Speculative memory access → cache state: lines fetched speculatively remain resident even after the instructions are squashed.
- • "Speculation is an optional optimisation" — remove it and modern single-thread performance largely disappears.
- • "Wrong-path work is wasted energy but harmless" — architecturally harmless, microarchitecturally observable.
- • "Spectre was a bug that got fixed" — it was a consequence of a design approach, and mitigations trade performance rather than eliminating the mechanism.
Consequences, controls and cost
- • Single-thread performance is far higher than a non-speculating design could achieve on the same silicon.
- • Multiple cache misses can be in flight simultaneously, which is what makes sequential array traversal fast.
- • Microarchitectural residue from discarded work is observable by timing, enabling cross-boundary information leaks.
- • For performance: give the machine predictable control flow and independent work so speculation succeeds and pays off.
- • For security: rely on platform mitigations and keep microcode and OS updated rather than attempting application-level defences.
- • In code handling secrets, avoid secret-dependent memory access patterns and secret-dependent branches — the standard constant-time discipline.
- • For most application code the honest answer is that this is not yours to control; understanding it matters more than acting on it.
- • Compare speculative and retired instruction counts where both are exposed; the gap is speculation that was discarded.
- • Read mispredict counters to estimate how often speculation fails in a given workload.
- • Measure the cost of platform mitigations by benchmarking with them enabled and disabled, where the platform permits it.
- • Deeper speculation means better performance and a larger window of exploitable microarchitectural state.
- • Mitigations reduce exposure at a real and sometimes substantial performance cost, particularly for syscall-heavy workloads.
- • Constant-time coding disciplines are restrictive and hard to verify, and they are only warranted where secrets are actually handled.
Scope
§224 — what these claims are specific to.
- MICROARCH-SPECIFICSpeculation depth, memory disambiguation policy and mitigation mechanisms differ per design and per microcode revision. In-order cores speculate minimally and are correspondingly less exposed and less fast.
- PLATFORM-SPECIFICWhich mitigations are active, and their cost, depends on CPU model, microcode, OS version and configuration — the same binary can perform quite differently across two machines with the same nominal CPU.