Store Buffers: Where Your Writes Wait
A store instruction finishes long before the value reaches coherent cache. In between it sits in a per-core queue that only its own core can see — which is the concrete mechanism behind the one reordering even x86-64 permits.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Why the buffer has to exist
A store cannot complete until the writing core owns the target cache line exclusively — otherwise another core could hold a stale copy. Acquiring ownership means a coherence transaction across the interconnect, which is orders of magnitude more expensive than executing an instruction. If the core stalled on every store until ownership arrived, throughput would collapse.
The store buffer removes that stall. The store retires into the buffer, the core moves on immediately, and the coherence work proceeds in the background. Retirement and visibility are decoupled: the instruction is architecturally complete while the value is still private to this core. That decoupling is the whole point, and the ordering weakness is the price.
It also explains the asymmetry in x86-64's memory model. Stores are buffered and drain later, so a later load can complete before an earlier store becomes visible — Store→Load reordering. Loads are not buffered in the same way, so the other three orderings can be maintained cheaply. TSO is not an arbitrary choice; it is close to the weakest model you can get away with while keeping a store buffer.
Store forwarding, and the illusion it maintains
If stores are invisible until they drain, why does a thread reading back its own store never see stale data? Because the load path checks the store buffer first. A load that matches a pending store in the same core's buffer takes the value directly from it rather than from cache — store forwarding. This is what preserves the single-thread guarantee described in Why Your Loads and Stores Happen Out of Order.
Store forwarding is also a performance feature with sharp edges. Forwarding typically succeeds cleanly when the load matches a single pending store in size and alignment. When it does not — a wide load overlapping several narrower stores, or a misaligned overlap — the core may have to wait for the stores to drain before the load can complete. This is a store-forwarding stall, and it is a real, measurable effect in code that writes bytes and reads words, or that type-puns through a union. See Alignment: Why Addresses Are Not Arbitrary for the layout side of it.
The important consequence is that the illusion is strictly local. Store forwarding makes *your* loads consistent with *your* stores. It does nothing for another core, which has no access to your buffer and will keep reading the old value from its own cache until your store drains and coherence invalidates it.
Initially x = 0, y = 0. Both lines start Shared in both caches.
Core 0 Core 1
------ ------
store x = 1 store y = 1
-> enters Core 0 store buffer -> enters Core 1 store buffer
x still 0 in coherent cache y still 0 in coherent cache
load r1 = y load r2 = x
-> checks own store buffer: -> checks own store buffer:
no pending store to y no pending store to x
-> reads coherent cache: 0 -> reads coherent cache: 0
RESULT: r1 == 0 && r2 == 0
Both stores drain some cycles later. Nothing malfunctioned:
each core's own program order is intact, each core would
read back its own store correctly via forwarding, and the
only casualty is the existence of a single global order.Draining it on purpose
A full memory fence is, in practice, an instruction that will not let subsequent memory operations proceed until the store buffer has drained. On x86-64 MFENCE does this, and so does any LOCK-prefixed instruction as a side effect — which is why a LOCK XCHG is a common and often cheaper idiom for a full barrier than MFENCE itself. See Memory Barriers: Ordering, Not Flushing for what a fence constrains in general and Atomic Instructions: What the Hardware Actually Guarantees for the operations that carry ordering intrinsically.
That is why fences are not free and why their cost is workload-dependent rather than fixed. A fence with an empty store buffer is cheap. A fence with a full buffer, whose entries are still waiting on coherence transactions for lines owned by other cores, can stall for a long time. The same instruction in the same binary has wildly different costs depending on what the surrounding code has been writing, which is exactly why microbenchmarking a fence in isolation tells you very little — see Every Way a CPU Microbenchmark Lies.
A useful mental correction follows: a fence does not "flush writes to memory" in the sense of pushing them out faster. The stores were already draining as fast as coherence allowed. What the fence does is make *this core* wait until that has happened before proceeding. It buys ordering by spending time, and it does not accelerate anything.
Key points
- A store retires into a per-core store buffer and becomes globally visible only when that entry drains — retirement and visibility are separate events.
- The buffer exists so a core need not stall waiting for exclusive ownership of a cache line on every write.
- Store forwarding lets a core read back its own pending stores, which is why a single thread can never observe the delay.
- This one mechanism is the reason x86-64 permits Store→Load reordering and is therefore not sequentially consistent.
- A fence does not accelerate draining; it makes the current core wait until draining has finished, so its cost depends on what is pending.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Store instruction → store buffer: the store retires into a FIFO entry and the core proceeds without waiting for the cache line.
- 2Store buffer → coherence: the entry requests exclusive ownership of the target line from the coherence protocol.
- 3Own load → store buffer: a matching load is satisfied by forwarding from the pending entry rather than from cache.
- 4Coherence → L1: ownership arrives, the entry writes into the cache line and leaves the buffer.
- 5L1 → other cores: only now can another core's load observe the new value; until then it reads its own cached copy.
- • "The store did not happen yet" — it has architecturally completed; it is only not yet *visible* to other cores. Retirement and visibility are different events.
- • "A fence flushes the cache" — it does not touch the cache hierarchy's contents; it constrains the order in which this core's operations proceed.
- • "Store buffers are an x86 thing" — essentially every out-of-order core has one; x86-64 simply has a memory model strict enough that the buffer is the *only* visible source of reordering.
- • "If I see a store-forwarding stall I should add a fence" — a fence makes it strictly worse; the fix is to make the load and store widths match.
Consequences, controls and cost
- • Store→Load reordering is observable between cores on every mainstream ISA, including the ones people describe as strongly ordered.
- • Code that writes narrow and reads wide — byte writes followed by a word read, or union type-punning — can hit store-forwarding stalls that look inexplicable in a profile.
- • Fence cost varies enormously with surrounding write traffic, so a fence benchmarked in isolation is not representative.
- • A write is not observable by another thread at the instant the instruction completes, no matter how the source reads.
- • Use acquire/release atomics rather than raw fences: they let the compiler emit the cheapest encoding that gives the ordering you actually need.
- • Avoid mixing access widths to the same location — write and read it at a consistent size to keep store forwarding on its fast path.
- • Batch stores to the same cache line together rather than interleaving them with stores to other contended lines, so entries drain without repeated ownership transfers.
- • Accept that you cannot control drain timing directly; you can only choose whether to wait for it. Measure instead of guessing.
- • On x86, sample store-forwarding stall events (Intel exposes counters in the `LD_BLOCKS` family) to see whether narrow-write/wide-read patterns are costing you.
- • Run the store-buffer litmus test with and without a fence between the store and the load, and count how the outcome distribution changes.
- • Compare the cost of the same fence instruction with an empty versus a busy store buffer to see the variance directly.
- • Read the disassembly to check whether your atomic became a `LOCK`-prefixed instruction, a plain store, or an explicit fence — the encoding tells you what you are paying.
- • The store buffer buys large throughput gains at the cost of the one reordering that makes reasoning about shared memory harder.
- • Draining it early with fences restores ordering but spends exactly the cycles the buffer was designed to save.
- • Matching access widths for forwarding can conflict with a compact data layout, which is a real trade against cache footprint.
Scope
§224 — what these claims are specific to.
- MICROARCH-SPECIFICDepth, drain policy, coalescing and forwarding rules vary by vendor and generation. Nothing here is architecturally guaranteed except the observable ordering consequences.
- ISA-SPECIFICThat the store buffer is the *only* source of visible reordering is an x86-64 TSO property. On AArch64 and POWER other mechanisms reorder as well, so draining the store buffer is not sufficient for ordering.
- SIMPLIFIEDReal cores have load queues, fill buffers, write-combining buffers and multiple store-buffer stages. This model keeps the single queue that explains the observable ordering and omits the rest.
Misconceptions
Where the rest of this lives
The store buffer is the hardware reason a plain flag write does not publish anything. What a correct publication protocol looks like, and how to prove it, belongs to concurrency.