8 lessons

Memory Ordering & Atomics

Why the order you wrote is not the order the machine performs, what a barrier actually constrains, and the hardware primitives — compare-and-swap and friends — every lock is built from.

SourceCompilerInstructionsFront EndExecutionRegistersCachesMemoryI/OBehavior
Sequential Consistency: The Model You Already Have

Everyone reasons about shared memory as if there were one global order of operations that every core agrees on. That model is intuitive, teachable, and not what any mainstream CPU implements — which is exactly why it is worth stating precisely before taking it away.

Q · What is the mental model of shared memory that almost everyone starts with, and in what precise way is it wrong?
Why Your Loads and Stores Happen Out of Order

Two independent agents reorder your memory operations before any other core sees them: the compiler, which rewrites the code, and the CPU, which executes and retires it out of order. Neither is malfunctioning, and on one core neither is detectable.

Q · Who reorders my memory operations, which reorderings are actually permitted, and why can I never see it happening on a single thread?
Store Buffers: Where Your Writes Wait

A store instruction finishes long before the value reaches coherent cache. In between it sits in a per-core queue that only its own core can see — which is the concrete mechanism behind the one reordering even x86-64 permits.

Q · Where does a value actually go when a store instruction retires, and why can another core still read the old value afterwards?
Memory Barriers: Ordering, Not Flushing

A barrier is one of the most misdescribed instructions in computing. It does not flush caches, it does not push data anywhere, and it does not lock anything. It constrains the order in which one core's memory operations may become visible relative to each other.

Q · What does a memory barrier actually constrain, and why is "it flushes the write buffer to memory" the wrong mental model?
Hardware Memory Models Are Not Language Memory Models

Your CPU has a memory model. Your language has a different one. The compiler stands between them, and code that "works on x86 and breaks on ARM" has almost always been written against the hardware model of the machine it was tested on.

Q · What is the difference between the memory model my CPU implements and the one my language specifies, and why does code that works on x86-64 break on AArch64?
Atomic Instructions: What the Hardware Actually Guarantees

An atomic read-modify-write is a single instruction that no other core can observe half-finished. That is a narrow and precise guarantee, and it is routinely mistaken for a much broader one about program correctness.

Q · What does the hardware actually promise when an operation is atomic, and what does it conspicuously not promise?
Compare-and-Swap: The Primitive Everything Is Built On
▶ lab

Change this value, but only if it is still what I last saw. That conditional write is what makes lock-free algorithms possible, and the trap in it — that "still the same value" is not the same as "nothing happened" — is called ABA.

Q · How does compare-and-swap turn a racy read-modify-write into a safe one, and what is the failure it cannot detect?
What a Mutex Actually Does

A mutex is not an operating-system object you call into. In the uncontended case it is one atomic instruction and no system call at all — which is why an uncontended lock is nearly free and a contended one costs thousands of times more.

Q · What happens in the machine when I lock a mutex, and why is the cost so wildly different depending on whether anyone else holds it?