Memory-Mapped I/O: When a Store Is Not a Store
Device registers live in the address space, so talking to hardware looks exactly like writing to memory. It is not memory: the write has a side effect, the read may change state, and every optimisation the machine normally applies has to be turned off.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
A store with a side effect
When a device is memory-mapped, part of the physical address space routes to the device instead of to DRAM. A store to that address sends a command; a load reads a live status. The syntax is identical to ordinary memory access, and the semantics are not remotely similar.
Consider a status register. Reading it returns which events are pending — and on many devices, reading it *clears* them. A compiler that decides the second read is redundant and elides it has not optimised anything; it has discarded interrupt notifications. A compiler that hoists a read out of a loop because "nothing in the loop writes to it" has broken a polling loop, because the thing that writes to it is not in the program at all.
This is what volatile exists for in C and C++: not thread safety, which it does not provide, but a statement that this location can change or have effects outside the program's control, so every access written must actually happen, exactly as written. It is one of the few places where the keyword is doing precisely its intended job.
1// ordinary memory2uint32_t x = *counter_ptr; // load a value; may be cached, may be3 // reordered, may be elided if unused4 5// device register, memory-mapped6uint32_t s = *STATUS_REG; // reads live device state7 // - may clear pending interrupt bits8 // - must NOT be cached (stale = wrong)9 // - must NOT be elided (side effect)10 // - must NOT be reordered past the11 // command that caused the status12 13*CONTROL_REG = START_TRANSFER; // this is not a data write.14 // it is a command. the device acts on it.Every optimisation, disabled on purpose
Caching a device register would be a correctness bug: the cached copy is stale the instant the device changes state, and a write that sits in a cache is a command that never reached the device. So these regions are mapped uncacheable, which means every access goes all the way out — and is correspondingly slow. An MMIO read is not comparable in cost to a cached load; it is closer in magnitude to reaching main memory, and often worse.
Reordering is equally unacceptable. The machine may freely reorder ordinary loads and stores that it can prove independent, but a device usually requires a specific order: configure, then start; check status, then read data. Since the dependency lives in the device rather than in the data flow, the CPU cannot infer it, and the driver must state it with barriers — which is the same mechanism Memory Barriers: Ordering, Not Flushing describes, used for a different reason.
And write combining, where several stores are buffered and issued together, must be off wherever each store is an individual command — while being highly desirable for a large frame buffer, where the writes really are just data. So MMIO regions are not one thing: they are mapped with different attributes depending on what the region actually is.
| Normal optimisation | On ordinary memory | On a device register |
|---|---|---|
| Cache the line | Essential to performance | Wrong — stale reads, writes that never reach the device |
| Elide a redundant load | Free performance | Wrong — the load has a side effect |
| Hoist a load out of a loop | Free performance | Wrong — the value changes outside the program |
| Reorder independent accesses | Enables out-of-order execution | Wrong — the device requires a specific order |
| Combine adjacent writes | Fewer, wider transactions | Depends — right for a frame buffer, wrong for commands |
Why this belongs in a course about performance
Most application programmers will never write to a device register. The lesson still earns its place, because MMIO is the clearest available demonstration of how much the machine is doing on your behalf. Every optimisation listed above is invisible when it is working — you notice it only in the one place where it must be switched off.
It also sets up a distinction that recurs. The CPU's freedom to reorder and cache is bounded by what it can prove about *data* dependencies. It cannot see dependencies that live in the device, in another core, or in a signal handler. That is the same gap that makes Why Your Loads and Stores Happen Out of Order necessary between cores and makes volatile insufficient for thread safety — three consequences of one limitation.
The practical residue for ordinary code is small but real: an uncached access is expensive, so a driver polling a status register in a tight loop is doing something far more costly than the source suggests, and this is one of the reasons a polled path is usually structured to check a DMA-written flag in ordinary memory rather than the device register itself.
1while (*STATUS_REG & BUSY) {2 // each iteration is an uncached read that leaves3 // the CPU, crosses the interconnect, reaches the4 // device and comes back.5 //6 // cost per check: comparable to a main-memory access7 // or worse, and it cannot be cached8 // or pipelined away.9}1while (completion->flag == PENDING) {2 // the device DMAs the flag into ordinary memory.3 //4 // cost per check: an L1 hit, because the line stays5 // in cache until the device writes6 // it and coherence invalidates it.7 //8 // one uncached access at the end, not one per check.9}Both loops poll. One crosses the interconnect on every iteration because device registers cannot be cached; the other hits L1 until the moment the value actually changes, because coherent DMA into ordinary memory lets the cache do its job. This is why completion flags exist rather than drivers spinning on status registers.
Key points
- Device registers occupy addresses, so access looks like memory access and behaves nothing like it.
- Reads can have side effects and writes are commands, so neither may be elided, hoisted or duplicated.
- These regions are mapped uncacheable, which makes each access roughly as expensive as reaching main memory.
- The device's ordering requirements are invisible to the CPU, so the driver must state them with barriers.
- Polling a DMA-written flag in ordinary memory is far cheaper than polling a device register, because the flag can be cached.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Core → address decode: the physical address falls in a range routed to a device rather than to DRAM.
- 2Address decode → mapping attributes: the region is marked uncacheable, so the access bypasses the cache hierarchy entirely.
- 3Core → interconnect → device: the transaction leaves the package and reaches the device, which acts on it.
- 4Device → core: a read returns live state, possibly changing that state as a side effect of being read.
- 5Driver → barriers: because the ordering constraint lives in the device, the programmer must enforce it explicitly.
- • "
volatilemakes it thread-safe." It prevents the compiler from eliding or reordering the access; it provides no atomicity and no inter-core ordering. - • "A read cannot have a side effect." On device registers it very often does — reading status frequently clears it.
- • "An MMIO read is just a load." It is an uncached transaction across the interconnect, comparable to or worse than a main-memory access.
- • "The CPU will keep my accesses in order because I wrote them in order." It reorders whatever it can prove independent, and it cannot see a dependency that lives inside the device.
Consequences, controls and cost
- • An MMIO access costs orders of magnitude more than a cached load, so tight polling of a register is far more expensive than it looks.
- • Compiler optimisations that are correct on memory are miscompilations on device registers, which is why `volatile` exists.
- • Missing barriers produce intermittent device misbehaviour that depends on unrelated code changes — one of the harder bug classes to isolate.
- • Regions must be mapped with attributes matching their purpose; a frame buffer and a command register want opposite treatment.
- • Use the platform's accessor functions rather than raw pointer dereferences — they encapsulate the required barriers and attributes.
- • Poll a DMA-written completion flag in cacheable memory instead of a device status register wherever the device supports it.
- • Map each region with the attributes its role requires, rather than treating all device memory identically.
- • Treat `volatile` as "this access must actually happen as written", and never as a substitute for atomics or synchronisation.
- • Count MMIO accesses on a hot path — a per-iteration register read is usually the cost, not the surrounding logic.
- • Compare a loop polling a device register against one polling a DMA-written flag; the difference is the uncached access.
- • Check whether the mapping attributes match the region's role, particularly for large buffers that would benefit from write combining.
- • Look for device misbehaviour that appears and disappears with unrelated code changes — a classic missing-barrier signature.
- • Uncached mappings are required for correctness and cost dearly on every access; there is no way to have both.
- • Barriers constrain the CPU's reordering freedom, which costs performance on the surrounding code as well as the access itself.
- • Completion flags in RAM are cheap to poll but add a DMA write and one more thing that can be stale on a non-coherent platform.
Scope
§224 — what these claims are specific to.
- PLATFORM-SPECIFICAddress ranges, mapping attributes and register semantics are specific to each platform and device. x86-64 additionally has a legacy port I/O space with its own instructions; AArch64 has no equivalent and uses memory-mapped access exclusively.
- SIMPLIFIEDOmits write-combining buffer flush rules, posted versus non-posted transaction semantics, and the differences between device memory types that real platforms define.
Misconceptions
Where the rest of this lives
The reason volatile exists is a language-level statement about what the compiler may not optimise away; the reason it is insufficient for threads is a hardware-level fact about reordering. Both halves are needed to use it correctly.