Follow a File Read
One read(fd, buf, 4096) goes from a libc wrapper through a mode switch into the VFS, the file system’s block map and the page cache; a hit is a microsecond memory copy, a miss is a block-layer request, a DMA transfer and a wake-up ~100 µs later on an SSD.
The problem
read() of 4 KiB takes 1 µs the second time and 100 µs — or 8 ms on a spinning disk — the first time. Nothing in your code changed. Every layer between your buffer and the flash chip has a say in which one you get.Progressive depth
The same mechanism at different altitudes — start where you are.
Your program asks the OS for bytes from a file. The OS checks whether it already has them in memory; if so it copies them over immediately, and if not it asks the disk, waits, keeps a copy in memory for next time, and then gives you yours. That is why the first read of a file is slow and the rest are fast.
From the call to the kernel
read(3, buf, 4096) in C, f.read(4096) in Python, fs.readSync in Node: each ends in the same place. The libc wrapper puts the system call number and the three arguments in registers and executes the CPU’s trap instruction (syscall on x86-64, svc on ARM64). The CPU switches to kernel mode, jumps to the kernel’s entry point, and the kernel dispatches to sys_read (System Calls). Cost so far: roughly 100–300 ns on a modern CPU, more with speculative-execution mitigations enabled — the floor for any I/O call, hit or miss.
sys_read validates the descriptor against the process’s table (File Descriptors), takes a reference on the open file description, reads its current offset, and calls the VFS — the virtual file system layer that gives every file system one interface. The VFS dispatches to the specific file system’s read operation (ext4_file_read_iter, xfs_file_read_iter); nearly all of them delegate the common case to a generic routine that works in units of pages.
The offset and length are turned into a page index: byte 8192 with 4 KiB pages is page 2 of this file. From here on, the kernel is not thinking about your 4096-byte request; it is thinking about whether page 2 of inode 918273 is in memory.
- Application`read()` / `f.read()`; user buffer at some virtual address↓
- libc → syscall instructionmode switch, ~100–300 ns↓
- Kernel: sys_read → VFSvalidate fd, take offset from the open file description↓
- File systembyte range → page index → (via extents) physical block↓
- Page cache lookuphit: copy to user buffer, return. miss: allocate a page, submit I/O, sleep↓
- Block layer → driver → devicerequest queue, NVMe submission queue entry, flash read, DMA into the page↓
- Interrupt → wake → copycompletion interrupt, process becomes runnable, copy 4 KiB to the user buffer
The page cache: hit or miss
The page cache is a kernel-wide map from (file object, page index) to a 4 KiB page frame in RAM, kept in a radix-tree-like structure (an xarray on Linux) per file. On a hit, the kernel copies the requested bytes from that frame into your buffer — copy_to_user, a bounds-checked memcpy — updates the offset in the open file description, and returns. Total round trip for a 4 KiB hit: about 1–2 µs, most of it the two mode switches and the copy. This is the case your code sees 99% of the time on a warm server, and it is why "reading a file" feels free.
On a miss, the kernel allocates a page frame (evicting something under Memory Pressure, Swap and the OOM Killer if needed), inserts it into the cache marked "not yet up to date", asks the file system to map page 2 to a physical block via the inode’s extent tree (Inodes), builds a bio describing "read 8 sectors starting at block 1,048,834 into this page", and submits it to the block layer. Your thread is then put to sleep in the uninterruptible-wait state (D in ps) (Process States) — it has nothing to do until the bytes arrive.
The block layer queues the request, possibly merging it with adjacent ones, and hands it to the device driver. For NVMe the driver writes a command into a submission queue in memory and rings a doorbell register; the drive reads the command, fetches the flash pages, and DMAs the data directly into the page frame with no CPU involvement, then posts a completion and raises an interrupt. The interrupt handler marks the page up to date and wakes your thread; the scheduler runs it; it copies the bytes out and returns. Miss cost on NVMe: ~50–100 µs; SATA SSD ~100–200 µs; a spinning disk 5–10 ms per seek plus rotation. A single line of code, a 10,000× range.
Readahead, and why sequential reads stay fast
If every 4 KiB read of a 1 GB file missed the cache, a sequential scan would cost 262,144 device round trips. It does not, because the kernel watches the access pattern per open file: two or three consecutive reads trigger readahead, which submits a request for the next window (default 128 KiB, read_ahead_kb per device) before you ask for it, doubling the window while the pattern holds. Your fourth read() finds page 3 already in flight or already present. Sequential scans therefore run at device bandwidth (gigabytes per second on NVMe) instead of device latency, and the per-call cost collapses back toward the hit cost.
Random access defeats this — a B-tree probe hits pages the kernel could not predict — which is why databases either issue their own prefetches (posix_fadvise(POSIX_FADV_WILLNEED), io_uring batches) or simply keep the working set in their own buffer. posix_fadvise(POSIX_FADV_SEQUENTIAL) and RANDOM are hints that adjust the readahead window; DONTNEED drops pages after a one-pass scan so a backup does not evict the database’s hot set from the cache.
mmap is the other way to read a file (Memory Mapping): map the pages into your address space and let page faults pull them in. The cache is the same page cache; the difference is that a hit costs a TLB miss and no syscall, and a miss costs a page fault instead of a sleep inside read. Both paths end at the same frames.
- Readahead turns latency-bound sequential reads into bandwidth-bound ones. It is per open file description and keyed on the offset pattern.
- A cold
grep -rover a source tree is thousands of misses; the second run is all hits. Benchmarks that forgetecho 3 > /proc/sys/vm/drop_cachesmeasure RAM. vmtouchandfincoreshow which pages of a file are resident right now.
O_DIRECT and who owns the cache
Opening with O_DIRECT asks the kernel to skip the page cache: the device DMAs straight into (or out of) your buffer. The buffer, offset and length must be aligned to the device’s logical block size (512 bytes or 4 KiB), the copy disappears, and so does readahead, write coalescing and the 30-second write-behind window — every read is a device round trip and every write goes to the device (though not necessarily through the drive’s own cache; fsync is still needed for durability).
Who wants this? Anything that already has a cache of its own. A database engine holding its Pages: The Unit of Everything in a buffer pool would otherwise cache each page twice — once in its pool, once in the kernel’s — and lose control of which pages stay resident. MySQL’s InnoDB defaults to O_DIRECT on Linux for its data files; PostgreSQL historically does not and leans on the page cache, which is why its shared_buffers guidance is "25% of RAM, leave the rest to the kernel". Neither choice is wrong; they are two answers to "who should decide what stays in memory".
The rest of the world should not use O_DIRECT. The page cache is one of the best-tuned pieces of software on the machine; bypassing it to "avoid a copy" trades a 1 µs memcpy for a 100 µs device read on every access that would have hit.
1int fd = open("data.bin", O_RDONLY | O_DIRECT);2void* buf = nullptr;3posix_memalign(&buf, 4096, 4096); // buffer aligned to the logical block size4ssize_t n = pread(fd, buf, 4096, 8192); // offset and length aligned too5// n == 4096: DMA'd straight from the device into buf, page cache untouched6// n == -1, errno == EINVAL: misalignedKey points
read()= libc wrapper → trap →sys_read→ VFS → file system → page cache. The layers exist so one interface serves every file system and every device.- Page-cache hit: bounds-checked memcpy, ~1–2 µs total. Miss: allocate page, map block via extents, submit to the block layer, DMA from the device, interrupt, wake — ~50–100 µs on NVMe, milliseconds on a disk.
- While the device works the thread sleeps in
Dstate; the CPU is free for something else. Blocking I/O wastes a thread, not a core. - Readahead makes sequential reads bandwidth-bound; random reads pay the miss cost every time unless the application prefetches.
O_DIRECTbypasses the page cache for systems that own their own cache (databases); everyone else should let the kernel cache.- Benchmarks that do not drop the cache measure memory bandwidth, not storage.
Why does this exist?
Mechanisms are answers to constraints. Open each question before reading the answer.
▸Why is there a VFS layer between the syscall and the file system?
So that read on ext4, XFS, NFS, a FUSE mount, /proc and a pipe all look identical to the application and share the page cache, the descriptor logic and the permission checks. Each file system implements a table of operations; the VFS calls through it.
▸Why sleep the thread instead of spinning?
A device read takes 50 µs to 10 ms — tens of thousands to millions of CPU cycles. Spinning would burn a core for nothing; sleeping lets the scheduler run another thread and costs only a context switch (~1–5 µs) when the interrupt arrives.
▸Why does the device write into memory itself (DMA)?
Copying 4 KiB through the CPU one word at a time would make the CPU the bottleneck at NVMe speeds. The controller writes into the page frame directly and interrupts once when done; the CPU touches the data only for the final copy to user space — or not at all with mmap or O_DIRECT.
File read trace
- read(fd, buf, 4096) · offset 0user mode↓
- syscall entry: mode switch, validate fd and buffer↓
- VFS: fd → file → inode; offset → page index↓
- file system: page cache lookupMISS↓
- map offset → block↓
- block layer: build bio, submit to the NVMe queue↓
- SSD reads; DMA writes straight into page-cache pagesthread sleeps, others run↓
- completion interrupt → wake the sleeping thread↓
- copy_to_user: 4 kB page cache → bufthe second copy↓
- return 4096 · back in user mode
How it fails
What the failure looks like from inside real software.
- A service that ran fast for months becomes I/O-bound after a memory upgrade elsewhere on the box — a neighbour’s working set evicted its page cache and every read now misses.
- p99 latency of a request is 10 ms while p50 is 200 µs: a small fraction of reads miss the cache and hit a spinning disk or an overloaded SSD.
- Threads pile up in
Dstate and load average climbs into the hundreds while CPU stays idle; the device queue is saturated (iostat -xshows%utilnear 100 and highawait). - A benchmark reports 5 GB/s "disk" reads: the file was in the page cache. Dropping caches or using
O_DIRECTreveals the device number. O_DIRECTread returnsEINVAL: buffer, offset or length not aligned to the logical block size.- Two threads share one descriptor and use
readinstead ofpread; the shared offset makes them consume interleaved chunks of the file.
Follow it through every layer
This lesson is one node of a longer journey. Zoom out, then zoom back in.