Disk and Storage: Latency, Throughput, IOPS and the fsync Tax
Three numbers that people use interchangeably and should not: latency per operation, bytes per second, and operations per second. Plus the one that dominates write-heavy systems and appears on no dashboard by default — fsync.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Three limits that all present as "the disk is slow"
A storage device has at least three separate ceilings and hitting any one produces the same complaint. Latency is how long one operation takes — on NVMe often tens of microseconds, on networked block storage typically hundreds of microseconds to milliseconds, ENVIRONMENT-SPECIFIC and worth measuring on your actual volume rather than assuming. Throughput is bytes per second, and it is what large sequential reads exhaust. IOPS is operations per second, and it is what many small random reads exhaust — usually long before bandwidth is anywhere near the limit.
The classic confusion: a volume rated at 500 MB/s and 3,000 IOPS, serving a workload of 4 KB random reads. At 3,000 IOPS of 4 KB each, that is 12 MB/s — 2.4% of the bandwidth rating — and the device is completely saturated. A throughput chart shows a flat line near zero while every read queues. Anyone reasoning from the MB/s number concludes storage is idle.
Cloud volumes add a fourth ceiling that behaves unlike the others: a credit or burst balance. The volume performs well during testing, exhausts its burst budget after some hours of sustained load, and drops to a much lower baseline. The symptom is a step change in latency with no corresponding change in your workload, and it is invisible unless you chart the balance itself.
| Limit | Workload that exhausts it | Tell | What helps |
|---|---|---|---|
| Latency per op | Anything synchronous and serial — a read the request waits on | Service time high, queue depth low | Faster device class, caching, fewer round trips |
| Throughput (MB/s) | Large sequential reads/writes: scans, backups, log shipping | MB/s pinned at the rating; queue depth high | Compression, fewer bytes, a bigger volume, parallel volumes |
| IOPS | Many small random ops — the usual database pattern | IOPS pinned, MB/s trivially low, queue depth high | Better indexes (fewer random reads), larger pages, more memory for cache |
| Durability flushes (fsync) | Commit-heavy write paths, WAL, per-message acks | Write latency far above device latency; fsync count tracks commits | Group commit, batching, relaxed durability *if the loss is acceptable* |
| Burst credits | Sustained load after a quiet period | Step change in latency with no workload change; balance chart falling | Provisioned IOPS, larger volume, or accept the baseline |
Queue depth is the saturation signal
For storage, as for CPU, utilization is the weaker signal and queueing is the stronger one. A device utilization figure — the fraction of time at least one request was in flight — saturates at 100% on any modern parallel device long before the device is actually out of capacity, because NVMe and networked storage serve many requests concurrently. It tells you the device was busy; it does not tell you that anything waited.
Average queue depth answers the useful question. A queue depth of 1 with a 200µs service time is a device doing exactly what it should. A queue depth of 40 with a 12ms average wait means requests are stacking up, and the wait is the part your users feel. Splitting await into service time and queue time — as iostat does — separates "this device is slow" from "this device is oversubscribed", and those have different fixes.
This distinction is exactly where the USE: Utilization, Saturation, Errors earns its keep on storage: utilization near 100% is unremarkable; queue depth climbing with rising await is the finding. It also explains a frequent misread — a database host showing "disk 100% utilized" that is serving its workload comfortably.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| Read throughput | 11.8 MB/s (rating 500 MB/s) | Under 3% of bandwidth. Reasoning from this number alone says storage is idle. It is not. | normal |
| IOPS | 2,980 (provisioned 3,000) | Pinned at the provisioned ceiling. This is the actual limit being hit. | smoking gun |
| Average request size | 4 KB | Small random reads — the classic index-lookup pattern. IOPS-bound by construction. | suspect |
| Average queue depth | 38 | Requests are stacking up; most of the latency is wait, not service. | smoking gun |
| await / service time split | 13.1 ms / 0.4 ms | Thirty times more waiting than serving — oversubscribed, not slow. | smoking gun |
| Device utilization | 100% | Expected on a parallel device under any sustained load. Not evidence on its own. | normal |
| CPU iowait | 31% | CPU idle waiting for these reads — which is how this arrives labelled "high CPU". | suspect |
fsync: the cost that is not on the chart
Ordinary writes are cheap because they are lies — the kernel accepts the data into the page cache and returns immediately, planning to write it later. Durability is the moment you stop lying: fsync forces the data to stable media and does not return until the device confirms. That confirmation is orders of magnitude more expensive than the buffered write it follows, and it is synchronous.
This is why commit-heavy workloads behave nothing like their throughput numbers suggest. A database committing every transaction individually pays one durability flush per commit, so commit rate is bounded by flush latency, not by CPU or bandwidth. The standard mitigation is group commit — batch many transactions into one flush — which trades a small amount of latency for a large amount of throughput, and is precisely the batching trade-off in Every Optimization Buys Something and Sells Something. The mechanism lives in Write-Ahead Logging and Follow a Write Through the Engine.
The reason this is worth a section of its own: fsync latency appears on almost no default dashboard, so a write path bounded by durability flushes presents as an unexplained gap. The trace shows 14ms inside a database span; the database shows low CPU and modest I/O; nobody is looking at flush latency. Any storage investigation on a write-heavy system should chart it explicitly, and treat any suggestion to relax durability as a data-loss decision with a business owner, not a tuning knob.
span: db.commit 14.2 ms ├─ parse + plan 0.3 ms ├─ buffer pool write (page cache) 0.2 ms ├─ WAL append (buffered) 0.1 ms └─ WAL fsync 13.4 ms ← 94% CPU during this span: ~2% Device throughput: under 1% of rating fsync latency: charted nowhere by default Group commit at 8 transactions/flush would amortize this to ~1.7 ms per transaction — at the cost of up to one extra flush interval of latency per commit.
Key points
- Latency, throughput and IOPS are three separate ceilings; a device can be saturated on IOPS at 3% of its bandwidth rating.
- Queue depth plus the await/service-time split is the saturation signal; device utilization saturates at 100% while healthy.
- iowait means CPU idle waiting for I/O — which is why storage problems routinely arrive labelled "high CPU".
- fsync is the durability tax and dominates commit-heavy write paths, yet appears on almost no default dashboard.
- Cloud volumes have burst budgets: a step change in latency with no workload change means the balance ran out.
Progressive depth
Overview
Storage has three separate limits — how long one operation takes, how many bytes per second, and how many operations per second — plus a fourth on write paths: the cost of forcing data to durable media. Hitting any of them looks the same from the application: slow requests and an idle-looking CPU.
Practical
Read IOPS, throughput and average request size together to identify which ceiling applies, then queue depth and the await/service split to tell "oversubscribed" from "slow". On write-heavy paths chart fsync latency explicitly. iowait is the signal that routes you here in the first place, and it is CPU-idle time, not CPU work.
Advanced
Two effects surprise people. First, queue depth cuts both ways: a device with parallelism needs a deep enough queue to reach its rated throughput, so *too little* concurrency underuses it while too much adds pure wait — there is an optimum, and it is device-specific. Second, cloud burst credits make short tests systematically optimistic, so a load test that passes in ten minutes can fail after two hours at the same rate (Coordinated Omission: When the Load Generator Lies is a different way the same optimism creeps in).
Internals
A read that misses the page cache becomes a block request through the I/O scheduler to the device, which may itself reorder and batch. fsync additionally forces the device to commit its own volatile write cache, which is why its latency is dominated by device behavior rather than by data size. Databases build directly on this boundary: a write-ahead log turns random page writes into a sequential append plus one flush, trading read-time indirection for write-time locality, and group commit amortizes the flush across transactions. See Write-Ahead Logging, Follow a Write Through the Engine and The Buffer Pool for how the engine arranges itself around exactly this cost.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Query → index lookup: a lookup that misses the buffer pool becomes a small random read against the device (The Buffer Pool).
- 2Random reads → device queue: at 4 KB each, the operation count reaches the provisioned IOPS ceiling while bandwidth stays trivially low.
- 3Queue depth → await: each new read waits behind the queue, so await becomes mostly wait time rather than service time.
- 4Await → request latency and iowait: the requesting thread blocks; the CPU registers idle-waiting-on-I/O, and the service looks like it has a CPU problem.
- 5Commit path → fsync: on the write side, each commit additionally waits for a durability flush, so commit rate is bounded by flush latency rather than by any throughput number.
- • "Disk throughput is 2% of the rating, storage is idle" — small random I/O exhausts IOPS long before bandwidth.
- • "Disk utilization is 100%, the disk is the bottleneck" — utilization saturates at 100% on parallel devices under normal load; look at queue depth and await.
- • "iowait is high, we need more CPU" — iowait is idle CPU waiting for I/O; more cores will wait faster.
- • "The database is slow" — often the device under it is oversubscribed, or the commit path is fsync-bound; the database is the messenger.
- • "It was fast in testing" — burst credits make short tests unrepresentative of sustained load (Load Test Shapes: The Shape Is the Hypothesis).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Per-device IOPS, throughput and average request size together — the size tells you which ceiling applies.
- • Average queue depth, and await split into service time versus queue time.
- • CPU iowait as a share of CPU time, to confirm the CPU is waiting rather than working.
- • fsync latency and fsync rate on any write-heavy path, added explicitly because it is rarely default.
- • Cloud burst-credit or throughput-balance metrics, if the volume has them.
- • Reduce operations, not just bytes: better indexes and higher cache hit rates remove random reads entirely ([[scan-vs-index-performance]], [[why-indexes]]).
- • Give the working set more memory so reads are served from cache rather than the device ([[buffer-pool]]).
- • For fsync-bound commits, enable or tune group commit so many transactions share one flush.
- • Match the volume to the workload — provisioned IOPS for small-random, throughput-optimized for sequential scans.
- • Batch and coalesce small writes; many tiny writes are the most expensive possible pattern for both IOPS and fsync.
- • Relax durability only as an explicit, owned business decision about acceptable data loss — never as a quiet tuning change.
- • Confirm IOPS is now below the ceiling at the same workload, and that queue depth and await dropped with it.
- • Confirm request p99 improved — device metrics improving without user-visible improvement means storage was not the binding constraint.
- • For fsync fixes, compare commit latency and commit throughput before and after at equal transaction rates.
- • Re-run under sustained load long enough to exhaust any burst credits, so the result reflects baseline performance.
- • Group commit raises throughput and adds up to one flush interval of latency to every commit.
- • Provisioned IOPS and faster volume classes cost significantly more per gigabyte, often for capacity you do not need.
- • More memory for caching reduces device load but raises instance cost and moves you closer to memory limits.
- • Relaxed durability trades a real, quantifiable risk of data loss for write throughput — a business decision, not an optimization.
- • Alert on queue depth and await rather than on utilization or throughput percentage.
- • Track burst-credit balance with an alert well before exhaustion, since exhaustion is a cliff.
- • Keep fsync latency on the write-path dashboard permanently once it has caused one incident.
- • Include a sustained soak in the load-test suite so burst-masked regressions cannot pass (Load Testing: What Question Is This Test Answering?).
Accuracy
Performance numbers are conditional. These are the conditions.
- ENVIRONMENT-SPECIFICDevice latency, IOPS ceilings and burst behavior differ enormously between local NVMe, networked block storage and spinning disks. Measure your volume; do not port numbers between environments.
- ILLUSTRATIVEThe signal panel, the 4 KB/3,000 IOPS example and the fsync span breakdown are teaching values chosen to show the shape of each ceiling.
- DATABASE-SPECIFICGroup commit, WAL flush behavior and durability settings differ by engine; PostgreSQL, MySQL/InnoDB and LSM-based engines make different trade-offs at the flush boundary.
Misconceptions
Apply it
Where the depth lives
The fsync tax is the reason WAL exists in the shape it does. Seeing the engine's design as a response to flush cost turns a storage metric into an architectural explanation.