Saturation: The Reading Utilization Cannot Give You
Utilization says how busy a resource was. Saturation says how much work could not be served immediately. A CPU at 55% with twelve threads waiting for eight cores is not half idle — it is oversubscribed, and only one of those two numbers says so.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Two different questions about the same resource
Utilization answers "what fraction of the time was this resource busy?". Saturation answers "how much work was waiting because it could not be served immediately?". These come apart constantly, and the gap between them is where most confusing performance incidents live.
The clearest example is CPU. A machine with eight cores showing 55% utilization sounds half idle. But utilization is averaged over an interval and across cores: if work arrives in bursts, the run queue can be twelve threads deep during each burst while the average across the whole interval reads 55%. Those twelve threads are waiting for a core. The system is oversubscribed at the moment it matters, and the utilization chart cannot show it — only run-queue length can (CPU Saturation: When Cores Become the Queue).
This is why the USE method insists on all three readings per resource — utilization, saturation and errors — rather than the utilization number everyone has on a dashboard (USE: Utilization, Saturation, Errors). Saturation is the one that identifies a constraint; utilization mostly tells you about cost efficiency.
| Resource | Utilization reading | Saturation reading | What saturation means here |
|---|---|---|---|
| CPU | Percent busy | Run-queue length vs core count; scheduler wait time | Threads are runnable but have no core (CPU Saturation: When Cores Become the Queue) |
| Memory | Percent used | Reclaim/scan rate, swap activity, allocation stalls | Allocation is now paying for reclamation (Reading Memory: RSS, Heap, Working Set and the Number on Your Dashboard) |
| Disk | Percent of time with I/O in flight | Average queue depth, await time | Requests are queued at the device (Disk and Storage: Latency, Throughput, IOPS and the fsync Tax) |
| Network | Bits per second vs link rate | Send-queue depth, drops, retransmits | Buffers full; loss triggers retransmission (Packet Loss Buys You a Timeout, Not a Retransmit) |
| Connection pool | In-use connections vs size | Waiter count, acquisition wait time | Requests are blocked before doing any work (Connection Pool Saturation: Waiting in Front of an Idle Database) |
| Worker pool | Busy workers vs total | Pending task count, enqueue-to-start delay | Work is accepted but not started (Twenty Workers, All Busy, Five Hundred Waiting) |
Reading a dashboard that looks fine
The panel below is the incident described in the symptom. Every utilization row reads normally. Every saturation row screams. This is the pattern to recognise: when utilization is unremarkable everywhere and the system is clearly unwell, you are looking at the wrong column of the USE table.
Notice that two different resources show saturation here — the run queue and the pool waiters — which is common and important. A single incident often has one primary constraint and secondary saturation caused by it: requests holding pool connections longer because their threads are waiting for CPU, for instance. The discipline is to establish which came first, usually by finding the resource whose saturation started rising earliest on the timeline (Reading a Timeline: Observation Order Is Not Causal Order).
The practical takeaway for dashboard design is to put a saturation signal next to every utilization signal (Dashboards Built Around Questions). A CPU panel without run-queue length, or a pool panel without waiter count, is a panel that will look healthy during the exact incident it exists to diagnose.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| CPU utilization | 55% | Averaged over 60 s and across 8 cores. Looks like headroom. | normal |
| run queue length | 12 (8 cores) | Twelve runnable threads, eight cores. Work is waiting for a core. | smoking gun |
| memory used | 60% | Comfortable. | normal |
| page reclaim rate | ~0/s | No reclamation pressure — memory is genuinely not the constraint. | normal |
| disk utilization | 40% | Device busy less than half the time. | normal |
| disk avg queue depth | 0.8 | Under one request queued on average — not the constraint. | normal |
| db pool in use | 46 / 50 | High but not exhausted; utilization alone would be ambiguous. | suspect |
| db pool waiters | 128 | 128 requests blocked waiting to acquire before doing any work. | smoking gun |
Saturation is a leading indicator; latency is a lagging one
Saturation rises before user-visible latency does. A queue must grow before waiting becomes long enough for anyone to notice, and that lead time — often minutes — is the difference between an alert that lets you act and an alert that tells you what users already know.
That makes saturation the better alerting signal for capacity problems, with one caution: saturation is normal in short bursts. A run queue that spikes to twelve for two seconds every minute is a system absorbing bursts, which is what queues are for. Alert on sustained saturation — depth above a threshold for a duration — rather than on any occurrence, or you will produce exactly the noise that makes people ignore alerts (Alert Fatigue: The Page Nobody Reads).
The complementary practice is to keep a symptom-based alert on the user-visible SLI as well (SLIs: Measuring What the User Actually Feels). Saturation alerts tell you a cause is developing; SLI alerts tell you users are affected. Having only the first produces pages nobody can justify; having only the second means you always arrive after the damage.
- Alert on sustained saturation, not instantaneous — depth above threshold for a duration, so ordinary bursts stay quiet.
- Pair every utilization panel with its saturation signal, or the dashboard will look healthy during the incident it exists for.
- Establish which resource saturated first when several show pressure; the later ones are usually consequences.
- Do not alert on utilization alone for capacity — it neither leads latency reliably nor identifies a constraint.
- Keep a user-facing SLI alert alongside, because saturation without user impact is not always worth waking someone (Alerts Worth Waking Someone For).
Key points
- Utilization is "how busy"; saturation is "how much could not be served immediately" — only the second identifies a constraint.
- Averaging hides oversubscription: 55% CPU across eight cores can coexist with a run queue of twelve during bursts.
- Every resource has its own saturation reading: run queue, waiter count, queue depth, reclaim rate, retransmits.
- When several resources show saturation, the one that rose first is usually the cause and the rest are consequences.
- Saturation leads latency by minutes, which makes it the better capacity alert — but only when alerting on sustained, not instantaneous, pressure.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Load → host: request rate rises; work arrives in bursts rather than smoothly.
- 2Bursts → run queue: during each burst more threads are runnable than there are cores, so threads wait for scheduling while 60-second average CPU stays near 55%.
- 3Scheduling wait → hold time: requests hold their database connections longer because their threads are descheduled mid-request.
- 4Hold time → pool: with connections held longer, the pool cannot keep up with acquisition demand and 128 requests queue as waiters.
- 5Root cause → team: CPU oversubscription during bursts is the primary constraint; pool saturation is its downstream consequence, and only the saturation readings show either.
- • "CPU is at 55%, so CPU is not the problem" — average utilization across cores and over a window cannot show burst oversubscription. Read the run queue.
- • "The pool is the bottleneck because waiters are high" — waiters are high, but the cause may be upstream. Check which resource saturated first.
- • "No resource is at 100%, so this must be application code" — saturation appears well before utilization reaches its ceiling, and often never reaches it at all.
- • "Saturation spiked, page someone" — brief saturation is normal burst absorption. Sustained saturation is the signal.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • For CPU: run-queue length compared against core count, plus scheduler wait time, rather than percent busy alone.
- • For pools: waiter count and acquisition wait time, not just in-use versus configured size.
- • For disk: average queue depth and await time alongside percent utilization.
- • For memory: reclaim and scan rates, allocation stall time, swap activity — used percentage says little on its own.
- • For queues: depth as a trend and enqueue-to-start delay, since a stable depth and a growing depth mean opposite things ([[queue-age]]).
- • Add the saturation signal for every resource on the dashboard so the constraint is identifiable without a live investigation.
- • Address the resource that saturated first; fixing a downstream symptom moves the queue rather than removing it ([[bottleneck-migration]]).
- • Reduce burstiness where possible — smoothing arrivals lowers peak saturation without adding any capacity.
- • Add capacity at the constrained resource specifically, sized to keep utilization below the measured knee ([[queueing]], [[headroom]]).
- • Confirm the saturation reading returned to baseline, not merely that latency improved — latency can improve for unrelated reasons.
- • Verify no new resource became saturated after the change, which is the normal outcome of adding capacity in one place.
- • Re-check during a burst rather than in steady state, since the original problem was only visible under burst conditions.
- • Saturation metrics are more numerous and more resource-specific than a single utilization number; dashboards get busier and need curation.
- • Alerting on saturation risks noise from normal bursts unless duration conditions are tuned, and tuning them takes real incident data.
- • Keeping utilization low enough to avoid saturation means running with idle capacity, which is a continuous cost ([[capacity-vs-efficiency]]).
- • Alert on sustained saturation per resource with a duration condition, and keep the user-facing SLI alert separately (Burn-Rate Alerts: How Fast Is the Budget Going?).
- • Chart utilization and saturation side by side permanently, so the pattern is recognisable to whoever is on call next time.
- • Record the measured saturation onset point during load testing so the alert threshold is derived rather than guessed.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe signal panel is constructed to show the utilization/saturation divergence pattern. The specific values are invented and are not measurements of any host.
- ENVIRONMENT-SPECIFICWhich saturation counters exist and what they are called depends entirely on the operating system, container runtime, database and client library in use. The concept is portable; the metric names are not.