What "CPU Is At 60%" Actually Means
A CPU number without a denominator is not a measurement. Sixty percent of how many cores, against which cgroup quota, counting which of user, system, iowait and steal — and is anything actually waiting for a core?
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Sixty percent of what?
The number on the chart is a ratio, and almost every argument about it is really an argument about the denominator. On an 8-core host, "60%" usually means 4.8 cores of work — but a single-threaded request handler pegging one core to 100% shows up as 12.5% aggregate, and the host looks bored while every request queues behind one saturated thread. Aggregate utilization is an average over cores, and averages hide exactly the thing you are hunting.
In containers the denominator moves again. A pod with a CPU limit of 500m on a 64-core node can be throttled to a standstill while the node-level chart reads 20%. The number that matters there is not utilization at all: it is container_cpu_cfs_throttled_seconds_total, which counts the time the kernel took the CPU away because the cgroup had spent its quota for that period. A service can be throttled hard at 45% "utilization" and nothing on a host dashboard will show it.
And on shared virtual hardware, some of the time simply is not yours. Steal time is CPU the hypervisor gave to another tenant while your vCPU was runnable. It presents as "my code got slower and I changed nothing", because that is exactly what happened. Steal above a couple of percent is a scheduling problem you cannot fix in your code, only by moving or resizing.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| CPU utilization (aggregate) | 60% | Says nothing on its own. Over 8 cores this is 4.8 cores of work; the distribution across cores is what matters. | normal |
| Busiest single core | 99% | One core pinned while seven idle: a single-threaded hot path or a lock everything serializes behind. | smoking gun |
| Run queue length (runnable tasks) | 14 on 8 cores | Six runnable tasks have no core to run on. This is the saturation signal; utilization is only the utilization signal. | smoking gun |
| cgroup CPU throttled | 2.1 s per 10 s window | The kernel removed the CPU for 21% of wall time. The container is at its limit regardless of what the host chart says. | smoking gun |
| Steal time | 0.3% | Negligible here. Above ~2–3% sustained, the hypervisor is the constraint and no code change will help. | normal |
| iowait | 18% | The CPU was idle *waiting for I/O*. Counted as "busy" by some collectors, which is how a disk problem gets misfiled as a CPU problem. | suspect |
The modes are different problems wearing one number
Splitting CPU time by mode turns one useless number into a routing decision. High user time means your code is computing — profile it (see Self Time, Total Time, and Where the CPU Went). High system time means the kernel is working on your behalf: syscall storms, tiny writes, excessive context switching, connection churn. That is a completely different fix, and a CPU profile of user-space code will show you nothing useful.
High iowait is the trap. It is not CPU work at all — it is the CPU sitting idle with at least one task blocked on I/O. A service with 40% iowait is telling you to go look at Disk and Storage: Latency, Throughput, IOPS and the fsync Tax or the database, not to buy faster cores. Some agents and dashboards fold iowait into "CPU busy", which is how a slow disk becomes a ticket titled "high CPU".
Getting this split is cheap and it is the highest-leverage thing you can do to a CPU dashboard. If your service chart shows one line called "CPU", the first improvement is not a better alert threshold — it is four lines instead of one.
| Mode dominates | What is actually happening | Where to look next |
|---|---|---|
| user | Your code is computing: serialization, compression, parsing, hashing, an expensive algorithm | Self Time, Total Time, and Where the CPU Went, Reading a Flame Graph, Algorithmic Cost in a Request Handler |
| system | Kernel work on your behalf: syscalls, context switches, page faults, connection setup, small writes | Context Switching, System Calls, connection reuse in Network Signals: Is It the Network, or the Service on the Other End? |
| iowait | CPU idle, tasks blocked on storage or network I/O — not a CPU problem at all | Disk and Storage: Latency, Throughput, IOPS and the fsync Tax, Which Signal Actually Means "The Database Is Slow" |
| steal | The hypervisor scheduled another tenant on your vCPU | Instance type, noisy-neighbour policy, dedicated hosts — not a code fix |
| idle but latency high | Nothing is CPU-bound; the time is spent waiting somewhere | Computing or Waiting?, Low CPU, High Latency: Lock Contention, Queueing: Why Systems Get Slow Before They Get Broken |
CPU (8 cores, container limit 4.0) user 12% ▏▏▏ system 41% ▏▏▏▏▏▏▏▏▏▏ iowait 3% ▏ steal 0% idle 44% runnable tasks: 3 throttled: 0.0 s / 10 s → system-dominated with an idle-ish box: syscall or context-switch overhead, not an expensive algorithm. A user-space CPU profile would look almost empty.
Utilization is not saturation
The single most useful correction to CPU intuition: utilization answers *how much of the resource was in use*, saturation answers *how much work could not be served immediately*. They diverge exactly when it matters. A batch job can run at 100% utilization for an hour with zero queueing and zero user impact. A latency-sensitive service can be at 70% utilization with a run queue of twelve and a p99 that has tripled. This is the U and the S of the USE: Utilization, Saturation, Errors, and collapsing them into one chart is why CPU dashboards start arguments.
The saturation signal for CPU is the count of runnable-but-not-running tasks — the run queue — or, on Linux, pressure-stall information, which reports the fraction of time tasks were stalled waiting for CPU. Either one answers the question a utilization chart cannot: *is anything waiting?* Once tasks queue for a core, every additional request pays for the queue ahead of it, and latency climbs far faster than utilization does. That knee is Queueing: Why Systems Get Slow Before They Get Broken, and it arrives well before 100%.
This is also why "scale when CPU > 80%" is a coin flip as an autoscaling rule. For a service where 80% utilization already means a run queue, it triggers too late; for a CPU-hungry batch worker it triggers constantly for no reason. Scale on the signal that tracks user pain — queueing, concurrency, or latency itself (see Autoscaling: Scaling on the Right Signal).
Key points
- A CPU percentage is meaningless without its denominator: core count, cgroup quota, and whether the collector counts iowait as busy.
- Aggregate utilization hides per-core imbalance — one pinned core on an 8-core box reads as 12.5%.
- Split by mode: user means profile the code, system means kernel overhead, iowait means go look at I/O, steal means the hypervisor.
- Utilization is not saturation. The run queue (or PSI) answers "is anything waiting for a core", which is what latency tracks.
- In containers,
cpu.statthrottling is the real limit signal; a node-level chart can read 20% while your pod is throttled hard.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Request arrives → thread pool: work is handed to a worker thread; if all workers are runnable but cores are scarce, the work is queued before any code runs.
- 2Runnable tasks → scheduler: more runnable threads than cores means each one waits a scheduling round before executing (see Context Switching).
- 3Queue wait → response latency: the request pays queue time plus service time, but only service time looks like "CPU work" on a profile.
- 4Utilization chart → operator: reads 65% and looks elsewhere, because the chart cannot show the six tasks that were waiting.
- 5Container quota → kernel: if the cgroup has spent its period budget, the kernel throttles all threads regardless of idle host cores, and latency spikes on a 20% host.
- • "CPU is only at 60%, so CPU is fine" — 60% aggregate with a run queue of fourteen is a saturated CPU; utilization cannot see queueing.
- • "CPU is at 100%, that is our problem" — for a batch worker that is the intended state; without a latency or throughput impact it is not a finding.
- • "High CPU means expensive code" — high *system* time usually means syscall or context-switch overhead, and a user-space profile will look empty.
- • "The host chart looks fine" — cgroup throttling is invisible at host level; the pod can be at its limit while the node idles.
- • "It got slower and nothing changed" — check steal time before rewriting anything; a noisy neighbour is not a code regression.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • CPU utilization split by mode (user / system / iowait / steal / idle), not a single "CPU" line.
- • Per-core utilization, or at minimum max-core alongside the average, to expose single-threaded pinning.
- • Run queue length (runnable tasks) or Linux PSI `cpu.some` — the saturation half of the [[use-method]].
- • For containers: `container_cpu_cfs_throttled_seconds_total` and the configured quota, as a ratio of wall time.
- • Cores available to the process (`nproc` under the cgroup, not the host) recorded alongside every CPU chart.
- • Fix the dashboard first: mode split, per-core or max-core, run queue, and throttling. Most "CPU mysteries" dissolve here without a code change.
- • If user time dominates, profile before optimizing and let the profile choose the target ([[cpu-profiling]], [[measure-before-optimizing]]).
- • If system time dominates, cut syscall and context-switch volume: batch small writes, reuse connections, right-size thread pools ([[concurrency-limits]]).
- • If iowait dominates, stop looking at CPU entirely and follow [[disk-io-performance]] or [[database-performance-signals]].
- • If throttling dominates, raise the quota or reduce per-request work — adding replicas does not help a pod that is throttled per-period.
- • If steal dominates, move the workload: different instance type, dedicated host, or a different availability zone.
- • Compare p95/p99 latency before and after against the same traffic band, not against a different hour — CPU effects are load-dependent.
- • Confirm the run queue (or PSI stall fraction) dropped, not just the utilization percentage.
- • For throttling fixes, confirm throttled-seconds per window went to ~0 and that latency improved; if throttling is gone but latency did not move, CPU was not the constraint.
- • Re-run the same load test at the same RPS and check where the latency knee now sits (see [[load-testing]]).
- • Per-core and per-mode metrics multiply series count — bounded, but it is real cardinality cost ([[cardinality]]).
- • PSI and run-queue metrics need node-level collection, which not every managed platform exposes; you may only get throttling as a proxy.
- • Raising a cgroup quota to remove throttling costs real money and reduces bin-packing density on the cluster.
- • Alert on saturation (run queue, PSI, throttling), not on utilization thresholds — utilization alerts produce noise and miss real events.
- • Keep core count and cgroup quota as labels or annotations on the CPU panel so nobody reads the ratio without its denominator.
- • Add a CI benchmark for the hot path found by profiling, so a future change that doubles its cost fails review (Regression or Tuesday? Telling a Real Change from Noise).
- • Record steal time in the same panel; an infrastructure regression should not look like an application regression.
Accuracy
Performance numbers are conditional. These are the conditions.
- ENVIRONMENT-SPECIFICEvery threshold here depends on core count, cgroup quota, hypervisor tenancy and collector conventions. "60%" on a 2-core burstable VM and on a 64-core dedicated host are unrelated numbers.
- ILLUSTRATIVEThe signal panel and the mode split are teaching examples with invented values, not measurements from a real service.
- RUNTIME-SPECIFICPSI,
cpu.statthrottling and steal time are Linux/cgroup v2 concepts. Names and availability differ on other kernels and on managed platforms that hide node metrics.
Misconceptions
Apply it
Where the depth lives
The run queue is not a monitoring artefact — it is the scheduler's actual data structure of runnable tasks. Seeing it there makes "utilization vs saturation" concrete rather than a slogan.