8 lessons

CPU, Memory, Disk & Network

The four resources a process competes for, the signal that identifies each as the constraint, and the difference between a memory leak and a cache nobody bounded.

Symptom→Signal→Measurement→Hypothesis→Evidence→Root Cause→Change→Validation→Regression Check

Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.

What "CPU Is At 60%" Actually Means

A CPU number without a denominator is not a measurement. Sixty percent of how many cores, against which cgroup quota, counting which of user, system, iowait and steal — and is anything actually waiting for a core?

Symptom · A CPU chart that everyone points at during incidents and nobody can act on: it is high during good periods and low during bad ones, and two engineers read the same line differently.
CPU Saturation: When Cores Become the Queue

Throughput stops rising, latency bends upward, and the run queue grows. Confirming CPU as the constraint takes three readings; the causes range from an O(n²) loop to logging in the hot path to a lock everything spins on.

Symptom · Latency climbs steeply once traffic passes some level, and adding load stops adding throughput. Below the knee everything looks healthy; above it, p99 degrades far faster than p50.
Algorithmic Cost in a Request Handler

An O(n²) loop over a collection that grew is a CPU bottleneck that scales with data, not traffic — which is why it passes load tests and fails in production. And the counterpoint: Big-O does not price cache locality, branch prediction or constants.

Symptom · Latency for one endpoint is fine for most users and terrible for a few, and the slow ones are consistently the accounts with the most data. Traffic-based load tests never reproduce it.
Reading Memory: RSS, Heap, Working Set and the Number on Your Dashboard

The memory number everyone watches is usually the wrong one. Resident, virtual, heap, cache and cgroup working set answer different questions, and allocation rate — the one nobody charts — often matters more than any of them.

Symptom · A memory chart sitting at 92% that has been there for months without incident, next to a service that gets OOM-killed while "memory usage" reads 60%.
Memory Leaks: Growth That Does Not Come Back
▶ lab

Stable workload, rising memory, and a sawtooth of OOM restarts. Confirming a leak takes a trend under steady load; finding it takes two heap snapshots and a diff of what is still reachable.

Symptom · Memory climbs steadily over hours or days under unchanged traffic, latency degrades as the runtime collects more often, and eventually the process is killed and restarts with a clean slate — repeatedly, on a schedule.
Leak or Unbounded Cache? The Question That Picks the Fix

Both grow, both end in an OOM kill, and they need opposite fixes. Three questions separate them: does the growth correspond to data you would use again, is there an eviction policy, and does usage stabilize?

Symptom · Memory climbs under load and does not come back down. The team splits into "we have a leak" and "that is just the cache warming up", and both camps can point at the same chart.
Disk and Storage: Latency, Throughput, IOPS and the fsync Tax

Three numbers that people use interchangeably and should not: latency per operation, bytes per second, and operations per second. Plus the one that dominates write-heavy systems and appears on no dashboard by default — fsync.

Symptom · Requests are slow, CPU is largely idle with elevated iowait, and the database or the service that writes files is the common factor across every slow trace.
Network Signals: Is It the Network, or the Service on the Other End?

Connection setup can cost more than the request it carries. RTT, bandwidth, retransmits, handshake counts and pool waits each answer a different question — and the first one to answer is whether the network is involved at all.

Symptom · Calls to a downstream service are slower than that service says it is. Its own latency chart shows 12ms; your client-side chart shows 180ms, and both teams believe their own numbers.