The OS Debugging Playbook: Four Symptoms, Fourteen Causes

Every playbook below has the same shape: a symptom, the candidate causes, and for each pair of causes the observation that separates them. The tools are the standard Linux ones — top, ps, vmstat, strace, perf, lsof, ss, /proc — and every transcript is labelled linux; macOS and Windows have counterparts (Activity Monitor/sample, Process Explorer/procdump) with the same logic and different names. The interactive presents the same playbooks as a clickable symptom tree: pick the symptom, answer each distinguishing observation, and arrive at the cause.

Two habits matter more than any tool. First, look at the process’s state before its code: ps -o pid,stat,pcpu,rss,vsz,nlwp,wchan -p <pid> gives state, CPU, resident and virtual memory, thread count and what the kernel is waiting on in one line. Second, watch the shape over time: a value climbing linearly forever, a value that plateaus, and a value that spikes and recovers are three different bugs even when the current number is the same.

Four causes: an infinite loop (or an accidental O(n²) on a big n), excessive legitimate work (traffic doubled, a batch job), contention (threads fighting for a lock, cache line or the GIL), and a spin lock (threads burning CPU while waiting). The first cut is user vs system time in top: a loop in your code is %us; a spin lock built on sched_yield or a futex storm is %sy; a runaway process making millions of syscalls (a read loop with a 1-byte buffer, a busy-poll on a non-blocking descriptor) is also %sy, and strace -c -p <pid> for ten seconds shows *which* syscall.

The second cut is per-thread CPU: top -H -p <pid> (or ps -L). One thread at 100% and the rest idle is a loop; all threads at a modest equal share is legitimate work; all threads at 100% with throughput flat is contention or spinning. The third cut is what the CPU is executing: perf top -p <pid> shows the hottest functions live (pthread_mutex_lock / __lll_lock_wait at the top is contention; atomic_load in a tight loop is a spin; your own function is a loop or real work). For an interpreted runtime, py-spy top, Node --cpu-prof, or the JVM’s async-profiler read the language stacks instead of the C ones. A flame graph (perf record -g + FlameGraph, or the profiler’s own output) turns the same data into one picture: a wide flat plateau is where the time goes.

The tell for the infinite loop is that the hot function never returns — the stack is identical every sample. The tell for excessive work is that the profile looks like the service’s normal profile, only larger, and request counts went up. The tell for contention is lock functions at the top and cores that add no throughput; high-cpu-spin-lock and high-cpu-infinite-loop are the two challenges in this domain, and event-loop-blocked is the single-threaded variant where one loop iteration never yields.

Four causes: a leak (objects allocated and never released), a large cache (by design, and possibly the wrong size), fragmentation (the allocator holds freed memory it cannot return), and an unbounded queue (a producer faster than its consumer, buffering in memory). The first cut is RSS vs virtual: VSZ is the address space reserved and is almost meaningless (a JVM or Go process reserves gigabytes it never touches); RSS is what is resident; a process with 40 GB VSZ and 1 GB RSS has no memory problem. Read VmRSS, RssAnon, RssFile and RssShmem from /proc/<pid>/status — anonymous RSS is *your* memory; file RSS is page cache that the kernel can drop.

The second cut is the growth curve. Plot RSS over hours: a leak is monotonic and roughly linear in traffic — it never comes down, including at 4 a.m. A cache grows to its configured limit and plateaus; if there is no limit, it looks like a leak and *is* one. An unbounded queue grows when the consumer falls behind and shrinks when it catches up — a sawtooth correlated with downstream latency (memory-leak-unbounded-queue). Fragmentation shows as RSS well above what the heap profiler says is live, with glibc malloc_stats() or MALLOC_ARENA_MAX making a difference — common in long-running multithreaded C/C++ and CPython services.

The third cut is a heap profiler, which answers *what* is growing: heaptrack or valgrind --tool=massif for native code, jemalloc’s prof mode, Node --heap-prof or a Chrome DevTools heap snapshot diff, Python tracemalloc snapshots compared across an hour, JVM jmap -histo. Take two snapshots an hour apart and diff; the type whose count grew by exactly the number of requests is the leak. If the profiler shows the live heap flat while RSS climbs, the allocator or a native library is the culprit, not your objects.

Three causes: a deadlock (threads waiting on each other), blocked I/O (waiting on a disk, a socket, a pipe with no writer), and a waiting dependency (a database, an upstream service, a DNS resolver that is not answering). The first cut is the state letter: D means the kernel is waiting on a device — disk, NFS, a driver — and no user-space lock is involved; S means interruptible sleep, which covers a lock (futex), a socket (recv), a pipe, a sleep or a condition variable. cat /proc/<pid>/wchan and /proc/<pid>/stack (root) name the kernel function it sleeps in.

The second cut is strace -p <pid> (or -f for all threads): a deadlocked thread shows a single futex(…, FUTEX_WAIT_PRIVATE, …) and nothing further, forever; a socket wait shows recvfrom(23, …) or epoll_wait returning nothing; a disk wait shows read(7, …) that never returns (and the process is in D). ss -tnp | grep <pid> then says who descriptor 23 is connected to — and if that peer is your database, playbook 3 has just become somebody else’s playbook. The third cut is a thread dump: gdb -p <pid> -batch -ex "thread apply all bt", jstack, py-spy dump, Node --inspect + kill -USR1. Draw the wait-for graph from the lock frames; a cycle is a deadlock (Deadlocks, Cycle Detection); a chain ending at a socket or a disk is a hang with an external cause.

The single-threaded case deserves its own line: an event loop blocked by one long synchronous call — a large JSON.parse, a sync file read, a regex with catastrophic backtracking — is a hang for every connection and 100% CPU on one thread (The Event Loop). process-hang-deadlock and process-hang-blocked-io are the two challenges; the dependency case is a networking problem, handled in Why Can’t I Connect?.

One symptom, two questions: what are the descriptors, and is the limit wrong or the usage wrong. ulimit -n shows the soft limit for a new shell; the running process’s own limit is in /proc/<pid>/limits, and it is frequently 1024 because a systemd unit or a container runtime did not raise it. ls /proc/<pid>/fd | wc -l is the count; lsof -p <pid> (or ls -l /proc/<pid>/fd) says what they are — regular files, sockets, pipes, eventfds, anon_inodes from epoll and timers. Group by type and by target: 50,000 sockets to one upstream, or 20,000 opens of the same log file, is the answer.

Sockets dominate in practice, and the state to look for is CLOSE_WAIT: the peer sent FIN, the kernel acknowledged it, and your code never called close() — typically a response handler that returns early on an error path, or an HTTP client that never consumes the body (The Connection Lifecycle: Close, Reset, TIME_WAIT, CLOSE_WAIT). ss -tan state close-wait | wc -l counts them; if the number climbs with traffic and never falls, that is the leak, and raising the limit only delays it (too-many-open-files, fd-leak-under-load). The second common shape is legitimate: a proxy or a database with 5,000 connections and a 1024 limit — here the fix really is LimitNOFILE=65536 in the unit file or --ulimit nofile= on the container.

Do not forget the system-wide limits: fs.file-max and /proc/sys/fs/file-nr for the host, fs.nr_open as the ceiling any process can be raised to, and ENFILE (system-wide) vs EMFILE (per-process) in the error text telling you which one you hit.

The interactive is this tree. Each leaf names the cause, the confirming observation and the challenge that rehearses it.

Two symptoms are missing from the tree on purpose. “The service is slow” is not an OS symptom until you have translated it into one of these four — or into a networking symptom (Why Can’t I Connect?, Where the Time Goes: The Request Timeline) — and “the process died” is an exit code: 137 is SIGKILL (usually the OOM killer, so go to high memory), 139 is SIGSEGV, 143 is SIGTERM from an orchestrator (Signals: Asynchronous Notifications From the Kernel). Translate first, then walk the tree.