Deadlocks: conditions, detection, prevention

A deadlock is a set of tasks each waiting for a resource held by another member of the set, forever. It requires all four conditions: mutual exclusion (the resource cannot be shared), hold and wait (a task holds one resource while waiting for another), no preemption (the resource cannot be taken away), and circular wait (a cycle in the waits-for graph). Break any one and it cannot occur (Deadlocks).

The practical one to break is circular wait: impose a global lock order and always acquire in that order — then no cycle can form. Alternatives: acquire all locks at once (breaks hold-and-wait, coarse), use try_lock and release everything on failure (a form of preemption, with backoff and the risk of livelock), or hold one lock only. Timeouts do not prevent deadlocks; they convert them into errors you must handle, which is sometimes exactly right for a request path (Mutexes).

Detecting it in a running service is about state. A deadlocked process is alive, at ~0% CPU, and its threads are in interruptible sleep (S on Linux) waiting on a futex; a thread dump shows thread 1 waiting for lock B held by thread 2, which waits for lock A held by thread 1 — a cycle you can literally read. Contrast: a process blocked on I/O has threads in D state with a kernel stack in a filesystem or network function; a slow dependency shows threads waiting in socket reads with connections open; a CPU-bound hang shows 100% CPU. The kernel does not detect application deadlocks — user-space mutexes are just memory to it — though databases run a waits-for cycle check on their lock tables (Process States).

The consequence in a server is usually pool starvation: two threads deadlock on a cache refresh, every other worker eventually needs that lock and queues behind it, the accept backlog fills, and the health check — if it goes through the same pool — times out. The process does not crash; the orchestrator restarts it only if the health check is wired to notice.