The question this answers
My async service is stalled but every OS thread is idle. Where do I look?
An async HTTP service running on four runtime threads with eleven thousand in-flight tasks, each of which awaits a database query, an upstream HTTP call, or a semaphore permit.
The runtime's ready queue and its pending-task registry, plus the resources tasks await on: a connection pool, an HTTP client's host pool, and a concurrency-limiting semaphore.
Every task that has been spawned is either running, on the ready queue, or parked on exactly one identifiable awaited resource — no task is pending with nothing that can ever wake it.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Why the thread view is structurally blind here
In a thread-per-request system the thread *is* the request, so a thread dump is a request dump. In an async runtime a task is a state machine the runtime resumes when something completes, and thousands of them share a handful of OS threads — A Task Is Not a Thread is the underlying distinction. When every task is awaiting something, every thread has nothing to resume, so every thread parks on the runtime's own idle wait. The thread dump is not wrong. It is answering a question about the wrong unit.
This produces the most misleading incident signature in async systems: CPU near zero, threads idle, load average low, and no requests completing. A capacity dashboard reads this as "we are over-provisioned". A thread dump reads it as "healthy". Only a view of the *tasks* shows eleven thousand of them, all parked, most of them on one thing.
The distinction from Reading a Thread Dump is not stylistic. There, the interesting fact was which thread owned the lock. Here, the interesting fact is the histogram of *what tasks are awaiting*, because the shape of that histogram names the exhausted resource directly.
--- THREAD VIEW (useless here) -------------------------------------
runtime-worker-0 IDLE parked in scheduler.waitForWork()
runtime-worker-1 IDLE parked in scheduler.waitForWork()
runtime-worker-2 IDLE parked in scheduler.waitForWork()
runtime-worker-3 IDLE parked in scheduler.waitForWork()
cpu 3% load 0.11 "the service is over-provisioned"
--- TASK VIEW (the actual system) ----------------------------------
tasks total 11,412 running 0 ready 0 pending 11,412
BY AWAITED RESOURCE count oldest
db.pool.acquire 9,840 41.2 s <-- the incident
http.client(payments).response 870 38.9 s
semaphore(export).permit 431 44.0 s
timer.sleep 268 0.4 s
channel.recv(jobs) 3 612.0 s <-- idle by design
SAMPLE TASKS
task#40711 state=PENDING age=41.4s
awaiting db.pool.acquire(pool="primary", size=20, in_use=20, waiters=9840)
spawned_at handler.getOrder:88
parent request#88213
task#40712 state=PENDING age=41.4s
awaiting db.pool.acquire(...) [identical to 9,839 others]
task#00019 state=PENDING age=612.0s
awaiting channel.recv(jobs)
spawned_at worker.consumeLoop:12 <-- a consumer waiting for work: healthyFour tasks, four different reasons for being pending
The value of a task dump is that "pending" decomposes. Task A pending on a database acquire, task B pending on an HTTP response, task C actually running, and task D pending on a semaphore permit are four different system conditions that a thread dump collapses into "four idle threads". Read the timeline below: only C is consuming anything, and the three pending tasks are each waiting on a resource with a different owner and a different fix.
Ages matter more than counts. A thousand tasks pending for 3ms is a busy service. Nine thousand pending for 41 seconds is an outage, and the oldest-age column finds it instantly. This is exactly the queue-age argument from Depth Is Not an Emergency; Age Is applied to task state instead of to a queue.
The resource histogram then selects the fix. Concentration on db.pool.acquire with in_use == size is pool exhaustion — Connection Pool Saturation: Waiting in Front of an Idle Database and Pool Saturation. Concentration on one HTTP host is a downstream stall, and the correct response is a timeout and a circuit breaker, not more tasks. Concentration on a semaphore is your own concurrency limit doing precisely what you asked — Bounding Concurrency — and the question becomes whether the limit is right.
What the runtime must record for this to exist
A task dump is not free the way a thread dump is. Stacks exist in the runtime whether or not you ask; a task's logical await chain generally does not, unless the runtime tracks it. That means task dumps require the runtime to keep a registry of live tasks, a spawn site per task, a parent link, and a description of the awaited resource — and keeping those costs memory per task and a small cost per await.
Which is why the practical advice is structural: name your tasks, spawn them through a wrapper that records the spawn site and parent, and give every await a resource label. Structured concurrency helps enormously here, because a task tree with owners is exactly the registry a dump needs — see Structured Concurrency and Cancellation Propagation.
The honest limitation: a task dump shows the *logical* await, not the stack of the code that will resume. In many runtimes a suspended task has no meaningful native stack at all, so "where is it" means "which await is it parked on", and the frames you would want are compiled into a state machine. That is why the spawn site matters so much — it is frequently the only source location available.
| Question | Thread dump | Task dump |
|---|---|---|
| How many units of work are in flight? | Number of threads — wrong by orders of magnitude | Number of live tasks — correct |
| What is blocking progress? | Only if a thread is genuinely blocked on a lock | The awaited-resource histogram names it directly |
| How long has this been stuck? | Not available from one dump | Task age, per task, directly |
| Who spawned this work? | The stack, if the frames survive | Spawn site and parent, if the runtime records them |
| Is a CPU-bound task stalling the loop? | Yes — a RUNNING thread with an application stack | Only if the runtime samples running tasks too |
| Cost to capture | Near zero; stacks already exist | Requires per-task registry and await labelling |
Key points
- In an async runtime the thread is not the unit of work, so a thread dump answers a question nobody asked.
- The signature is idle threads, near-zero CPU, and thousands of pending tasks — a shape that every capacity dashboard reads as "over-provisioned".
- The diagnostic artefact is a histogram of awaited resources with an oldest-age column; concentration plus age names the exhausted resource.
- Pending is not one state: awaiting a pool, awaiting a downstream, holding no permit, and idling on an empty channel are four different conditions.
- Task dumps require the runtime to record a task registry, spawn sites and await labels — unlike thread stacks, they do not exist for free.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • The runtime keeps a registry of live tasks with an id, a spawn site, a parent link and a creation timestamp.
- • Each suspension point records what the task is awaiting as a labelled resource — pool name, host, semaphore name, channel — rather than an opaque "pending".
- • On capture, walk the registry and emit one row per task, grouping identical awaits and reporting count plus oldest age per group.
- • Sample currently-running tasks separately, since a single CPU-bound task is the other way an async service stalls — see Blocking the Event Loop.
- • Diff two captures a minute apart: tasks present in both with the same await and a growing age are genuinely stuck.
- • Pool size 20, arrival 500/s, query latency rises from 5ms to 3s: the first 20 tasks hold connections, the next 9,840 park on acquire, and the threads go idle because nothing is ready to poll. No task is broken; the resource is simply gone.
- • A payment host stops responding with no timeout configured. Eight hundred and seventy tasks park on the response future forever; each holds its request buffer, and memory grows linearly with the outage. See Timeouts.
- • A single CPU-bound serialization task occupies the only runtime thread for 800ms. Every other task is ready but unpolled — the task dump shows them as ready rather than pending, which is the distinction that identifies this as loop starvation rather than resource exhaustion.
- • A task awaits a channel whose sole producer has already exited. Nothing will ever wake it; it sits pending forever with an ever-growing age and no owner — an orphan, invisible to any thread view. See Orphaned Tasks.
- • A task dump guarantees the set of live tasks and their awaited resources at one instant, if and only if the runtime tracked them.
- • It does not guarantee a native stack. In many runtimes a suspended task has no stack to show, and the spawn site is the only source location you get.
- • It does not prove a task is stuck — only that it was pending at capture. Age plus a second capture is what turns pending into stuck.
- • It shows tasks the runtime knows about. Work handed to a thread pool, a native library, or a detached callback may be entirely absent.
- • The task registry is itself shared mutable state touched on every spawn and completion; a naive global map becomes a contention point at high task rates.
- • Capturing while tasks are spawning and completing requires either a consistent snapshot (a pause) or an inconsistent one (rows that are already stale).
- • Await labelling adds a small allocation or string reference per suspension, on the hottest path an async runtime has.
- • The blind-spot failure: no task view exists, so a resource-exhaustion outage is diagnosed for hours as "the service is idle, it must be the load balancer".
- • Orphaned tasks pending forever on a channel nobody will ever send to, accumulating until memory pressure reports the problem instead.
- • Unbounded pending growth: with no limit on spawned tasks, an outage downstream converts directly into an out-of-memory kill. See Unbounded Concurrency.
- • Loop starvation mistaken for exhaustion: tasks are READY rather than PENDING, and the culprit is one CPU-bound task, not a downstream.
- • Detached tasks whose failures are never observed, so a task dies with an exception and the parent waits on a result that will never arrive.
- • Any stall in an async service, where the thread view is guaranteed to be uninformative and the task view is usually conclusive in one reading.
- • Finding orphaned and leaked tasks, which have no thread-level representation at all.
- • Attributing a memory-growth incident to in-flight work: eleven thousand pending tasks each holding a request buffer is a visible, explainable footprint.
- • Diagnosing CPU-bound stalls, where the answer is one running task and a profile is the better instrument.
- • Very high task-churn systems, where the registry and labelling cost is a real fraction of the runtime's work.
- • When the dump is treated as a stack trace: the spawn site is not where the task is, and reading it as one sends people to the wrong function.
- • Pending task count and its growth rate — a monotonic rise during steady arrivals means completions have stopped.
- • Oldest pending age per awaited resource; this single column usually names the incident.
- • Ready-but-unpolled count, which distinguishes loop starvation from resource exhaustion.
- • Tasks spawned minus tasks completed, as a running gauge — the cheapest orphan detector there is.
- • Concentration: the fraction of pending tasks on the single most common awaited resource.
- • A per-task registry with parent links is real machinery that must not leak — the registry itself becomes a memory leak if completed tasks are not removed.
- • Await labelling has to be threaded through every library that can suspend, and third-party code will not have it.
- • The dump format needs aggregation to be readable; eleven thousand individual rows is not a diagnostic artefact.
- • Interpreting it requires knowing which awaits are healthy — an idle consumer on an empty channel looks identical to a stuck task if you only read age.
- • Per-resource metrics instead of a dump: pool waiters, in-flight requests per host, permits available. Cheaper, always on, and they name the same resource — What to Instrument in a Concurrent System.
- • Distributed tracing with a span per await, which gives the same attribution across services and survives the process dying — Distributed Tracing.
- • Structured concurrency, which makes the task tree an explicit object you can walk at any time without a separate registry — Structured Concurrency.
- • Aggressive timeouts on every await, which converts "pending forever" into an error you can count, and is a fix rather than an observation — Timeouts.
What people believe, and what is true
Idle threads and low CPU mean spare capacity.
In an async runtime that is the signature of total resource exhaustion. The threads are idle because nothing can make progress, not because there is nothing to do.
A task dump is just a thread dump with more entries.
A thread dump reads stacks that already exist. A task dump reads a registry that must have been maintained, and often has no stacks at all to show.
Thousands of pending tasks is a bug.
Thousands of pending tasks is normal for async; thousands of pending tasks with a 41-second oldest age concentrated on one resource is the bug.