The question this answers
When the root operation is cancelled, which of the twelve tasks it transitively started actually stop — and in what order does cleanup run?
A checkout request that calls fraud scoring, which calls a feature store and a model server; and inventory reservation, which calls two warehouse services. Twelve tasks across four levels. The client disconnects after 900 ms.
The cancellation token tree — each node reads its parent's state — and every resource held anywhere in the tree: connections, transactions, in-flight outbound requests.
When the root is cancelled, every descendant either observes the cancellation and releases its resources, or is explicitly documented as uncancellable; no descendant survives its ancestor.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The tree, and where it breaks
Propagation works by construction when every task is started from a scope and every scope derives its cancellation state from its parent. Cancelling the root marks it; each child sees its parent's state; the mark travels down the tree in one step, and then each task independently notices at its own next check point. The order of *noticing* is unspecified. The order of *cleanup* is bottom-up, because each scope waits for its children before running its own cleanup — which is what makes "release the connection after all queries using it have stopped" work.
It breaks in three places, and all three are ordinary-looking code. A task started *without* the parent's token — create_task(work()) where work never received a signal — is a detached subtree; cancelling the root does not reach it. A library call that does not accept a token is a leaf that will not stop, and everything it holds stays held. And a layer that catches cancellation and returns normally severs the tree at that point: its own children may be cancelled, but its parent is told the work completed, which is worse than a hang because it is silent.
The wait graph below is the third failure with a twist: fraud scoring catches the cancellation, cleans up, and then blocks trying to return its connection to a pool that is exhausted — because a hundred other cancelled requests are doing the same thing. Cancellation cleanup can itself deadlock, and it deadlocks at exactly the moment the system is already degraded, which is why cleanup paths need the same care as normal paths.
The order that matters
Two orders are in play and confusing them causes real bugs. Marking is top-down and effectively instantaneous: setting the root's state makes every descendant's check return "cancelled" immediately, because they read through to the root. Settling and cleanup are bottom-up: a scope cannot finish until its children have finished, so the deepest tasks unwind first and each level cleans up only after everything using its resources has stopped.
That bottom-up ordering is the whole reason to use scopes rather than a flat list of tasks. It is what guarantees that when the checkout handler releases its database connection, no descendant is still issuing queries on it. With a flat list you have to establish that ordering yourself, at every level, and getting it wrong produces use-after-release bugs that look like driver bugs.
The schedule traces it. Note the two important beats: the model-server call takes 400 ms to notice because it is inside a read() with no signal support, and the whole tree waits for it — a single uncancellable leaf sets the settling time for everything above it. And note that cleanup at level 2 does not begin until both level-3 children have settled, which is correct and is also why the total unwind is 480 ms rather than 20.
| # | checkout (root) | fraud scope (L2) | feature store (L3) | model server (L3) | inventory (L2) | State |
|---|---|---|---|---|---|---|
| 1 | client disconnect → cancel root | · | · | · | · | root=cancelled t=900ms |
| 2 | · | · | at next await, raises cancelled; sends query cancel to server | · | · | ft=cleaning t=904ms |
| 3 | · | · | releases connection in finally; settled | · | · | ft=settled t=918ms |
| 4 | · | · | · | · | observes cancel; rolls back reservation transaction | iv=cleaning t=910ms |
| 5 | · | · | · | inside a blocking read with no signal support | · | md=running t=918ms ✕ Marked but not stopped. A leaf that cannot be cancelled sets the settling time for every ancestor above it. |
| 6 | · | cannot run its own cleanup — still waiting on model server | · | · | · | fr=waiting t=918ms |
| 7 | · | · | · | read returns at 1 300 ms; sees cancelled; discards result | · | md=settled t=1300ms |
| 8 | · | both children settled → releases its own resources; settled | · | · | · | fr=settled t=1312ms |
| 9 | both L2 children settled → releases connection; returns 499 | · | · | · | · | root=settled t=1380ms |
Keeping the chain intact
The practical work is mostly plumbing, and the discipline is: the token is a parameter, not a global. A request-scoped context passed explicitly can be checked by a reviewer and enforced by a type signature. A token stored in thread-local or async-local storage looks cleaner and silently loses its value at exactly the boundaries you care about — a thread-pool submission, a callback from a driver, a worker thread — because those boundaries do not carry the storage across.
The second rule is that wrapping is where the chain is repaired. A library that does not accept a signal can be wrapped: run it with a race against the cancellation, and accept that the underlying work continues while you stop waiting. That is not real cancellation and must be labelled as such — you have converted an uncancellable leaf into a leaked one, which is the right trade when the alternative is the whole tree hanging on it, and the wrong one if that leaf holds a connection.
The third is that every cleanup gets its own bound. A rollback that hangs must not be able to hold the tree open indefinitely; give it a short deadline and, if it expires, log loudly and abandon it. And check for cancellation on *entry* to expensive work as well as on exit from waits: a task that is created after the root was already cancelled should not begin at all, and without an entry check it will run its whole body before noticing.
1// 1. DERIVE — a child's signal is composed from the parent's, never created fresh.2function childSignal(parent: AbortSignal, budgetMs: number): AbortSignal {3 return AbortSignal.any([parent, AbortSignal.timeout(budgetMs)])4}5 6// 2. FORWARD — the token is a parameter. Every layer takes it and passes it on.7async function fraudScore(order: Order, signal: AbortSignal): Promise<Score> {8 signal.throwIfAborted() // 4. entry check: do not start9 const s = childSignal(signal, 300)10 const [features, model] = await Promise.all([11 featureStore.get(order.userId, { signal: s }),12 modelServer.score(order, { signal: s }),13 ])14 return combine(features, model)15}16 17// 3. BOUND CLEANUP — a hanging rollback must not hold the tree open.18async function withReservation(order: Order, signal: AbortSignal) {19 const txn = await db.begin({ signal })20 try {21 return await reserve(txn, order, signal)22 } finally {23 await Promise.race([24 txn.rollbackIfOpen(),25 sleep(2000).then(() => {26 log.error('rollback exceeded 2s; abandoning', { order: order.id })27 }),28 ])29 }30}31 32// THE HOLE — a library with no signal support. Racing stops the WAIT,33// not the WORK. Label it: this leaks a task, deliberately.34async function legacyCall(x: Input, signal: AbortSignal): Promise<Out> {35 return Promise.race([36 legacy.doWork(x), // keeps running after we leave37 new Promise<never>((_, rej) =>38 signal.addEventListener('abort', () => rej(signal.reason), { once: true })),39 ])40}Key points
- Marking is top-down and instant because children read through to the root; settling and cleanup are bottom-up because each scope waits for its children.
- Bottom-up settling is what makes it safe for a parent to release a connection — no descendant can still be using it.
- The chain breaks at three ordinary-looking places: a task started without the token, a library that accepts no token, and a layer that catches cancellation and returns normally.
- A single uncancellable leaf sets the settling time for every ancestor above it; correctness and promptness are separate properties.
- Pass the token as a parameter. Ambient storage silently loses it at thread-pool submissions, driver callbacks and worker boundaries.
- Every cleanup step needs its own deadline, and cleanup must not require the resource it is releasing.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Each scope holds a cancellation state that is derived from its parent's, so a check at any depth reads the whole chain.
- • Cancelling the root sets one state; no traversal or notification walk is required for descendants to observe it.
- • Each task notices independently at its own next check point, in an unspecified order.
- • On noticing, a task runs its cleanup and reports settled to its parent scope.
- • A scope runs its own cleanup only after every child has reported settled, producing bottom-up unwinding.
- • The root returns once its immediate children have settled, at which point the whole subtree is guaranteed quiescent.
- • Root cancelled at t=900; feature store notices at 904 and settles at 918; model server is in a blocking read and notices at 1300; the fraud scope cannot clean up until 1312. One leaf paced the tree.
- • Detached subtree: an inner function calls create_task without passing the signal. The root is cancelled, everything else settles, and that subtree runs to completion holding a connection — and because the scope never knew about it, nothing waits and nothing reports.
- • Severed chain: fraud scoring catches the cancellation, logs "scoring unavailable", and returns a default score. Checkout proceeds with a fabricated fraud result and returns 200 to a client that already disconnected — a correctness bug produced entirely by a swallow.
- • Cleanup deadlock: cancelled requests all try to acquire a pool connection to send their query cancels, while the pool is fully held by rollbacks that need to finish first. The cleanup path formed a circular wait (The Four Conditions).
- • Entry race: a child task is created microseconds after the root was cancelled. Without an entry check it runs its full body, issues its query, and only notices at its first await — work started after the cancellation.
- • The clean schedule: every layer takes the token, every leaf honours it, every cleanup is bounded — the tree settles in one round-trip of the slowest cancellable leaf.
- • Derived tokens guarantee that a cancellation at any level is visible to every descendant immediately, with no propagation walk to fail.
- • Scope-based settling guarantees a parent does not proceed past its cleanup while descendants are live.
- • It does NOT guarantee promptness — the tree settles at the pace of its slowest cancellable step.
- • It does NOT reach tasks started outside the tree. A detached task is not slow to cancel; it is unreachable.
- • It does NOT survive a layer that swallows the cancellation. The subtree below may stop while the parent is told everything succeeded.
- • It does NOT guarantee cleanup succeeds. A rollback can fail or hang, and without its own bound it holds the whole tree open.
- • Cancellation storms concentrate cleanup: hundreds of requests releasing connections, rolling back transactions and sending query cancels in the same instant, against pools already under pressure.
- • The cancellation state is read on every check across every task — read-mostly and cheap, but it should be an atomic read, not a locked one (Atomics: What Is Actually Indivisible).
- • Deep trees serialise settling level by level, so unwind time grows with depth even when every leaf is fast.
- • Cleanup that needs the resource it releases creates the wait cycle shown above, and it appears only under load, which is when it matters most.
- • Detached subtree: work started without the token, unreachable by any cancellation.
- • Severed chain: a layer catching cancellation and returning normally, so the parent proceeds on a fabricated result.
- • Uncancellable leaf holding the entire tree open while it finishes.
- • Cleanup deadlock, where releasing a resource requires acquiring the same exhausted resource.
- • Use-after-release when a flat task list lets a parent free a connection a child is still using.
- • Double cleanup when a task observes cancellation from two sources and its cleanup is not idempotent.
- • Work started after cancellation, because tasks check only at their first await rather than on entry.
- • On deep call trees, which is every non-trivial request path — the deeper the tree, the more the automatic propagation is worth compared with manual plumbing.
- • During overload, where propagating one cancel abandons an entire subtree of doomed work in a single action.
- • Where resources are held across levels: propagation plus bottom-up cleanup is what makes the release ordering correct without thinking about it.
- • For fan-out with fail-fast, where one branch's failure should stop every sibling and everything they started (Structured Concurrency).
- • When some descendants must complete regardless — an audit write inside a cancelled tree needs an explicitly *non*-derived token, which is easy to get wrong in both directions.
- • When cleanup is expensive: propagating a cancel to a tree of fifty tasks can produce more work than letting the remaining 100 ms of work finish.
- • When the tree includes uncancellable leaves, because propagation converts a leak into a hang and a hang is more visible but not more available.
- • When ambient context is used for propagation and the codebase has thread-pool or worker boundaries — the chain breaks invisibly and looks like it works in every test.
- • Settling time from cancel to root return, by percentile. The tail directly identifies uncancellable leaves.
- • Count of tasks still live after their root settled — the direct measure of detached subtrees, and it should be zero.
- • Cleanup failures and cleanup timeouts as their own counters, separate from work failures.
- • Pool acquisition latency during cancellation bursts, which is where cleanup contention shows up first (Connection Pool Saturation: Waiting in Front of an Idle Database in Performance).
- • Cancellations caught but not re-raised — instrumentable with a lint rule or a wrapper, and each occurrence is a potential severed chain.
- • Every function on the path grows a token parameter, and correctness depends on the *least* diligent layer.
- • Cleanup paths multiply: each resource acquisition needs a release that is idempotent, bounded, and correct when the surrounding operation never completed.
- • Exceptions to propagation — work that must survive cancellation — require a deliberate second lifetime, and that exception must be visible in the code rather than implied.
- • Testing requires cancelling at many points in the tree, because the interesting bugs are at boundaries: just before an await, just after a task is created, during cleanup.
- • A deadline carried in the request context: every layer checks the clock instead of a token. Simpler, no plumbing of a token object, and it cannot express "the client left" (Deadlines vs Timeouts).
- • Bound the work instead of cancelling it: if no operation can exceed 200 ms, propagation is unnecessary because everything settles on its own.
- • A per-request process or container that can be terminated wholesale, when the language cannot propagate cancellation into its leaves.
- • Accept the leak with a bound: let doomed work finish, but cap total concurrency so leaked work cannot exceed a known fraction of capacity (Bounding Concurrency).
Cancelling a parent task
# cooperative cancellation — the only kind that exists in practice
async def child(token):
while work_remains():
if token.cancelled: raise CancelledError # ← the check IS the mechanism
do_a_batch() # ← must be short enough to notice
await parent.cancel() # sets the flag on every child, then WAITS for them
# it cannot pre-empt a running thread; there is no safe killThree children, and the moment the parent returns
spawn(fetch_user) # nobody holds the handle spawn(fetch_orders) spawn(build_report) return "ok" # the children outlive this frame
What people believe, and what is true
Cancellation propagates automatically because tasks are nested.
It propagates through the token, not through the call stack. A task started without the token is nested in the source and detached at runtime.
Once cancelled, the tree is quiescent.
The tree is quiescent when it has *settled*, which is later — sometimes much later, at the pace of the slowest uncancellable leaf.
Catching cancellation and returning a default is graceful degradation.
It severs the chain. The parent believes the work completed and proceeds on a value that was never computed, which is a correctness bug rather than a degradation.