The question this answers
My caller disconnected. Should I stop the work I am doing on their behalf — and can I safely?
Best-effort cessation of work whose result is no longer wanted, with no guarantee that it stops, and no guarantee about how far it got. Cancellation is safe to the extent that the cancelled operation has a defined state at every point it can be interrupted — which is a property of the operation, not of the cancellation mechanism.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A worker knows whether it has received a cancellation signal and whether its own deadline has passed. It does not know whether the caller is genuinely gone or merely quiet, whether a cancellation was sent and lost, or whether the caller will retry the same logical operation in a moment. Absence of a cancellation is not evidence that the caller is still waiting — which is why the deadline, not the signal, is the reliable stopping condition.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
Two ways to learn nobody is listening, and only one is reliable
The first is implicit: the connection closes. Cheap, requires no protocol support, and it is the mechanism behind a browser tab being closed mid-request. It is also unreliable in both directions — a connection can stay open long after the client process died (until keep-alive notices, which may take minutes), and it can close while the client is very much still there and about to reconnect. Worse, in any architecture with a proxy, load balancer or connection pool between client and server, the client’s disconnect frequently does not close the server-side connection at all.
The second is explicit: a cancellation message, a gRPC CANCELLED, a context cancellation propagated through the call graph. Precise, expresses intent, and it is a message — so it can be lost, delayed, or arrive after the work finished. A system that relies solely on explicit cancellation has a permanent tail of work that was never told to stop.
This is why the deadline is the load-bearing mechanism and cancellation is the optimisation. A deadline requires no message: it expires locally, on its own, with no cooperation from anyone. Cancellation stops work sooner when it arrives, and the deadline stops it eventually when the cancellation does not. Build the deadline first; add cancellation to reclaim the interval between "caller left" and "budget expired".
When cancelling is not safe
Cancelling a pure read is free: stop, release resources, discard. Cancelling a write is where honesty is required, because interrupting an operation leaves the system in whatever state that operation had reached, and that state may be one the application never intended to exist.
The clear cases first. Cancelling *before* the write is issued is safe — nothing happened. Cancelling *after* a single-shard transaction has committed is also safe in the sense that the state is consistent, though the caller will never learn the outcome, which is A Timeout Tells You Nothing About Whether It Happened again. The dangerous case is cancelling in the middle of a multi-step effect: three of five services updated in a saga; two of four keys written in a batch; a file half-uploaded; a payment authorised but not captured. There is no rollback, because there was never a transaction spanning them.
Note the asymmetry that makes this genuinely hard: the cancellation itself is ambiguous in the same way a timeout is. A caller that sends a cancel does not know whether the work stopped, whether it completed anyway, or whether it stopped halfway. So cancellation cannot be used to establish that an effect did not happen. It is a request to stop wasting effort, never a statement about state.
The practical rules that follow: make cancellation checks happen at points where the state is well-defined — between steps, not inside them. Treat any in-progress durable write as an atomic region and let it finish rather than tearing it down. For multi-step effects, either make each step individually idempotent so a later retry converges, or record intent durably first so a compensator can clean up what cancellation abandoned. And for effects that are externally visible and not reversible, do not cancel at all — the wasted capacity is cheaper than the partial state.
| Operation | Safe to cancel? | Why | What to do instead |
|---|---|---|---|
| Read / query / recomputeprotocol | Yes | No state changes; discard and release | Cancel freely |
| Single-key idempotent writeassumption | Yes | Either it happened or it did not; a retry converges | Cancel; rely on idempotence |
| Multi-step saga, mid-flightprotocol | No | Leaves 3 of 5 steps applied with no rollback | Let the step finish; run compensation deliberately |
| External side effect (charge, email)protocol | No | Cancellation cannot un-send; outcome unknowable | Complete and record; reverse as a new action |
| Streaming uploadtypical | Partially | Partial object may be visible to readers | Write to a temp key; commit by rename |
Where the waste actually is: queues and fan-out
Synchronous cancellation reclaims the milliseconds between the caller leaving and the deadline firing — real, but modest. The large wins are in two other places.
Queues. A job sitting in a queue for four minutes when its requester waited two seconds is pure waste, and the check to avoid it is a comparison at dequeue time. Systems that carry deadlines into their queues routinely find that a substantial share of dequeued work is already expired during an incident, and dropping it is what lets the queue drain at all. This is the single highest-leverage place to apply the idea.
Fan-out. A request that scattered to 50 shards and has received enough responses to answer should cancel the other 30. This is the mechanism behind tied requests in Send a Second Request After p95 and Take Whichever Answers First: send to two replicas, and the moment one starts executing it cancels the other. Without cancellation, hedging doubles load; with it, the extra load is bounded by the window between dispatch and start.
And one caution that comes from operating these systems: cancellations are traffic. During an incident, thousands of clients disconnecting produce thousands of cancellation messages to an already-saturated service. If cancellation is expensive to process — if it takes a lock, writes a record, or fans out further — you have added load precisely when you cannot afford it. Cancellation must be cheaper than the work it saves, which is the same economics as rejection in Rejecting Work on Purpose — and Rejecting It Cheaply Enough to Help.
1async function processOrder(ctx: Ctx, order: Order) {2 // Safe point: nothing durable has happened yet.3 if (isCancelled(ctx)) return { status: 'cancelled' }4 5 const reserved = await inventory.reserve(order) // step 1 (durable)6 7 // Safe point: step 1 completed and is individually reversible by compensation.8 // We do NOT check inside reserve() — an interrupted reserve has no defined state.9 if (isCancelled(ctx)) {10 await inventory.release(reserved) // deliberate compensation11 return { status: 'cancelled' }12 }13 14 // Uninterruptible: an external effect that cannot be un-done. Cancelling here15 // buys a few hundred milliseconds and risks a charge with no order attached.16 const charge = await withoutCancellation(() => payments.capture(order))17 18 return { status: 'ok', charge }19}20 21// Queue consumers: the highest-value check in most systems, and it is one line.22function shouldRun(job: Job): boolean {23 return job.deadlineAtMono > nowMono() // expired => drop, nobody is waiting24}Key points
- A deadline expires locally and needs no message; cancellation is a message and can be lost. Build the deadline first.
- Connection close is an unreliable cancellation signal in both directions, and proxies routinely hide it entirely.
- Cancellation is ambiguous exactly like a timeout: sending one tells you nothing about whether the effect happened.
- Check for cancellation between steps, never inside a durable write — an interrupted step has no defined state and no rollback.
- The biggest wins are dropping expired work at dequeue and cancelling redundant fan-out branches, not shaving synchronous milliseconds.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The request carries a deadline and a cancellation channel through the call graph.
- • The caller cancels explicitly on disconnect, on an early sufficient answer, or on its own deadline expiring.
- • Each worker checks the channel and the deadline only at points where its state is well-defined — between steps.
- • Durable multi-step effects are wrapped so that an in-progress step completes, and compensation is invoked deliberately rather than by abandonment.
- • The deadline serves as the backstop: work stops when the budget expires regardless of whether any cancellation arrived.
- • The cancellation message is lost, and only the deadline stops the work.
- • The cancellation arrives after the work committed, leaving the caller believing it was cancelled.
- • A proxy or connection pool masks the client disconnect, so the server never learns.
- • Cancellation interrupts a multi-step effect and leaves state that no code path expects.
- • Cancellation is not propagated past a queue or thread-pool handoff, so the expensive part of the work continues.
- • Queue workers producing results nobody reads: high queue age, high completion rate, and a result-discard or write-conflict rate to match. Capacity is fully consumed doing work that arrived too late.
- • Orphaned partial state: reconciliation finds inventory reserved with no order, or an upload with a partial object visible to readers. No error was logged anywhere, because from each service’s view nothing failed.
- • Charge without order: a payment captured while the surrounding request was cancelled, so the ledger has an entry the order system knows nothing about. Found by reconciliation, never by monitoring.
- • Cancellation storm: a mass client disconnect during an incident produces a wave of cancellation messages that adds load to the saturated service. Request rate rises as user traffic falls.
- • Cancellation is one-way and needs no agreement, which is exactly why it cannot confirm anything about state.
- • Establishing that an effect did *not* happen requires shared durable state — an intent record or an idempotency key — and that is a coordination point with its own availability.
- • Tied requests use cancellation as a mutual-exclusion hint between replicas; it reduces duplicate work but never guarantees it, so the operation must tolerate both replicas running.
- • If cancellation is lost, the deadline still bounds the waste; the system degrades to the un-cancelled case rather than breaking.
- • Cancelled reads leave no trace. Cancelled multi-step writes leave partial state that only reconciliation will find.
- • The caller cannot distinguish "cancelled before any effect" from "cancelled after the effect committed", so any correctness argument must hold for both.
- • Detect: count work completed for requests already past deadline, and dequeue-time expiry rate. Both should be small and are usually not measured at all.
- • Contain: place cancellation checks only at defined boundaries, and mark uninterruptible regions explicitly in code rather than by convention.
- • Recover: for abandoned multi-step effects, run compensation deliberately — cancellation abandons, it does not undo.
- • Reconcile: sweep for orphans created by cancellation (reservations without orders, uploads without commits) on the same schedule you reconcile timeouts, because they are the same class of bug.
- • Verify: inject client disconnects during a load test and confirm that downstream work stops, that no partial state appears, and that the cancellation traffic itself does not spike load.
- • Work completed after its deadline, per service — the direct measure of what cancellation would save.
- • Jobs dropped at dequeue because they were expired, which under load should be a large number and is the healthy signal.
- • Cancellation delivery rate versus caller-disconnect rate; a large gap means cancellations are being lost or not propagated.
- • Orphan counts from reconciliation, attributed to cancellation rather than to timeouts, so the two causes stay distinguishable.
- • Expensive read paths — search, aggregation, model inference — where abandoning a computation returns real capacity immediately.
- • Queue-backed work, where dropping expired jobs at dequeue is nearly free and often the difference between a queue that drains and one that does not.
- • Fan-out and hedging, where cancelling redundant branches is what keeps the extra load bounded.
- • Operations with external side effects, where cancelling risks partial state and cannot un-do anything already visible outside the system.
- • Short operations, where the cancellation machinery costs more than the work it saves.
- • Systems whose steps are not individually idempotent or compensable, where cancellation converts a slow request into a data-integrity incident.
- • Rely on deadlines alone: simpler, needs no channel, and wastes the interval between the caller leaving and the budget expiring.
- • Let the work finish and discard the result — always safe, and the right choice for anything with an external effect.
- • Make every step idempotent and re-drivable, so an abandoned request is a retry problem rather than a partial-state problem.
- • Convert to an explicit asynchronous job with a cancel endpoint, giving cancellation defined semantics and an observable state machine instead of best-effort interruption.
Cancellation: the caller is gone
| Operation | Safe to cancel? | Why |
|---|---|---|
| read / query | yes | No effect to undo. Stopping is pure saving. |
| idempotent write with a key | yes | A half-applied write is re-applied identically by the retry; the key collapses the duplicate. |
| non-idempotent write (charge, send, increment) | no | The remote side may have applied it. Stopping the caller does not stop the effect, and you now have no record of it. |
| multi-step saga in flight | no | Cancelling mid-saga leaves state that matches no intention. It needs compensation, which is a forward action, not an undo. |
| work already dispatched downstream | no | Your cancellation is a message. It may not arrive; the downstream may finish anyway. |
What people believe, and what is true
The client disconnected, so we can safely stop.
You can stop safely only if the work has a defined state at the point you stop it. A saga three steps in has no such state and no rollback.
Cancellation guarantees the work did not happen.
It guarantees nothing. The work may have completed before the cancel arrived, and the cancel may never have arrived at all.
Connection close is a reliable cancellation signal.
It can arrive minutes late, never arrive, or be absorbed by a proxy. It is a hint; the deadline is the mechanism.
Cancellation is free.
Cancellations are messages, and a mass disconnect during an incident produces a wave of them against an already-saturated service.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
If nobody is waiting for the answer, doing the work is waste. Stop when you can — but only at points where stopping leaves the system in a state you understand.
Practical
Carry a deadline everywhere and treat it as the real stopping condition; add explicit cancellation to stop sooner. Check for cancellation between steps, never inside a durable write, and mark external effects uninterruptible. The highest-value single change is dropping expired jobs at dequeue.
Advanced
Model each operation as a sequence of states and ask which are safe to be left in permanently, because cancellation, a crash and a network partition all abandon the operation in exactly the same way. That collapses cancellation safety into the same analysis as crash recovery: an operation safe to abandon at any point is one whose every intermediate state is either invisible or compensable. Systems built this way get cancellation almost for free, and systems that are not cannot get it at any price.
Apply it
- 💬 A user closes the browser tab mid-request. What should each hop below do, and what can each actually know?
- 💬 Which operations in your system are unsafe to cancel, and what makes them unsafe?
- 💬 Where in a queue-backed system does cancellation pay for itself most?