The question this answers
When this process is told to stop, what happens to the work in flight, the work in the queue, and the workers blocked waiting for more?
A payment-webhook processor: an HTTP listener accepts deliveries, a bounded queue holds them, six workers charge cards and write results. A deploy sends SIGTERM with a 30-second grace period.
The queue, the shutdown flag, and the worker registry. During shutdown the flag becomes the most contended piece of state in the process, because every loop iteration reads it.
No accepted request is lost without being recorded; no card is charged twice; and the process exits within the grace period or is killed with a known, bounded amount of unfinished work that is recoverable.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
The five steps, in order, and why the order is not negotiable
Shutdown is one sequence and reversing any two steps breaks it. Stop accepting first: close the listener or stop pulling from the broker, so the set of work in the system becomes finite. Until you do this the queue can grow while you are trying to drain it, and drain never terminates. Finish in flight second: let the six workers complete the charges they have already started, because interrupting a charge mid-way is exactly the case that produces a double charge on retry. Signal the consumers third: close the channel or push poison pills so workers blocked on an empty queue wake up and observe that no more work is coming. Wait for workers fourth, with a timeout — an unbounded join is how a process hangs forever on one stuck worker. Then exit.
The step teams skip is the third. It is invisible in normal operation: workers block on take, and while the producer is alive that is correct behaviour. The bug only manifests when the producer stops, and its signature is unmistakable — the service takes *exactly* the termination grace period to stop, on every single deploy, and then dies to SIGKILL. Nobody investigates because nobody is paged for a slow deploy.
The timeline shows both. The top half is the broken sequence: the listener closes, in-flight work finishes, and then six workers sit parked forever with nothing to wake them. The bottom half adds the signal and exits in four seconds. Cloud platforms make the cost concrete: a pod that ignores SIGTERM is SIGKILLed at the end of the grace period, and everything in the queue at that instant is gone (Graceful Shutdown: The 502 Spike Nobody Investigates and Liveness vs Readiness in Cloud).
Poison pill or close the channel
Two mechanisms deliver the end-of-stream signal, and the difference matters. Closing the channel is the better one where it exists: it is a broadcast, so every blocked receiver wakes, and the count of receivers does not appear anywhere in the code. Add a seventh worker and nothing needs changing.
Poison pills — a sentinel value that means "stop" — are what you use when the queue has no close, which includes Python's asyncio.Queue and most hand-rolled structures. A pill is *consumed by one receiver*, so you must send exactly one per consumer. This is where the bug lives: the pill count is written once, next to a pool size that later becomes configurable, and now six workers get four pills and two hang. Worse, if a worker has already exited due to an error, a pill goes unconsumed and the item count never reaches zero.
The other difference is ordering. A pill enters the queue at the tail, so it is processed after everything ahead of it — the drain is automatic and free. A close is observed as soon as the buffer is empty, which is also correct, but a close implemented as "wake everyone and discard the buffer" throws away queued work. Verify which your implementation does; "closes the channel" and "drains then closes" are different behaviours with the same method name (Channels).
| # | Main / shutdown handler | Worker 1 | Worker 2 | Worker 3 | Worker 4 | State |
|---|---|---|---|---|---|---|
| 1 | SIGTERM received; stop accepting | · | · | · | · | accepting=false queued=5 workers_live=4 |
| 2 | · | · | throws on a malformed record and exits its loop | · | · | queued=4 workers_live=3 |
| 3 | push 3 poison pills (POOL_SIZE was 3 when this was written) | · | · | · | · | queued=7 pills=3 workers_live=3 |
| 4 | · | drain items, then take pill → return | · | · | · | queued=4 pills=2 workers_live=2 |
| 5 | · | · | · | drain items, then take pill → return | · | queued=2 pills=1 workers_live=1 |
| 6 | · | · | · | · | drain items, then take pill → return | queued=0 pills=0 workers_live=0 |
| 7 | join all four registered workers | · | · | · | · | workers_live=0 joined=3 ✕ The handler waits for worker 2, which exited an hour ago and was never deregistered. The join never completes. |
| 8 | grace period expires; SIGKILL | · | · | · | · | exit=SIGKILL ✕ A clean drain was achieved and the process still died uncleanly, because the shutdown path trusted a worker count that had drifted from reality. |
The policy question: what happens to the queue
Here is the part that is a decision rather than an implementation detail. When shutdown begins there are, say, 240 items in the queue. You can drain them — process everything before exiting, which maximises work done and risks exceeding the grace period. You can abandon them — exit immediately, which is correct if the items are re-deliverable from a broker or the client will retry. You can persist them — write the remainder somewhere durable and exit fast, which is the right answer when the work is not reproducible. Or you can hand them back: refuse to acknowledge, so the broker redelivers to another instance.
What happens to work still in the queue is a policy decision, not an accident. The failure is not choosing "abandon"; the failure is having no answer and discovering during an incident that the answer was "silently dropped, 240 times, on every deploy, for eight months". Write the policy down next to the shutdown code and make it a comment someone can disagree with.
For the payment processor the answer is dictated by idempotency. If each webhook carries an idempotency key and the charge is keyed on it, abandoning is safe: the partner retries and the second attempt is a no-op (Idempotency Keys: The Mechanism and Consumer-Side Idempotency in API Design, Idempotency in Architecture). Without that key, a charge interrupted between "card charged" and "result written" is genuinely ambiguous, and the correct shutdown policy is to never start a charge you cannot finish inside the remaining budget — check the deadline before taking the next item, not after (Deadlines vs Timeouts).
10:14:02.001 INFO shutdown: SIGTERM received, grace=30s deadline=10:14:32.001 10:14:02.002 INFO shutdown: readiness probe -> NOT_READY (LB drain begins) 10:14:02.003 INFO shutdown: listener closed, accepting=false, queued=240 inflight=6 10:14:04.118 INFO shutdown: in-flight complete (6/6), elapsed=2.1s 10:14:04.119 INFO shutdown: policy=DRAIN_THEN_ABANDON budget_remaining=27.8s 10:14:04.120 INFO shutdown: channel closed; 6 workers signalled 10:14:04.121 INFO worker-3: deadline check -> 27.8s remaining, p99 item=180ms, continuing 10:14:06.844 WARN worker-5: not responding to join after 2.7s, last_item=wh_8812f 10:14:09.902 INFO shutdown: drained 240/240, dropped=0, workers_joined=5/6 10:14:09.903 WARN shutdown: worker-5 abandoned after join timeout; item wh_8812f left unacked for redelivery 10:14:09.905 INFO shutdown: exit 0, total=7.9s, within budget
Key points
- The order is fixed: stop accepting, finish in flight, signal consumers, join with a timeout, exit. Reversing any two steps breaks the drain.
- A service that takes exactly its grace period to stop on every deploy has workers blocked on a queue nobody closes. That is the signature.
- Closing a channel is a broadcast and needs no count; poison pills need exactly one per live consumer, and that count drifts.
- What happens to items still in the queue — drain, abandon, persist, hand back — is a policy you write down, not an outcome you discover.
- Check the remaining deadline before starting the next item, so you never begin work you cannot finish inside the grace period.
- Join with a timeout and log which worker did not respond; an unbounded join turns one stuck worker into a SIGKILL.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Install a signal handler that sets a shutdown flag and starts the sequence; do not do the draining inside the handler itself.
- • Fail the readiness probe first so the load balancer stops sending new requests before the socket closes — otherwise clients see connection resets (Health Checks in Cloud).
- • Close the listener or stop pulling from the broker so the in-system work set becomes finite.
- • Let in-flight operations complete; record a deadline so each worker can decide whether to start another item.
- • Signal end-of-stream: close the channel, or push exactly one sentinel per live consumer.
- • Join workers with a budget derived from the grace period minus what has already elapsed; log any that do not respond, with the item they were on.
- • Apply the queue policy for whatever remains, flush metrics and logs, and exit with a status that distinguishes clean from budget-exceeded.
- • Producer exits; three consumers are parked in take() on an empty queue; nothing signals them; join blocks until SIGKILL. The most common concurrency bug in production pipelines, and it only occurs on shutdown.
- • Shutdown flag is set while a worker is between its flag check and its blocking take: the worker checks the flag (false), then blocks, and the signal that was sent a microsecond earlier is already gone. A check-then-block race, and the reason the signal must be in the queue rather than in a flag (Lost Wakeups: The Notify That Arrived Before the Wait).
- • Four workers, three pills: three exit, one blocks forever. The pill count drifted from the pool size.
- • Worker crashed an hour ago and stayed in the registry; the join waits for a thread that no longer exists.
- • Worker takes an item with 200 ms of grace remaining and the item takes 1.8 s: SIGKILL lands mid-charge, and whether the card was charged is now unknown.
- • Correct schedule: accepting=false; in-flight completes; close broadcasts; every worker drains and observes end-of-stream; join returns; exit — with each step observable in the log.
- • A correct drain guarantees every item accepted before shutdown began is either processed, persisted, or explicitly recorded as abandoned.
- • It guarantees the process exits within the grace period, or exits with a status that says it did not.
- • It does NOT guarantee zero data loss. If the policy is abandon, items are lost by design — the guarantee is that the loss is bounded, counted and intended.
- • It does NOT guarantee in-flight external effects are undone. A charge already sent to the payment provider has happened; shutdown cannot roll it back, only record it.
- • It does NOT guarantee anything on SIGKILL. There is no handler, no unwind and no flush — everything after that point is the responsibility of durability, not of shutdown code.
- • A join with a timeout guarantees the process makes progress. It does NOT guarantee the abandoned worker stopped doing anything.
- • The shutdown flag is read by every worker on every loop iteration, making it the hottest shared variable in the process during drain — it should be an atomic load, not a locked read (Atomics: What Is Actually Indivisible).
- • A close that wakes every waiter at once produces a burst of contention on the queue lock as all workers race to drain (Thundering Herd).
- • Draining concentrates load on downstream dependencies: six workers going flat out on the remainder can spike the payment provider at the exact moment the deploy is rolling (Fan-Out: Waiting for the Slowest of Seven in Performance).
- • Logging during shutdown contends on the log writer, and a synchronous flush of a large buffer can itself consume a meaningful share of the grace period.
- • Shutdown hang: consumers blocked on a queue with no end-of-stream signal.
- • Undercounted poison pills leaving some workers blocked forever.
- • Stale worker registry causing an unbounded join on a thread that already exited.
- • Silent data loss: items in the queue discarded with no counter and no log line.
- • Duplicate side effects: an operation interrupted between the external effect and the local record, retried after restart.
- • Deadlock during shutdown when the shutdown handler takes a lock a worker already holds while that worker waits on something the handler must do.
- • Connection resets for clients because the socket closed before the load balancer stopped routing.
- • On every deploy, which in a normal service is dozens of times a week — shutdown correctness is exercised far more often than most failure paths.
- • On autoscaling scale-in and node preemption, where instances are terminated routinely and the grace period may be shorter than a deploy's (The Instance Lifecycle in Cloud).
- • When work is not idempotent, because the only way to avoid ambiguity is to finish or explicitly not start.
- • When the queue holds meaningful work — a drain that saves 240 real payments per deploy is worth the code.
- • When draining takes longer than the grace period, converting a fast clean exit into a slow dirty one. Bound the drain by the deadline, not by the queue.
- • When the work is re-delivered anyway. Draining a broker queue that would redeliver in seconds adds risk and complexity for no gain — refuse to acknowledge and exit.
- • When shutdown code is complex enough to have its own bugs. It runs rarely, is hard to test, and a broken shutdown handler can hang a process that would otherwise have exited fine.
- • When "graceful" is used to avoid making the work durable. Durability solves the problem in all cases including SIGKILL; a drain solves it only when you get a signal and enough time.
- • Shutdown duration per instance, as a distribution. A tight cluster at exactly the grace period is the blocked-worker signature; a rising p99 is a drain that is outgrowing its budget.
- • Exit status split: clean exit versus SIGKILL. Any SIGKILL at all should be visible, and most teams do not graph this at all.
- • Items remaining at exit, and items abandoned — as counters, so the policy is observable rather than assumed.
- • Time in each shutdown phase, logged explicitly, because "shutdown is slow" is useless and "in-flight completion took 24s" is actionable.
- • Duplicate-operation rate after a deploy: a spike correlated with rollouts means work is being interrupted and retried non-idempotently ("What Changed?" — Deploy Markers and the Invisible Deploys in Performance).
- • A lifecycle now exists explicitly: something must own the ordering of stop-accepting, signal, join and exit, and that owner is new code with no natural home.
- • Every worker loop gains a deadline check and an end-of-stream branch, which changes the shape of code written when shutdown was not a consideration.
- • The queue policy must be decided, documented and revisited whenever the work changes character.
- • Testing requires driving a real signal against a running process with a non-empty queue — an integration test, not a unit test, and one most teams do not have.
- • Make the work durable and idempotent, then abandon on shutdown. This is strictly stronger than draining because it also survives SIGKILL and hardware failure (Idempotency in Architecture).
- • Use a broker with acknowledgement: do not ack until the work is complete, and unacked items redeliver automatically when the instance disappears (Message Queues in Architecture).
- • Checkpoint long operations so a restart resumes rather than repeats — turns a 20-minute job from un-drainable into interruptible.
- • Shorten the unit of work. Items that take 50 ms make drain trivial; items that take 5 minutes make every shutdown policy hard, and reducing the item size is often easier than perfecting the drain.
What people believe, and what is true
The workers will exit when the producer does.
A thread blocked in take() has no idea the producer exited. Nothing about a producer terminating touches a consumer waiting on a queue — the signal has to be sent.
Graceful shutdown means no work is lost.
It means the loss is bounded and chosen. SIGKILL, hardware failure and an over-budget drain all still lose work; only durability covers those.
A slow deploy is a deploy-tooling problem.
A service that consistently takes its full grace period to stop is reporting a blocked-consumer bug. The tooling is doing exactly what it was told.