Graceful Shutdown

1. 1
   SIGTERM received

   Set a shutting-down flag; stop the scheduler; log the start.

   fails by Not being PID 1, so the signal never arrives (Containerizing a Backend).

2. 2
   Fail readiness

   Readiness probe starts returning failure. Liveness must still pass.

   fails by Failing liveness too, so the platform kills the process mid-drain.

3. 3
   Drain delay

   Keep serving normally for a few seconds while the LB removes you.

   fails by Skipping it — the cause of nearly every deploy-time 502.

4. 4
   Stop accepting

   Close the listener; pause queue consumers; send Connection: close.

   fails by Closing existing connections too, cutting off in-flight requests.

5. 5
   Drain in-flight

   Wait for active requests and claimed jobs, up to a deadline.

   fails by Waiting unbounded on a streaming connection until SIGKILL.

6. 6
   Release work

   Finish or explicitly return queue messages; cancel outbound calls.

   fails by Letting leases expire, causing redelivery while the handler still runs.

7. 7
   Close resources

   Outbound clients, then DB pool, then telemetry exporter last.

   fails by Closing the pool while a handler is mid-query; closing telemetry first and losing the shutdown trace.

8. 8
   exit(0)

   Clean exit inside the grace period.

   fails by Nonzero exit recorded as a crash, affecting restart backoff and rollout health.

Liveness and readiness diverge here on purpose: readiness says "do not send me traffic", liveness says "I am not wedged, do not kill me". Conflating them turns a graceful drain into a hard restart.

1let ready = true
2let inFlight = 0
3 
4app.get('/readyz', (_req, res) =>
5  ready ? res.status(200).send('ok') : res.status(503).send('draining'))
6 
7app.use((req, res, next) => {
8  inFlight++
9  // during drain, tell keep-alive clients not to reuse this connection
10  if (!ready) res.set('Connection', 'close')
11  res.on('finish', () => inFlight--)
12  res.on('close', () => { if (!res.writableEnded) inFlight-- })
13  next()
14})
15 
16const DRAIN_DELAY_MS = 5_000   // < LB deregistration time
17const DRAIN_DEADLINE_MS = 20_000 // < platform grace period
18 
19let shuttingDown = false
20async function shutdown(signal: string) {
21  if (shuttingDown) return // SIGTERM can arrive twice
22  shuttingDown = true
23  log.info({ signal, inFlight }, 'shutdown: begin')
24 
25  // hard stop: never let the platform be the one to end this
26  const hardKill = setTimeout(() => {
27    log.error({ inFlight }, 'shutdown: deadline exceeded, forcing exit')
28    process.exit(1)
29  }, DRAIN_DEADLINE_MS)
30  hardKill.unref()
31 
32  scheduler.stop()          // 1. no new timed work
33  ready = false             // 2. readiness starts failing
34  await sleep(DRAIN_DELAY_MS) // 3. let the LB notice
35 
36  await Promise.all([
37    closeListener(server),  // 4. no new connections
38    consumer.pause(),       //    no new queue messages
39  ])
40 
41  await waitFor(() => inFlight === 0, DRAIN_DEADLINE_MS - DRAIN_DELAY_MS)
42  await consumer.releaseInFlight() // 5. nack, do not let leases expire
43 
44  await httpClient.close()  // 6. dependency order
45  await db.end()
46  await telemetry.shutdown() //    last: the drain itself is observable
47 
48  clearTimeout(hardKill)
49  log.info('shutdown: clean')
50  process.exit(0)
51}
52 
53process.on('SIGTERM', () => void shutdown('SIGTERM'))
54process.on('SIGINT', () => void shutdown('SIGINT'))

Three things are easy to miss. The Connection: close header during drain, without which a keep-alive client sends its next request into a closing process. The idempotence guard, because SIGTERM genuinely arrives twice. And telemetry.shutdown() last, so the logs and spans describing the drain actually leave the process.