Running a Backend on Kubernetes

1spec:
2  replicas: 6
3  strategy:
4    rollingUpdate:
5      maxUnavailable: 0      # never dip below capacity
6      maxSurge: 2
7  template:
8    spec:
9      # must exceed drain delay + drain deadline in the app
10      terminationGracePeriodSeconds: 45
11      containers:
12        - name: api
13          image: registry/api@sha256:...   # digest, not a tag
14 
15          # "am I wedged?" — no dependency calls, ever
16          livenessProbe:
17            httpGet: { path: /livez, port: 8080 }
18            periodSeconds: 10
19            failureThreshold: 3
20 
21          # "should I get traffic?" — deps + draining flag
22          readinessProbe:
23            httpGet: { path: /readyz, port: 8080 }
24            periodSeconds: 2        # short: drain must be noticed fast
25            failureThreshold: 2
26 
27          # protects a slow start from the liveness probe
28          startupProbe:
29            httpGet: { path: /livez, port: 8080 }
30            periodSeconds: 5
31            failureThreshold: 30
32 
33          resources:
34            requests: { cpu: "500m", memory: "512Mi" }
35            limits:   { memory: "512Mi" }   # equal to request
36            # no CPU limit: requests give a share; a tight limit
37            # adds throttling that looks like slow code
38 
39          env:
40            - name: NODE_OPTIONS          # runtime must know the limit
41              value: "--max-old-space-size=384"
42 
43          securityContext:
44            runAsNonRoot: true
45            readOnlyRootFilesystem: true
46            allowPrivilegeEscalation: false

The two most consequential lines are the ones that look like tuning. terminationGracePeriodSeconds must be larger than the application's own drain budget, or the careful shutdown never finishes. And --max-old-space-size below the memory limit is what stops a runtime that reads host memory from planning a heap the cgroup will not allow — an OOMKill with no application error.

app.get('/health', async (_req, res) => {
  await db.query('SELECT 1')     // dependency check
  await redis.ping()
  res.send('ok')
})

// livenessProbe:  /health
// readinessProbe: /health
//
// DB gets slow ->
//   readiness fails  (correct: stop traffic)
//   liveness fails   (wrong: kill every pod)
// -> full restart of the fleet, into a slow DB

// LIVENESS: "is this process wedged?"
// No I/O. No dependencies. Cheap and local.
app.get('/livez', (_req, res) => res.send('ok'))

// READINESS: "should I receive traffic?"
app.get('/readyz', async (_req, res) => {
  if (draining) return res.status(503).send('draining')
  if (!configValidated) return res.status(503).send('starting')
  try {
    await db.query('SELECT 1')   // cached for ~1s to avoid
    res.send('ok')               // probing the DB every 2s
                                 // from every replica
  } catch {
    res.status(503).send('db unavailable')
  }
})

Readiness failing removes the pod from the load balancer and is reversible in seconds. Liveness failing destroys the process and its warm state. Tying the destructive action to a dependency you do not control means an external problem becomes a self-inflicted one — and the restarts add load to the very dependency that was struggling.