Probes: Readiness, Liveness and Startup
Readiness gates traffic, liveness restarts the container, startup covers a slow boot. Confusing the first two turns a dependency outage into a cluster-wide restart storm.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is each probe actually allowed to decide, and what must never be inside a liveness check?
The platform has to answer two different questions automatically — "should this instance receive requests right now?" and "is this process beyond saving?" — and the answers have completely different consequences.
Point all three probes at /health. One endpoint, one truth about whether the service is working — and make it thorough, so it checks the database and the downstream services too.
The two questions have different right answers. A pod that has lost its database should stop receiving traffic and should not be restarted, because restarting it will not bring the database back.
- The two questions have different right answers. A pod that has lost its database should stop receiving traffic and should not be restarted, because restarting it will not bring the database back.
- A thorough liveness check turns every dependency outage into a restart storm. Every replica evaluates the same failing dependency at roughly the same time, every one fails liveness, and every one is killed — simultaneously, across the fleet (Cascading Failure: When the Response to Failure Causes More Failure in the backend view).
- The restarts then make recovery harder: cold processes, empty caches, a stampede of reconnections onto the dependency that was already struggling, and zero ready replicas while they all boot (Thundering Herd in the concurrency view).
- Liveness and readiness pointed at the same endpoint means you cannot express "not ready yet, do not kill me" — which is exactly the state a service is in while it warms up.
- A liveness check that is too aggressive for a slow-starting application kills it before it ever becomes ready, producing a restart loop that looks like a crash and is actually the platform doing what you configured.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Readiness controls endpoint membership. Fail it and the pod's address is removed from the service's endpoints, so new traffic stops arriving. The container keeps running, keeps its state, and can recover (Service Discovery in Operation).
- Liveness controls restarts. Fail it repeatedly and the kubelet kills the container and starts it again. The only problem this fixes is a process that is permanently stuck in a way a fresh start resolves — a deadlock, an unrecoverable internal state.
- Startup suspends the other two until the application has finished booting. While it is failing, liveness and readiness are not evaluated at all, which is what lets a slow start have a generous allowance without making the steady-state liveness check generous too.
- Each probe has a period, a timeout, a failure threshold and, for readiness, a success threshold. The effective time to react is the period multiplied by the threshold, and both directions matter: too fast flaps, too slow leaves a broken pod in rotation.
- The probe runs from the kubelet on the node, against the container. It shares the container's CPU quota, so a throttled container can fail a probe for reasons that have nothing to do with the application (CPU Throttling: The Latency With No Error).
- Readiness is also the mechanism behind a safe rollout: a new pod takes no traffic until it is ready, and a rolling update waits for readiness before continuing. Get readiness wrong and every deployment strategy built on it becomes decorative (Rolling: Two Versions, One Database).
Three probes, three consequences
The whole lesson is the third column. Two of these gate traffic or waiting; one destroys a running process. What belongs in each follows directly from what failing it does.
| Probe | Question it answers | Consequence of failing | What belongs in it | What must never be in it |
|---|---|---|---|---|
| Readiness | Should this pod receive requests right now? | Address removed from the service endpoints; container untouched | Warm-up state, local resources, dependencies this path genuinely requires | Anything expensive enough to matter at the probe interval |
| Liveness | Is this process beyond recovery without a restart? | Container killed and restarted, with backoff | A trivially local check that the process can still respond | Any network call to a dependency — this is the trap |
| Startup | Has the application finished booting? | Container killed once the allowance is exhausted; other probes suspended until it passes | The same shallow check as liveness, with a generous threshold | A threshold so large that a broken container hides in Waiting |
The liveness trap, minute by minute
This is the incident this lesson exists to prevent. Note that nothing in the application is broken at any point, and that every automated action taken makes the situation worse.
- T+0changeA shared database has a brief availability problem; queries begin failing
- T+0signalEvery replica of every service whose liveness endpoint queries that database begins failing its probe
- T+1actionFailure thresholds are crossed. The kubelet on every node kills those containers, in-flight requests included
- T+1signalReady replica count across several services falls to zero at once. The outage is now total, not partial
- T+2actionContainers restart cold: empty caches, new connections, full startup cost, all simultaneously
- T+2signalThe reconnection surge lands on the database that was already struggling, extending its problem
- T+3signalContainers that come up fail liveness again, are killed again, and enter restart backoff
- T+4signalPaging says "everything is down". The database blip has been over for a while and is no longer visible in the symptoms
- T+6actionOn-call raises the liveness failure threshold so the killing stops; pods stay up and drain their backlog
- T+9recoveryCaches refill, error rate returns to baseline, ready count recovers
- AfterrecoveryLiveness reduced to a local check; the dependency check moves to readiness with a degraded mode. Policy check added so no liveness probe can make a network call to a dependency
Without the liveness probe, the same database blip would have produced elevated errors for its duration and nothing else. The probe converted a dependency degradation into a self-inflicted total outage, and then delayed recovery.
What should readiness actually check?
Liveness has a settled answer — as little as possible. Readiness genuinely does not, because the right answer depends on whether the dependency is unique to this replica or shared by all of them.
Does the readiness probe check the database?
when The dependency is shared by every replica, so failing readiness everywhere would remove the entire service from rotation and replace degradation with an outage.
cost Traffic keeps arriving at replicas that will return errors. Callers see failures rather than being routed elsewhere, and need their own timeouts and breakers (Circuit Breaker in the backend view).
when The dependency is per-replica, or replicas can genuinely differ — a per-shard connection, a local cache load, a leader lease.
cost If it turns out to be shared after all, every replica leaves the endpoint list at the same moment and the service is completely unavailable.
when A meaningful subset of requests can be served without the dependency and you can distinguish them.
cost The most correct and the most work: the application must track which capabilities are available and route accordingly.
when Almost never for a service taking traffic — only for workloads that receive none.
cost Every rollout drops requests, because new pods take traffic before they can serve it (Draining: Stopping Without Dropping).
How to do it properly
Most important first.
- Make liveness the shallowest check you can write. "Is this process able to respond at all" — not whether it can do useful work. Many mature services are correct with no liveness probe whatsoever.
- Make readiness reflect the ability to serve this request path, including dependencies the path genuinely requires — but understand that if every replica shares that dependency, readiness failure removes the whole service from rotation (Health Checks in the backend view).
- Use a startup probe for anything with a slow or variable boot, instead of inflating liveness' initial delay and living with a sluggish steady-state check.
- Separate the endpoints.
/livezand/readyzthat return different things, backed by different logic, is the cheapest possible protection against this entire failure class. - Consider degraded readiness: if the service can still serve some traffic without a dependency, staying ready and failing those specific requests is often better than removing every replica from rotation.
- Set thresholds from the application's real behaviour, and prefer a slightly slow reaction over a flapping one — a pod that oscillates in and out of the endpoint list is worse than one that is consistently out.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Nothing contains the liveness trap. Every replica runs the same probe against the same dependency on its own timer, so a shared dependency failure trips all of them within one probe period — including replicas of every other service configured the same way. This is the case where the mitigation is the outage, and the only real containment is not putting the dependency in the probe.
What can go wrong
- The liveness trap: a dependency check inside liveness, so a dependency outage restarts every replica of every service that checks it — an outage caused entirely by the mitigation.
- Readiness that checks a shared dependency, removing every replica from the endpoint list simultaneously and turning partial degradation into total unavailability.
- Liveness timeout shorter than the application's worst legitimate pause — a long garbage collection or a throttled period — producing restarts during exactly the load that caused the pause.
- No readiness probe at all, so traffic arrives before the application can serve it and every deploy drops requests.
- Readiness probe that always returns success regardless of state, which makes rollouts complete instantly and meaninglessly.
- Probes that are expensive: a health endpoint that queries the database on every probe interval, on every replica, adding a constant load nobody accounted for (The Connection Budget).
- A startup probe with a threshold so generous that a genuinely broken container sits in
Waitingfor a very long time before anyone is told.
- "Liveness makes the service self-healing." It restarts processes. If the fault is outside the process, it converts a dependency problem into a restart storm and removes your remaining capacity.
- "A health check should check everything." A readiness check may check what this pod needs to serve. A liveness check must check almost nothing.
- "Restarts are cheap." They discard in-flight work, empty caches, and reconnect to dependencies all at once. At fleet scale, restarts are a load-generating event.
- "No liveness probe means no supervision." The container still restarts if the process exits. Liveness only adds killing a process that has not exited — which is a narrower case than most people assume.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Ready replica count equals desired replica count in steady state, and dips only during deliberate rollouts.
- Restart counts flat when a downstream dependency has an incident — this is the direct test of whether the liveness trap is present.
- During a rolling update, no requests are served by a pod that has not passed readiness, verified by error rate at the edge rather than by pod status.
- Probe failure events name the check that failed and are readable in the pod's events, rather than appearing as an unexplained restart.
- The last-terminated reason distinguishes a liveness kill from an
OOMKilled— the two produce identical restart counts and need opposite investigations (OOMKilled: Over the Memory Limit).
- Probe settings are part of the pod template, so changing them replaces every pod. Fixing a bad liveness probe during an incident is itself a rollout, at the worst possible moment.
- The fastest mitigation for a liveness storm is to make the probe pass — removing the dependency check, or the probe itself — but that is still a rollout, so raising the failure threshold high enough to stop the killing buys time with the same mechanism.
- Nothing rolls back the restarts themselves. Whatever in-flight work they destroyed is gone, and the caches they cleared refill at the dependency's expense.
- Automate the traffic gating completely. Readiness is the correct thing to automate: it is a fast, local, reversible decision with a bounded consequence.
- Be conservative about automating the restart decision. Liveness is automation with an irreversible action and no judgement, which is exactly the combination §151 warns about (How to Automate Something).
- Automate a policy check that liveness and readiness do not point at the same path, and that liveness does not call anything over the network to a dependency (Policy as Code).
- A shallow liveness probe will not detect an application that is running but useless. That is the trade, and it is usually the right one: a deep check catches a rare failure and creates a common one.
- Readiness that includes dependencies gives correct routing when only some replicas are affected, and total unavailability when all of them are. Which is better depends on whether the dependency is per-replica or shared.
- Generous thresholds avoid flapping and leave a broken pod in rotation for longer. There is no setting that is fast and stable; pick which failure you prefer.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICThree probes with three distinct consequences is Kubernetes. A cloud load balancer has one health check, and it only gates traffic — it never restarts anything, so the liveness trap cannot exist in that form; the equivalent failure is every instance failing the check at once and the target group emptying. A VM autoscaling group with health-check-based instance replacement recreates the trap exactly, and more slowly and expensively, since the remedy is terminating and rebuilding a machine. A PaaS usually offers one health endpoint and decides for you what to do with it.
- GENERALThe underlying rule — never let an automated destructive action depend on the health of a shared external dependency — applies to any self-healing mechanism on any platform.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.