The question this answers
What exactly does an orchestrator repair on its own, and which failures does it dutifully make worse?
A worker container occasionally exits on an unhandled edge case, and machines are rebooted for patching every week. Neither event may reduce serving capacity for longer than it takes to start a replacement, and neither may require a human.
Continuous restoration of the declared replica count: a failed container is restarted, a lost pod is recreated, and a lost machine's pods are rescheduled elsewhere — without anyone being paged.
The mechanism, in five steps
Desired is 3. Pods A, B and C are running. Pod B's process exits with a non-zero code. What happens next is deliberately unremarkable: the node agent notices the container exited and restarts it in place, subject to a backoff. If the node itself disappears, the pod cannot be restarted in place, so after a grace period the pod is marked gone, the replica-set controller observes actual 2 against desired 3, and creates a new pod — which the scheduler places on a surviving node.
Two different repairs are happening there and it is worth keeping them apart. Container restart is local: same pod, same IP, same node, backoff between attempts. Pod replacement is a control-loop action: a *new* pod with a new name, a new IP and a new node. The first is fast and preserves nothing but the pod identity; the second is slower and starts genuinely fresh.
The timing is the part that surprises people during a machine failure. The cluster does not know instantly that a node is gone — it waits for the node to stop reporting, then waits a grace period before evicting its pods, precisely because a brief network blip should not trigger a fleet-wide reshuffle. During that window your capacity is genuinely reduced and every dashboard says the pods are Running. That gap, typically some tens of seconds to a few minutes, is why redundancy is a design decision and not something healing gives you for free.
- 1Steady state
Desired 3, actual 3, all Ready and receiving traffic.
None — but this is also what a cluster looks like when nothing is verifying the replicas are on different machines.
- 2Container exits<1s
Pod B's process exits non-zero. The node agent notices within a second or so.
If the process hangs instead of exiting, nothing notices at all without a liveness probe.
- 3Restart in place0–5s, then backoff
Same pod, same IP, same node. Restart count increments. Backoff grows with repeated failures.
A crash caused by config restarts into the same crash — this is where CrashLoopBackOff comes from.
- 4Node stops reporting~40s to detect
node-07 goes silent. The cluster waits before concluding it is gone.
Its pods still show
Running. Capacity is already reduced and no dashboard says so. - 5Pods marked for eviction~5 min default
After the grace period the node is
NotReadyand its pods are marked for deletion.Tune this too low and a network blip reshuffles the fleet; too high and outages last longer.
- 6Controller creates replacementsseconds
Actual 2 vs desired 3 — a new pod is created with a new name and IP.
If the surviving nodes have no free capacity, the replacement stays Pending and the gap never closes.
- 7New pod Ready10s–2min
Image pulled, container started, readiness passed, added to endpoints.
Total recovery is dominated by image pull and application warm-up, not by the control loop.
What it does not heal
Self-healing restores a *count*. It has no model of correctness. This is the single most important limitation in the module, because it is where teams misplace their trust.
A pod that crashes because of a bad configuration value will be restarted forever. The loop is working exactly as designed: the count is short, so create a pod; the pod dies, so create another. The backoff grows to a few minutes and the cluster settles into a stable, permanent state of failure — CrashLoopBackOff — that will happily persist for weeks. Nothing about that is a bug, and nothing about it will alert you unless you asked for an alert on restart counts.
The same blindness covers everything else that is not a count. A pod whose process is alive but wedged is Running, so nothing acts — that is what a liveness probe is for, and a liveness probe with a poorly chosen endpoint creates its own restart loop. A dependency being down is not a replica gap. A data corruption is not a replica gap. And critically, healing hides degradation: a workload that OOMs every eleven minutes and is silently restarted looks fine on an availability dashboard and terrible to the users whose requests were in flight.
| Failure | Does self-healing fix it? | What actually happens | What you need instead |
|---|---|---|---|
| Container exits on an unhandled error | Yes | Restarted in place within seconds; capacity restored. | Nothing. This is the case it was built for. |
| Node dies | Yes, after a delay | Pods evicted after the grace period, rescheduled onto surviving nodes. | Spare capacity on other nodes, and spread rules so the loss is partial. |
| Bad config or missing secret | No — it amplifies | Restarted into the same crash forever; CrashLoopBackOff becomes a stable state. | An alert on restart count, and a rollout gate that catches it before full replacement. |
| Process alive but wedged | No | Pod stays Running and keeps receiving traffic. Nothing acts. | A liveness probe that tests real work — carefully, see Liveness vs Readiness. |
| Dependency down (database, queue) | No | Pods are healthy and failing every request. There is no count gap to close. | Readiness that reflects dependencies, plus application-level retries and circuit breaking. |
| Slow memory leak | Masks it | Periodic OOM kill and restart. Availability looks fine; latency and in-flight requests do not. | Alert on restart count and on OOM kills specifically — see OOM Kills and CPU Throttling. |
| All replicas on the failed node | No | Total outage, then a full recovery from zero. | Topology spread or anti-affinity. Healing cannot un-place pods that were already placed badly. |
Reading the events instead of guessing
When healing is not working, the cluster almost always says why, in the events attached to the object. This is the most under-read diagnostic surface in Kubernetes: engineers reach for logs, which show the application's view, when the events show the *platform's* view — why a pod was killed, why an image would not pull, why nothing was scheduled.
Two patterns below are worth memorizing. Reason: OOMKilled with Exit Code: 137 means the container exceeded its memory limit and was killed by the kernel — not a crash in your code, a limit you set. Back-off restarting failed container with a growing interval means the loop has given up hurrying; the pod will keep trying every few minutes indefinitely, which is the stable-failure state described above.
$ kubectl describe pod api-c5f7
Containers:
api:
State: Waiting Reason: CrashLoopBackOff
Last State: Terminated Reason: Error Exit Code: 1
Restart Count: 147 <-- stable failure, 6 hours old
Events:
Warning BackOff 4m (x312 over 6h) kubelet Back-off restarting failed container
# the loop is working; the deployment is wrong
$ kubectl describe pod worker-8b6e
Containers:
worker:
State: Running
Last State: Terminated Reason: OOMKilled Exit Code: 137
Restart Count: 38 <-- healing is MASKING a leak
Limits: memory: 512Mi
Events:
Warning OOMKilling 11m (x38 over 7h) kernel Memory cgroup out of memory
# availability graph: 100%. users: not fine.Key points
- Two distinct repairs: a container is restarted in place on the same node, or a pod is replaced entirely with a new name, IP and node.
- Node failure detection is not instant — expect tens of seconds to minutes during which pods show
Runningand capacity is already gone. - Self-healing restores counts, not correctness. A pod crashing on bad config is restarted into the same crash indefinitely.
- CrashLoopBackOff is a stable state, not a transient one; without an alert on restart counts it persists silently for weeks.
- Healing masks degradation: a workload OOM-killed every eleven minutes looks healthy on an availability dashboard.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The node agent watches container exits and restarts failed containers in place, with an exponential backoff up to a few minutes.
- • Nodes send periodic heartbeats; missing them for a threshold marks the node
NotReady. - • After an eviction grace period, pods on an unreachable node are marked for deletion, which finally creates a count gap.
- • The replica-set controller observes actual below desired and creates replacement pods.
- • The scheduler places the replacements on nodes with free capacity; the node agent pulls, starts and probes them.
- • Alerting on restart counts and OOM kills — without it, the two worst outcomes above are completely invisible.
- • Spare capacity: healing can only reschedule if somewhere has room, which is a capacity-planning obligation, not a cluster feature.
- • Probe design, including the discipline not to add a liveness probe that restarts a pod for a dependency being slow.
- • Applications that tolerate being killed at any moment — idempotent work, checkpointed progress, no in-memory state you cannot lose.
- • CrashLoopBackOff as a stable state: the loop restarts a fundamentally broken pod forever and reports no incident.
- • A liveness probe pointed at a slow dependency: every replica is restarted simultaneously during a database slowdown, turning latency into a full outage.
- • No spare capacity: replacements are Pending, desired stays above actual, and the cluster runs degraded with no alarm.
- • Healing that hides a leak: repeated OOM kills keep availability metrics green while dropping every in-flight request.
- • Eviction thresholds tuned too aggressively: a brief network partition triggers a fleet-wide reschedule that is worse than the blip.
- • Recovery time is dominated by image pull and application warm-up, not by the control loop, so Why Image Size Is an Infrastructure Problem and Startup Time & Cold Start set your real MTTR.
- • Large fleets amplify simultaneous failure: losing one node of five is a 20% capacity drop that the survivors must absorb instantly.
- • Mass rescheduling after a node loss creates a registry and dependency thundering herd — hundreds of pods pulling and connecting at once.
- • Automatic restart also restores a compromised workload; the loop will faithfully re-run an image an attacker replaced in the registry.
- • Eviction and rescheduling move workloads between nodes, so node-level isolation assumptions must hold across the whole pool.
- • Restart loops are a useful detection signal — a workload restarting on inputs it never used to receive is worth investigating, not just fixing.
- • A pod that restarts frequently re-reads its secrets frequently, which makes secret-access audit trails noisy in ways worth anticipating.
- • Spare capacity for healing is real capacity that is billed continuously and used rarely — the direct cost of automatic recovery.
- • Crash loops cost image pulls, egress and control-plane work continuously while delivering nothing.
- • Masked degradation is a cost too: a leaking workload right-sized against its inflated usage locks in overspend.
- • Restart count per container, alerted on rate rather than absolute value — the single highest-value alert in an orchestrated system.
- • OOM kill events specifically, separated from ordinary restarts, because they mean something different and are fixed differently.
- • Ready replicas versus desired, per workload, which is the direct measure of whether healing is currently succeeding.
- • Pending pod count and node readiness, which together explain a gap that healing cannot close.
- • The signal that lies: uptime and availability percentages. A pod restarted 38 times in seven hours reports excellent availability.
- • A process supervisor with a restart policy on a single VM — same restart semantics, no cluster, and it covers the common case.
- • An instance group with health checks that replaces unhealthy machines, which is self-healing at machine granularity.
- • A managed platform that restarts your process for you and exposes the restart count without you configuring anything.
- • For work that can simply be retried, a queue with visibility timeouts heals better than restarting the consumer — the message returns and another worker takes it.
- • Buys unattended recovery from transient failure; costs the risk that permanent failure is quietly tolerated instead of escalated.
- • Buys resilience to machine loss; costs spare capacity that is billed all month and used for minutes.
- • Buys fast restarts; costs the requirement that every workload be safe to kill at any moment, with no in-memory state worth keeping.
Desired 3, actual 2: what the controller does next
loop: observe ready 3 diff 3 desired − 3 ready = 0 act no action
What people believe, and what is true
Kubernetes is self-healing, so failures fix themselves.
It restores replica counts. Bad config, wedged processes, failed dependencies and data corruption are all outside what a count can express.
CrashLoopBackOff is a temporary state.
It is a stable one. The backoff caps at a few minutes and the pod keeps failing indefinitely, with no alert unless you configured one.
If pods show Running, the workload is fine.
Running means a process started. It says nothing about restart history, readiness, or whether the pod has been OOM-killed thirty-eight times today.