The question this answers
When actual state drifts away from desired state, who notices, and what exactly do they do about it?
The API must have ten healthy replicas at all times. Containers crash on bad input, machines are rebooted for patches, and a deploy replaces every replica. Nobody is watching a terminal at 03:00.
A continuously running loop that compares what you asked for against what is actually running and takes the smallest action that closes the gap — forever, without being asked again.
Desired 10, actual 8
Forget every manifest you have seen. Start with a table with two columns. On the left, what you asked for: ten containers running api:v7. On the right, what is actually running: eight. Two crashed twenty minutes ago on a machine that nobody is looking at.
Now ask the only question that matters: who notices, and what do they do? In a hand-run system the answer is a person, eventually, if a graph is on a wall and someone is looking at it. That answer has three failures baked in: noticing is manual, it is slow, and it does not happen at night. Every serious orchestration idea in the last two decades is an attempt to replace that person with a loop.
The loop is not clever. It reads desired state, reads actual state, subtracts, and acts on the difference. 10 − 8 = 2, so create two. It then does it again, and again, indefinitely — which is what makes it different from a script. A script runs when someone runs it. A loop is running right now, and it was also running at 03:00.
DESIRED (what you declared) ACTUAL (what the machines report)
------------------------------ --------------------------------
api:v7 × 10 replicas api-3f2a Running node-04
512Mi memory each api-9c11 Running node-04
probe GET /healthz api-77bd Running node-11
api-e402 Running node-11
api-1a90 Running node-02
api-b3cd Running node-02
api-4e88 Running node-09
api-0d21 Running node-09
api-c5f7 CrashLoop node-07 <-- 20 min ago
api-8b6e <gone> node-07 <-- node rebooted
gap = 2 replicas
question: who computes this line, and how often?The control loop, derived
To close that gap automatically you need exactly four things, and Kubernetes is a large, careful implementation of exactly these four. One: a place to store the desired state so it survives restarts and disagreements — a consistent datastore behind an API. Two: a way to learn the actual state — agents on every machine reporting what they are running. Three: something that computes the difference. Four: something authorized to act on it.
Kubernetes calls part three and four *controllers*, and there are dozens of them, each responsible for one kind of gap. A controller is a while true loop with a very boring body: observe, diff, act. The Deployment controller closes gaps in replica count. The node controller closes gaps created by machines disappearing. The scheduler closes the gap between "this workload exists" and "this workload is assigned to a machine".
The consequence that matters operationally: Kubernetes never finishes. There is no "deployed" state. There is only a loop that currently has nothing to do, and the same loop is what will replace a container an hour from now when it dies. That is why you never restart a pod by hand to fix a count — the loop already did, or it is about to, and your manual action is either redundant or a race.
Declarative is not a style preference
People describe the difference between imperative and declarative infrastructure as a matter of taste. It is not. An imperative instruction is a *verb executed once*: "start two containers". A declarative statement is a *noun that stays true*: "there are ten containers". The first is correct at the moment it runs and meaningless afterwards. The second is a standing claim the system is obligated to keep making true.
That difference is exactly why the loop can heal. It knows what "correct" means at all times, because correct was written down as a state rather than as a sequence of actions. It is also why the loop can *fight you*: change a running workload by hand and the controller will observe a gap between your change and the declared state, and dutifully undo it. Newcomers experience this as Kubernetes being stubborn. It is the feature working — see Drift: When the File and Reality Disagree for the same phenomenon in Terraform.
The honest limit: the loop guarantees the declaration, not the outcome. Declare ten replicas of an image that crashes on startup and you get ten crash loops, restarted with perfect discipline, forever. Self-Healing, and What It Does Not Heal is explicit about what this mechanism cannot repair.
# runs once, and is correct for exactly as long as nothing changes docker run -d --name api-1 registry/api:v7 docker run -d --name api-2 registry/api:v7 # ... eight more # node-07 reboots at 03:00. Nothing in this file has an opinion about that. # The count is now 8 and there is no record anywhere that it should be 10.
# a statement of fact the cluster must continuously make true kubectl apply -f api.yaml # spec.replicas: 10 # node-07 reboots at 03:00. # The controller observes actual=8, desired=10, and schedules 2 replacements. # No human is involved and no second command is run.
The imperative version encodes an action; the declarative version encodes a state. Only the second one can be checked, and only something that can be checked can be repaired automatically.
Key points
- The founding question is not "how do I run containers" but "who notices when actual state differs from desired state, and restores it".
- Kubernetes is a control loop: store desired state, observe actual state, compute the gap, act on it — repeated forever rather than executed once.
- Controllers are many small loops, each responsible for one kind of gap: replica count, node loss, unscheduled workloads.
- Declarative means the declaration stays true, which is what makes automatic repair possible — and also why manual changes get reverted.
- The loop enforces your declaration exactly. A wrong declaration is enforced just as reliably as a right one.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Desired state is submitted to an API server and persisted in a consistent datastore; this is the only source of truth.
- • A node agent on every machine reports which containers are actually running and in what state.
- • Controllers watch both, compute the difference for their own object kind, and issue create or delete instructions.
- • The scheduler assigns newly created workloads to machines; the node agent then starts them and reports back.
- • The loop runs continuously, so the same mechanism that performs your deploy also performs the 03:00 repair you never see.
- • The datastore that holds desired state — its availability, its backups, and the fact that losing it loses the definition of correct.
- • The correctness of every declaration: replica counts, images, probes and resource requests are all numbers a human chose.
- • Reconciliation you did not intend: anything changed by hand outside the declaration will be reverted, so emergency fixes must be made in the declaration.
- • Understanding that the absence of alerts may mean the loop is healthy — or that the control plane is unreachable and nothing is being reconciled at all.
- • A crashing image: the loop restarts it with perfect discipline, producing an endless crash loop rather than an alert. The count is never satisfied, and the cluster is busy failing.
- • Control plane unavailable: existing containers keep serving traffic, but no gap is observed or closed. Failures accumulate silently.
- • Conflicting controllers or an operator fighting a human: a setting flaps back and forth, and the audit trail shows changes nobody admits to making.
- • Insufficient capacity: the gap is computed correctly and the replacement workload cannot be placed, so desired stays 10 and actual stays 8 indefinitely.
- • The loop cost is proportional to object count and change rate, not to request traffic — a quiet cluster with a hundred thousand objects is more expensive to reconcile than a busy one with a thousand.
- • Watch and list pressure on the API server is the usual first ceiling; noisy controllers and large object churn hit it before workload load does.
- • Reconciliation latency grows with queue depth, so during a large deploy the same loop that heals failures is busy performing your rollout.
- • The API server is the single authorization chokepoint for everything in the cluster; every controller, agent and workload authenticates to it.
- • The datastore holds desired state including secret objects, which is why its encryption at rest and its backups are a security concern and not only a reliability one — see ConfigMap vs Secret — and the Honest Limit of a Secret.
- • Anyone who can change desired state can change what runs, so write access to manifests is production access — treat the repository as a privileged system.
- • Controllers run with broad permissions by design; a compromised controller is a cluster-level compromise, not a workload-level one.
- • The control plane is fixed cost: it bills for existing, at the same rate whether it reconciles two workloads or two hundred.
- • Reconciliation itself consumes control-plane capacity, so high object churn — CI creating and deleting namespaces, for example — drives cost without serving a single user request.
- • The dominant cost remains the worker nodes; the loop's contribution to the bill is small, and its contribution to the operational budget is not.
- • Desired versus ready replica counts per workload — the direct read-out of the gap this whole system exists to close.
- • Controller work-queue depth and reconciliation errors, which tell you whether the loop is keeping up.
- • API server request latency and rejection rate; when this degrades, everything appears to be fine and nothing is being fixed.
- • The signal that lies: "all pods Running". Running is not Ready, and a pod can be Running while failing every request — see Liveness vs Readiness.
- • A process supervisor on a single machine — the same observe/diff/act loop with a scope of one host, and completely sufficient for one service.
- • A managed container service that runs a desired count for you without exposing controllers, schedulers or a datastore.
- • An autoscaling group of VMs with health checks: the cloud provider replaces unhealthy instances, which is the same loop at machine granularity — see One Big VM or Several Small Ones.
- • For a workload that is genuinely stateless and bursty, a serverless platform reconciles concurrency instead of replicas and you never see the loop at all.
- • Buys unattended repair and a single definition of correct; costs a persistent, consistent datastore that is now the most important thing you operate.
- • Buys the ability to describe the whole system as data; costs the loss of imperative control — you request state, you do not command actions.
- • Buys uniformity across every workload; costs a genuinely large vocabulary before an engineer can debug anything.
Desired 3, actual 2: what the controller does next
loop: observe ready 3 diff 3 desired − 3 ready = 0 act no action
What people believe, and what is true
Kubernetes is a YAML-based deployment tool.
YAML is the serialization format for the desired state. The system is a set of controllers reconciling that state continuously; the file format is incidental.
Once the deploy succeeds, Kubernetes is done.
There is no done. The loop that performed the deploy is the same loop that replaces a container at 03:00; it is running right now with nothing to do.
Self-healing means the application will recover.
It means the declared count will be restored. A container that crashes on a bad config will be restarted into the same crash indefinitely.