Cluster, Control Plane, Nodes, Pods
One model to debug against: a cluster is a control plane holding desired state and nodes running the workloads, with controllers continuously closing the gap between them.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What are the moving parts, and which one do I look at when something is wrong?
kubectl apply returns success and nothing runs. Without a model of who does what after that command returns, every such situation becomes guesswork across a dozen components.
Treat the cluster as an API that deploys things: you send a manifest, it deploys. Success from kubectl apply therefore means the workload is running.
kubectl apply returns as soon as the API server has persisted your desired state. It says nothing about whether anything was scheduled, pulled, started or became healthy.
kubectl applyreturns as soon as the API server has persisted your desired state. It says nothing about whether anything was scheduled, pulled, started or became healthy.- Every interesting failure happens after that return: no node has capacity, the image cannot be pulled, the container crashes on startup, the readiness probe never passes.
- Without the model, the wrong thing gets debugged. Teams read application logs for twenty minutes on a pod that was never scheduled and therefore has no logs.
- It hides the asynchrony that defines the platform. Desired state and actual state are allowed to differ, and the gap is normal (Apply Is Not Running).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A cluster is a set of machines presented as one pool, plus the components that manage them.
- The control plane holds the desired state and decides. The API server is the only front door and the only writer to storage; etcd persists the state; the scheduler assigns unscheduled pods to nodes; controllers watch objects and act to close gaps.
- A node is a machine that runs workloads. Its kubelet watches for pods assigned to it, tells the container runtime to start them, runs their probes and reports status back. A network proxy component programs the local rules that make Service addresses work.
- A pod is the unit the scheduler places: one or more containers that share a network namespace and can share volumes (Pods: The Unit That Gets Scheduled).
- The flow of an apply is therefore: your manifest becomes persisted desired state, a controller creates dependent objects, the scheduler picks a node, that node's kubelet starts containers, and status flows back up. Each hop can stall, and each stall has a different symptom.
The parts, and what each is responsible for
Four layers, each with one job. When something is wrong, the useful first move is naming which layer you are questioning — most confusion is caused by asking a node-level question of a control-plane component or the reverse.
What actually happens after apply returns
This sequence is the reason "it says it deployed" is not an answer. Each step is performed by a different component, at its own pace, and each one has a characteristic way of stalling.
Learning this pipeline converts most cluster debugging into a linear search: find the first step that did not complete, and look at whatever that step depends on.
- 1API server accepts
Authenticates, validates, admits and persists the object. This is when your command returns.
fails by Rejected by validation or an admission policy — a fast, loud, honest failure (Policy as Code).
evidence The object exists when you read it back.
- 2Controller creates dependents
The Deployment controller creates a ReplicaSet; the ReplicaSet controller creates Pods (ReplicaSets: The Layer You Should Not Manage).
fails by Quota exhausted, so pods are never created and the ReplicaSet reports a failure condition.
evidence Pod objects exist with the expected owner reference.
- 3Scheduler assigns a node
Finds a node with enough allocatable CPU and memory for the pod's requests.
fails by Pod stays Pending with an "insufficient resources" event — a capacity problem, not an application problem (The Scheduler, and Why a Pod Is Pending).
evidence The pod has a node name.
- 4kubelet pulls and starts
Pulls the image, sets up the pod sandbox, starts containers.
fails by ImagePullBackOff from a bad tag, a missing registry credential or a rate limit (Artifact Registries).
evidence Container state is Running.
- 5Probes decide readiness
Runs the readiness probe and reports the result up.
fails by Running but never Ready, usually because readiness depends on something that is down (Probes: Readiness, Liveness and Startup).
evidence The pod is Ready and appears in the Service's endpoints.
- 6Traffic arrives
The Service's endpoint set includes the pod; node-local rules route to it.
fails by Label selector does not match, so the Service has no endpoints and callers get connection refused (Services: A Stable Address Over Moving Pods).
evidence Endpoints list the pod IP and requests reach it.
Declared, not commanded
The difference between telling a system what to do and telling it what should be true is the whole design. It is also why the platform can recover from failures nobody scripted: the desired state is still there after the failure, and the controller is still comparing against it.
The cost is that your commands are not the source of truth — the stored object is. Anything you do imperatively that is not written back into the manifest will be quietly undone, or worse, quietly kept, and neither is discoverable later.
ssh to each host, stop the old container, pull the new image, start it, move on. Repeat on failure. On a node dying, do it again by hand somewhere else.
Record that three replicas of image@sha256:… should exist. A controller compares that against what exists and acts — on the initial apply, after a crash, and after a node dies, using the same code path each time.
Recovery is not a separate procedure in the declarative model; it is the same loop running again. That is what makes it survive failures nobody anticipated — and why an out-of-band kubectl edit is so damaging: it changes reality without changing the thing reality is compared against (Manual Production Changes).
How to do it properly
Most important first.
- Debug top-down through the model. Does the object exist? Was a pod created? Was it scheduled? Did the image pull? Did the container start? Did it become ready? Each question has a different answer source (Reading a Broken Workload).
- Read events before logs. Scheduling failures, image pull failures and probe failures appear as events on the object, and a pod that never started has nothing in its logs to read.
- Treat the API server as the source of truth for intent and the node as the source of truth for reality. When they disagree, the disagreement is the finding.
- Remember that everything is asynchronous. "Applied" means recorded, not running; that distinction is the single most useful thing in this lesson.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A misunderstanding here contains itself to the workload being changed; the genuine multi-zone risk is a control plane outage during a rollout, which is contained by workloads continuing to run without it.
What can go wrong
- Control plane unavailable: existing workloads keep running because kubelets already have their assignments, but nothing new gets scheduled and nothing reconciles. Symptoms look like "deploys hang" rather than "site down".
- A node goes unready. Its pods are eventually rescheduled elsewhere, which is fine if there is spare capacity and an outage if there is not (Headroom).
- A workload that is running fine while its desired state has drifted, because someone edited the live object by hand (Manual Production Changes).
- Status that is stale rather than wrong — the object says Ready and the process is wedged, because readiness is whatever the probe measures (Probes: Readiness, Liveness and Startup).
- "The control plane being down means the site is down." Usually not. Running pods keep running and Services keep working; what stops is change and recovery.
- "
kubectl applysucceeded, so it is deployed." It means the API server accepted and stored your intent. - "Pods run on the control plane." Workload pods run on nodes; the control plane's job is to decide, not to serve your traffic.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- You can answer "is this running?" from the pod status and node assignment, not from whether the apply command succeeded.
- Events for a failing workload name the stage that failed, and it matches the symptom.
- A node drain moves workloads without a user-visible error, which demonstrates that the loop actually works in your cluster.
- Because desired state is the input, rolling back means changing desired state back — reapplying the previous manifest, or using the workload controller's own revision history (Rollback: Only Useful If It Is Actually Safe).
- What you cannot roll back by reapplying: anything the change did outside the cluster, most obviously a database migration (Why Migrations Are the Dangerous Change).
- Automate the path from a versioned manifest to applied desired state, so the cluster reflects a reviewed repository rather than accumulated commands (Immutable Infrastructure).
- Automate the check that applied state matches the repository, since drift is invisible by construction (Drift).
- Keep the interpretation human. A controller can restore desired state; it cannot decide whether the desired state was right.
- The asynchronous model makes the system self-healing and makes "did my change work?" a genuinely harder question than on a platform that deploys synchronously and tells you.
- Presenting many machines as one pool removes placement work and removes visibility into where things run — which matters when a specific node is the problem.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICThe control-plane/node split with a single API server front door is Kubernetes' architecture. On a PaaS the equivalent decision layer is invisible and you get a deploy status instead; on plain VMs there is no decision layer and placement is yours.
- SIMPLIFIEDThe control plane also runs admission, authentication, garbage collection and dozens of built-in controllers. This model keeps the four parts you debug against and omits the rest deliberately.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.