Kubernetes Runtime

Reconciliation as the core mental model, plus the runtime behaviours that produce most real incidents: scheduling, requests and limits, OOM kills, throttling and probes.

Reconciliation: The Loop Under Everything

You write desired state; a controller observes actual state; the difference is the instruction. That loop never stops running, which is the whole idea.

Q · What is actually happening between the moment you apply a manifest and the moment the cluster matches it — and why does it keep happening afterwards?
Apply Is Not Running

Accepted, scheduled, pulled, started, ready and receiving traffic are six different moments, separated in time — so desired state is never instant reality.

Q · Why does `kubectl apply` return success while nothing new is serving yet, and what is happening in the gap?
The Scheduler, and Why a Pod Is Pending

Placement is a filter-then-score decision made against declared requests. `Pending` is not a failure state — it is the scheduler telling you no node satisfied the constraints.

Q · What decides which node a pod runs on, and what is the cluster telling me when a pod stays `Pending`?
Requests and Limits

A request is a scheduling reservation; a limit is an enforced ceiling. CPU and memory behave completely differently when you reach the ceiling, and that difference is the lesson.

Q · What do requests and limits actually control, and why does exceeding a CPU limit feel nothing like exceeding a memory limit?
How Resource Settings Go Wrong

Four failure shapes come from two numbers being wrong in two directions each — and each shape has a distinct symptom that tells you which one you are looking at.

Q · My workload is unhealthy and the code did not change. Which resource number is wrong, and in which direction?
OOMKilled: Over the Memory Limit

Memory cannot be taken back, so the only enforcement available is termination. Over the limit, the kernel kills the process — it does not slow it down or warn it.

Q · What exactly happens when a container reaches its memory limit, and why does it die rather than degrade?
CPU Throttling: The Latency With No Error

Over a CPU limit the container is descheduled until the next period rather than killed. Nothing errors, nothing restarts, and the tail latency gets worse for reasons nothing in the application explains.

Q · Why is my service slow when its CPU graph looks unremarkable and nothing in the logs is wrong?
Probes: Readiness, Liveness and Startup

Readiness gates traffic, liveness restarts the container, startup covers a slow boot. Confusing the first two turns a dependency outage into a cluster-wide restart storm.

Q · What is each probe actually allowed to decide, and what must never be inside a liveness check?
Reading a Broken Workload

One decision tree covers most Kubernetes failures: is it running, is it ready, is it routed — and each "no" points at a different, small set of causes.

Q · The service is not serving. Where do I look first, and what does each state actually mean?