Scheduling: How a Pod Chooses a Node

The scheduler watches for pods with no node assigned and runs a two-phase decision for each. Filter removes every node that cannot host this pod: not enough *allocatable* CPU or memory left after existing requests, a taint the pod does not tolerate, a node selector or affinity rule that does not match, a required volume that lives in another zone, a host port already taken. Score ranks the survivors — spreading across zones and nodes, preferring nodes that already have the image, balancing resource usage. Then it binds: writes the chosen node onto the pod, and the node agent takes over.

The word doing the work in the filter phase is requests. The scheduler does not look at how much memory a node is currently *using*; it looks at the sum of the requests of the pods already assigned to it. A node running at 20% actual memory can be completely unschedulable because the pods on it requested everything. Conversely, a node can be at 95% real memory pressure and still accept pods, because those pods requested very little. Understanding that one indirection resolves the majority of confusing scheduling behaviour — and it is why Requests vs Limits: Two Numbers That Do Different Jobs is the next lesson.

It is also worth knowing what the scheduler will not do. It does not move running pods to improve packing; placement is decided once, at bind time. If your cluster becomes badly balanced, it stays badly balanced until something restarts the pods. The scheduler is a placement decision, not a continuous optimizer.

A pod stuck in Pending is the most common Kubernetes support question and one of the easiest to answer, because the scheduler writes down exactly which filter eliminated how many nodes. The message reads like a tally: so many nodes failed on insufficient memory, so many on a taint, so many on affinity. Every line names a different fix.

The reflex to resist is deleting the pod. Its replacement will be evaluated by the same scheduler against the same nodes with the same constraints, and will be equally unplaceable — you have simply lost the events that explained why. Read the events first.

The second reflex to resist is assuming a capacity problem. In the example below, three separate causes are stacked: a genuine capacity shortfall, a taint that was doing its job, and a topology spread rule that the cluster could not satisfy. Adding nodes fixes only the first, and if the new nodes carry the same taint it fixes nothing at all.

Beyond resources there are four placement tools, and they are best learned in order of how often they are genuinely needed. Topology spread constraints are the one nearly every production workload should use: they say "keep my replicas on different nodes, or in different zones", which is what turns replicas: 3 into actual redundancy. Node selectors pin a workload to a labelled class of machine, which is the simple way to say "this needs GPU nodes".

Taints and tolerations are the inverse and the one people invert in their heads. A taint is on the *node* and repels pods; a toleration is on the *pod* and permits it to be placed there anyway. Crucially, a toleration does not attract — it only removes an objection. So "GPU nodes should only run GPU work" needs both: a taint so ordinary pods stay off, and a node selector so GPU pods actually go there. Using only a toleration produces the exact failure of a batch job scheduled onto expensive accelerators to do nothing.

Affinity and anti-affinity are the most expressive and the most abused. Required (hard) rules make pods unschedulable when they cannot be satisfied, which is usually not what the author intended; preferred (soft) rules degrade gracefully. Pod anti-affinity across a large cluster is also genuinely expensive to evaluate. The honest guidance: use topology spread for redundancy, node selectors plus taints for specialized hardware, and reach for affinity only when you can name the specific placement that must not happen.