The Scheduler, and Why a Pod Is Pending
Placement is a filter-then-score decision made against declared requests. Pending is not a failure state — it is the scheduler telling you no node satisfied the constraints.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What decides which node a pod runs on, and what is the cluster telling me when a pod stays Pending?
Workloads have to be placed on machines that have room for them, satisfy their constraints, and are not about to be taken away — continuously, for thousands of pods, without a human choosing.
The cluster has plenty of free CPU and memory, so pods will land somewhere. If a pod is Pending, the cluster is out of capacity and needs more nodes.
The scheduler places against requests, not against current usage. A cluster whose nodes are idle can be completely unschedulable if the requests already committed add up to the allocatable total (Requests and Limits).
- The scheduler places against requests, not against current usage. A cluster whose nodes are idle can be completely unschedulable if the requests already committed add up to the allocatable total (Requests and Limits).
- Fit is per node, not per cluster. Ten nodes with a little room each cannot host one pod that needs a lot of room — free capacity does not pool.
- Most
Pendingpods are not a capacity problem at all: a node selector that matches nothing, a taint with no matching toleration, a volume bound to a zone the pod cannot reach, an exhausted namespace quota. - Adding nodes as a reflex is expensive and often fixes nothing, because the constraint that excluded every node is still there on the new one (Cost Drivers).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- The scheduler watches for pods with no
nodeName. For each one it runs two phases: filter, which removes every node that cannot run this pod, and score, which ranks the survivors. It then binds the pod to the winner by writingnodeName. - Filtering is boolean and unforgiving: insufficient allocatable CPU or memory for the pod's requests, unmatched node selector or affinity, an untolerated taint, a volume that cannot attach in that zone, node not
Ready, no free ports for host ports. - Scoring is preference: spread across nodes and zones, prefer nodes that already have the image, honour affinity and topology-spread preferences, balance resource use. Scoring never rescues a node that filtering removed.
- Allocatable is not the node's hardware total. The kubelet reserves capacity for itself and the system, so a node advertises less than it physically has (Building a Capacity Model).
- If every node is filtered out, the pod stays
Pendingand the scheduler emits aFailedSchedulingevent that names how many nodes were rejected and why. That message is the diagnosis, not a symptom. - A cluster autoscaler is a separate controller that watches for unschedulable pods and adds nodes. It can only help when the reason was capacity — it cannot satisfy a selector that matches no node group (Autoscaling).
Filter, score, bind
Placement is one decision made once per pod. Understanding it as two phases explains why most fixes fail: people tune the thing that ranks nodes when their problem is the thing that eliminated them.
- 1Observe
The scheduler sees a pod with no
nodeNamein its watch stream.fails by The scheduler itself is down — every new pod stays
Pendingcluster-wide.evidence Scheduler pods are running and their leader election is stable.
- 2Filter
Eliminates every node that cannot run the pod: requests, selectors, taints, volume topology, readiness.
fails by All nodes eliminated →
Pendingwith aFailedSchedulingevent naming the counts per reason.evidence At least one node survives; the event does not appear.
- 3Score
Ranks surviving nodes by spread, affinity preferences, image locality and resource balance.
fails by Never fails — with one survivor it is a formality.
evidence Placement matches your spread intent across nodes and zones.
- 4Bind
Writes
nodeNameonto the pod object.fails by A race where the chosen node filled up in between; the pod returns to the queue and is retried.
evidence Pod has a
nodeNameand the kubelet on that node has picked it up. - 5Kubelet takes over
Pulls the image, creates the container, starts reporting status.
fails by Image pull or startup failure — a different problem from scheduling (Apply Is Not Running).
evidence Container state leaves
Waiting.
Scheduling ends at bind. Everything after it is the kubelet, which is why "the pod is scheduled but not running" is a different investigation entirely.
Why no node passed the filter
Each of these produces a Pending pod and a FailedScheduling event, and they need completely different responses. The event message distinguishes them: it lists node counts per exclusion reason, and reading it is the fastest diagnosis in Kubernetes.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Requests exceed any node's free allocatable | Insufficient cpu / Insufficient memory for N nodes | Committed requests, not usage, fill the nodes | Lower the request if it was inflated, or add capacity if it was honest (Requests and Limits) |
| Node selector or required affinity | node(s) didn't match Pod's node affinity/selector | No node carries the label the pod insists on | Label a node group, or relax the rule — adding nodes will not help |
| Taint with no toleration | node(s) had untolerated taint | Nodes are reserved for other workloads, or are cordoned or unhealthy | Add the toleration if the pod belongs there; otherwise this is the taint working |
| Volume zone mismatch | node(s) had volume node affinity conflict | The bound volume exists in one zone and the pod cannot be placed elsewhere | Schedule into that zone, or use storage that is not zone-bound (Why Stateful Workloads Are Harder) |
| Strict anti-affinity or topology spread | Only some replicas schedule; the rest stay Pending | The spreading rule is DoNotSchedule and there are not enough distinct domains | Add domains, or make the constraint a preference rather than a requirement |
| Namespace resource quota exhausted | The pod is never created; the ReplicaSet reports a quota error | This is admission, not scheduling — the object was rejected upstream | Read the ReplicaSet events, not the pod list; the pod does not exist to be Pending |
| Cluster genuinely at capacity | Pending clears itself minutes later | The cluster autoscaler is adding a node | Nothing — unless the wait is longer than your rollout tolerates (How Autoscaling Fails) |
What to do about an unschedulable workload
Once the event has told you which filter excluded the nodes, there are four honest options and they have genuinely different costs. Reaching for capacity first is the expensive habit.
The event says every node was filtered out. What do you change?
when The request was set defensively rather than measured, and real usage is well below it.
cost Requests are also the workload's protection under contention; cut them too far and it is the first thing squeezed (How Resource Settings Go Wrong).
when The exclusion reason is genuinely insufficient allocatable capacity and requests are honest.
cost Direct, ongoing spend, and node startup time is part of your rollout latency (Cost Drivers).
when A selector, taint toleration or spread rule is excluding nodes that would be perfectly fine.
cost You are giving up whatever the constraint was protecting — usually failure-domain spread or workload isolation.
when The cluster is full of work that genuinely matters less, and priority classes reflect that.
cost Preemption stops another workload at a moment nobody chose; it is only safe if priorities were set deliberately.
How to do it properly
Most important first.
- Read the
FailedSchedulingevent before doing anything else. It states the reason per node group, and it is almost always the whole answer. - Set requests that reflect what the workload actually needs, because requests are the currency the scheduler spends. Unset requests make placement effectively blind (How Resource Settings Go Wrong).
- Use topology spread and anti-affinity deliberately so that a single node or zone failure cannot take every replica of a service (Reducing Blast Radius).
- Keep a deliberate amount of unallocated capacity so that a rolling update — which needs room for new pods before old ones go away — can actually schedule (Headroom).
- Prefer relaxing a constraint you added over adding nodes, once you have confirmed the constraint is what excluded them.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
A scheduling constraint is per workload, so a bad one usually strands one Deployment while everything else runs. It becomes zone-wide when the constraint is in a shared template or when a zone loses capacity and every pod pinned there is unschedulable at once. Existing pods keep running throughout — this failure blocks change rather than stopping service.
What can go wrong
- Requests inflated "to be safe", so the cluster runs out of schedulable room long before it runs out of real capacity, and the cost of the fleet rises with nothing to show for it.
- Anti-affinity strict enough that replicas cannot be placed at all — the rule intended to spread the workload prevents it running.
- A rolling update deadlocked: no room for a new pod, and the strategy will not remove an old one first.
- A cluster autoscaler that adds nodes indefinitely because pods are unschedulable for a reason nodes cannot fix.
- A pod pinned by a volume to one zone, on a day that zone has no capacity — the workload is unschedulable and looks like a Kubernetes problem rather than an availability-zone one.
- Preemption evicting a lower-priority workload you had forgotten was important, to make room for a higher-priority one.
- "
Pendingmeans the cluster is full." It means no node passed the filters. Capacity is one of at least six reasons, and not the most common one. - "The nodes are idle, so there is capacity." Idle is usage; the scheduler spends requests. The two are related only by how well the requests were chosen.
- "Autoscaling solves scheduling failures." It solves one class of them, and it will happily add nodes forever for the classes it cannot solve (How Autoscaling Fails).
- "The scheduler balances load." It balances requests at placement time and then never revisits the decision. Nothing rebalances a node that became hot afterwards.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- Pods have a
nodeNameand leftPendingwithin the time you consider normal for the cluster. FailedSchedulingevents are absent, or explained and expected.- Replicas of a service are distributed across the failure domains you intended — checked by listing pods with their nodes and zones, not assumed from the manifest.
- Allocated requests against allocatable capacity per node shows the headroom you planned for, rather than a number nobody has looked at since the cluster was built.
- Placement decisions are not directly reversible — a bound pod stays where it is. You change future placement by changing the spec, which means new pods.
- To move a workload, change desired state and let the loop replace pods. Deleting a pod to force rescheduling works but gives up the rollout protections you would normally have.
- Draining a node evicts its pods for rescheduling elsewhere, and will stall if a PodDisruptionBudget forbids the eviction — which is the budget doing its job, not a bug.
- Automate the fleet-level response: cluster autoscaling on unschedulable pods, with a bound on the maximum size so a scheduling bug cannot become an unbounded bill.
- Keep placement policy human. Affinity, spread and priority encode which workloads matter and which failures you are willing to accept — that is an ownership decision, not a tuning parameter (Blast Radius: If This Is Wrong, How Much Does It Affect?).
- Tight requests raise utilisation and reduce the room available when a node fails or a rollout needs to place new pods first.
- Strong spreading improves failure tolerance and lowers density, which costs money on every node you now cannot pack.
- Priority and preemption keep critical workloads schedulable, at the price of evicting something else at a moment nobody chose.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- KUBERNETES-SPECIFICPer-pod filter-and-score placement against declared requests is Kubernetes. A VM autoscaling group has no per-workload placement at all — it asks the cloud for an instance of a fixed size and either gets one or gets a capacity error, so the equivalent failure surfaces as a launch failure in the cloud provider rather than a
Pendingobject. A PaaS places for you and reports "no capacity" as a deploy failure with no visibility into why. - SIMPLIFIEDFilter and score are described as two phases. Real scheduling has more structure — extension points, preemption and pluggable scoring — but no amount of scoring recovers a node that filtering removed, which is the part that matters operationally.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Distributed Systems — placement as a constraint-satisfaction problem, and why bin packing and failure-domain spread pull in opposite directions.