The question this answers
What do I actually promise when I write requests and limits, and why do CPU and memory behave so differently at the boundary?
The API needs enough memory to serve its p99 request without being killed, enough CPU to keep latency stable under burst, and it must not be able to starve the workloads sharing its node.
A capacity reservation the scheduler honours when placing the pod, and an enforced upper boundary that protects the node and its other tenants from any single container.
Two numbers, two jobs
A request is a scheduling statement: "reserve this much for me on whichever node you pick." It is subtracted from the node's allocatable capacity whether or not the container ever uses it, and it is the only number the scheduler looks at. A limit is a runtime statement: "never let this container exceed this." It is enforced by the kernel through cgroups, and the scheduler does not consider it at all.
The gap between them is deliberate. Request low and limit high, and you get a container that packs efficiently and can burst — at the risk of a node where everyone bursts at once. Set them equal and you get predictable, isolated behaviour at the cost of paying for the peak continuously. Omit them entirely and the scheduler places the pod blind, the kernel imposes no ceiling, and the pod becomes the first candidate for eviction when the node runs short.
Kubernetes formalizes that spectrum as quality-of-service classes, and the class is derived, not declared. Requests equal to limits on every container is Guaranteed — the last to be evicted. Requests set and lower than limits is Burstable. Nothing set at all is BestEffort — the first to be evicted when a node is under memory pressure, which is exactly the wrong outcome for a workload someone forgot to configure rather than deliberately deprioritized.
| CPU | Memory | |
|---|---|---|
| Kind of resource | Compressible — it can be given less | Incompressible — it cannot be taken back |
What requests does | Reserves scheduling capacity; sets a relative share under contention | Reserves scheduling capacity |
What limits does | Caps usage per scheduling period via a quota | Sets a hard ceiling enforced by the kernel |
| At the limit | The process is throttled — it waits. Nothing dies. | The process is killed. OOMKilled, exit code 137. |
| Symptom when exceeded | Latency rises. No restarts. CPU graph looks fine. | Sudden restart. Restart count climbs. |
| Setting it too low | Silent, continuous latency damage | Repeated kills under normal load |
| Setting it too high | Weaker isolation between tenants on the node | Wasted reservation; poor bin-packing; a leak takes the node down with it |
| Common guidance | Set the request; think hard before setting a limit | Always set a limit, and set it equal to the request |
Why CPU and memory need opposite instincts
The asymmetry is not a Kubernetes design quirk; it comes from the resources themselves. CPU is compressible: if a process wants more cycles than it may have, the kernel simply gives it fewer and it runs slower. Nothing is lost, nothing dies, and the process is not even told. Memory is incompressible: if a process wants a page it cannot have, there is no "slower" — the allocation must either succeed or the process must die. The kernel chooses the latter.
That leads to genuinely opposite advice for the two fields. For memory, set the limit and set it equal to the request. Equal values buy the Guaranteed class and predictable eviction behaviour, and a memory limit is the only thing standing between a leak in one container and every other pod on the node being evicted. For CPU, set the request — it is what gets you scheduled and what determines your share under contention — and be genuinely skeptical about the limit.
The CPU limit is the most commonly harmful field in a Kubernetes manifest. It caps a container even when the node is completely idle, so a service that could have absorbed a burst in 30 ms takes 300 ms instead, for no benefit to anyone. And because throttling produces no restart, no error and no elevated CPU graph, it is almost never diagnosed correctly — see OOM Kills and CPU Throttling. There are real reasons to set one — hard multi-tenancy, a noisy-neighbour incident you can actually point to, a batch job that must not crowd out serving traffic — but "we set limits on everything" is a policy, not a reason.
# 1. Nothing set: BestEffort. Placed blind, evicted first, no ceiling on a leak.
resources: {}
# 2. Copied from a template nobody measured.
resources:
requests: { cpu: 100m, memory: 128Mi } # 8x below real usage -> scheduler over-packs the node
limits: { cpu: 200m, memory: 2Gi } # CPU capped at 200m: throttled during every burst
# memory ceiling 16x the request: a leak evicts neighbours
# 3. "Safe" values chosen by rounding up until the alerts stopped.
resources:
requests: { cpu: 4, memory: 8Gi } # reserves half a node to run a process using 300m/700Mi
limits: { cpu: 4, memory: 8Gi } # bin-packing destroyed; cluster cost roughly 10x# Measured over two weeks including the Monday peak:
# memory: steady 640Mi, p99 810Mi, no growth trend
# cpu: median 180m, p95 520m, bursts to ~1.4 during cache refresh
resources:
requests:
cpu: 500m # near p95: scheduled with enough share to be stable under contention
memory: 1Gi # above p99 with headroom; equal to the limit below
limits:
memory: 1Gi # equal to request -> Guaranteed class, predictable eviction, leak contained
# cpu: deliberately unset. The node has idle capacity during the burst; using it is free
# and capping it would turn a 30ms refresh into a 300ms one for nobody's benefit.
# Revisit only if a specific noisy-neighbour incident justifies it.Requests are a measurement, not a guess: too low and the scheduler over-packs the node, too high and you pay for a reservation nobody uses. The memory limit is protection. The CPU limit is a constraint you should have to justify.
Choosing the numbers without guessing
Both numbers should come from measurement, and the measurement must include the peak — a request derived from a quiet Tuesday guarantees a bad Monday. The method that holds up: run the workload with generous values and no CPU limit, observe for a full business cycle, then set the memory request and limit somewhat above observed p99, and the CPU request near observed p95.
Two warnings about right-sizing that apply directly here. First, the peak you must survive is not the average traffic peak but the failover peak: if you run three replicas across three zones and one zone fails, the survivors take 50% more load each. Size for that, or your zone-redundancy design collapses the first time it is used — see Right-Sizing Without Causing an Outage. Second, resize gradually and watch, because a memory request reduction that turns out to be five percent too aggressive produces OOM kills under exactly the load you cannot afford them.
The numbers are also not permanent. Applications change, dependencies change, and a request set eighteen months ago describes a version of the service that no longer exists. Reviewing them periodically is the single highest-return cost exercise in a cluster, because over-requesting is the most common form of cloud overspend and it is invisible on every utilization dashboard.
1# What the scheduler sees: requests already committed on each node.2kubectl describe node node-04 | sed -n '/Allocated resources/,/Events/p'3 4# What is actually being used, per pod, right now.5kubectl top pods -n production --sort-by=memory6 7# The four series that decide the numbers (metric names vary by collector):8# memory working set, p99 over 14 days ....... -> memory request AND limit, with headroom9# cpu usage, p95 over 14 days ................ -> cpu request10# throttled seconds / total periods ........... -> is an existing cpu limit hurting you?11# OOM kill events by container ................ -> is the memory limit too low?12 13# The cluster-wide question worth asking monthly:14# sum(requests) / sum(allocatable) -> what you are paying for15# sum(actual usage) / sum(allocatable) -> what you are using16# A large gap between those two is over-requesting, and it is invisible on a usage dashboard.Key points
- A request is a scheduling reservation and the only number the scheduler reads; a limit is a kernel-enforced runtime ceiling the scheduler ignores.
- CPU is compressible: exceeding the limit throttles the process. Memory is incompressible: exceeding the limit kills it.
- Set the memory limit, equal to the request. That buys the Guaranteed class and contains a leak to the container that has it.
- Be skeptical of CPU limits: they cap a container even on an idle node, and the damage is invisible latency rather than an error.
- Derive both from measurement over a full business cycle, and size for the failover peak, not the average one.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • Requests are summed per node and subtracted from allocatable capacity; the scheduler filters nodes on the remainder.
- • CPU requests are translated into a relative share weight, so under contention a container gets cycles in proportion to its request.
- • CPU limits become a quota per scheduling period; once the quota is consumed the container waits until the next period.
- • Memory limits become a cgroup memory ceiling; an allocation that would exceed it triggers the kernel OOM killer for that cgroup.
- • The relationship between requests and limits across all containers derives the pod's QoS class, which determines eviction order under node pressure.
- • Measuring usage per workload over a full business cycle, and revisiting the numbers as the application changes.
- • Namespace-level defaults and ranges so a workload that forgets to set anything does not land in BestEffort.
- • Alerting on throttling and on OOM kills separately, because they are different problems with opposite fixes.
- • Tracking cluster-wide requested versus used capacity, which is where over-requesting becomes visible as money.
- • Sizing for failover load, not steady-state load, when replicas are spread across zones.
- • Memory limit slightly too low: the container is OOM-killed under peak load, exactly when the traffic matters most.
- • CPU limit too low: p99 latency triples during bursts with no restarts, no errors and a CPU graph that looks unremarkable.
- • Requests far below real usage: the scheduler over-packs the node, and everything on it degrades together under load.
- • Requests far above real usage: bin-packing collapses, the cluster runs at low utilization, and the bill is several times what it needs to be.
- • No values at all: BestEffort class, evicted first under node pressure, usually the workload nobody meant to deprioritize.
- • Cluster cost scales with the sum of requests, not the sum of usage, so accuracy of requests is the primary cost lever.
- • As replica count grows, a request that is 200 MiB too generous becomes tens of gigabytes of reserved, unused memory.
- • Horizontal scaling based on utilization is computed against requests, so wrong requests produce wrong autoscaling — see Horizontal Pod Autoscaling — and Why New Capacity Is Always Late.
- • Limits are a denial-of-service control between tenants on a node: without a memory limit, one container can evict everything else on the machine.
- • A missing memory limit turns an application memory-exhaustion bug into a node-level incident, which is a real availability attack surface.
- • Namespace resource quotas prevent one team from consuming a shared cluster's capacity, deliberately or otherwise.
- • For genuinely untrusted workloads, limits are not sufficient isolation — separate node pools or separate clusters are the boundary.
- • You pay for requests, whether or not the capacity is used. This is the single most important cost sentence in the module.
- • The gap between total requested and total actually used is the cluster's waste, and it is invisible on a utilization dashboard.
- • Rightsizing requests routinely returns more money than any other cluster optimization, and it requires no architectural change.
- • Over-provisioning to avoid OOM kills is a real cost trade — cheaper than an outage, more expensive than measuring.
- • Requested versus used, per workload and cluster-wide — the direct measure of what you are paying for versus getting.
- • CPU throttling as a ratio of throttled periods to total periods, which is the only reliable way to see a CPU limit hurting you.
- • OOM kill events by container, separated from ordinary restarts.
- • Memory working set trend over days, which distinguishes a leak from a workload that simply needs more.
- • The signal that lies: average CPU utilization. A container throttled hard against a low limit shows modest average CPU while its latency triples.
- • A compute model where you pick an instance size rather than two numbers per container — a VM or a managed container platform hides this entirely.
- • Serverless, where memory is the one dial and CPU is allocated proportionally, removing the whole question.
- • Vertical autoscaling to recommend or apply values from observed usage — useful for recommendations, careful with automatic application to stateful workloads.
- • Namespace defaults and ranges, so most workloads inherit sane values and only the unusual ones need a decision.
- • Buys predictable placement and tenant isolation; costs the accuracy of two numbers per container that nobody wants to maintain.
- • Buys leak containment via memory limits; costs restarts when the limit is even slightly too low.
- • Buys noisy-neighbour protection via CPU limits; costs burst capacity that was free and latency that nobody will diagnose correctly.
- • Buys efficiency through tight requests; costs the risk that the failover peak was never in the measurement window.
Throttled or OOMKilled: the two outcomes
$ kubectl get pod api-7f4-a1 NAME READY STATUS RESTARTS AGE api-7f4-a1 1/1 Running 0 12m container_cpu_cfs_throttled_seconds_total rising (58% of periods) container_memory_working_set_bytes 380Mi of 512Mi limit
What people believe, and what is true
The limit is what the pod gets.
The request is what is reserved for scheduling. The limit is a ceiling that may never be reached, and the scheduler ignores it entirely.
Setting CPU limits everywhere is a best practice.
A CPU limit throttles a container even on a completely idle node. Set it when you can name the noisy-neighbour problem it solves; otherwise the request is what matters.
Exceeding a limit restarts the pod.
Only for memory. Exceeding a CPU limit makes the container wait — no restart, no error, just latency you will misdiagnose as a network or database problem.