The question this answers
We are paying for 100 units of capacity and using 12 — how much of that gap is waste, and how much is keeping us up?
The platform must absorb a five-fold traffic spike when the marketing email goes out, must keep serving when one availability zone fails, and must not spend money on capacity that serves neither purpose. Those three requirements pull in opposite directions and the gap between allocated and used is where they meet.
A decomposition of unused capacity into named buckets — failover reserve, burst headroom, scheduling overhead, and genuine waste — so that a reduction targets the last one and leaves the first three intact.
The number that starts every argument
Open any cluster dashboard and the same picture appears: a large allocated total, a much smaller used total, and someone in a meeting asking why the company is paying for the difference. It is a fair question with a bad default answer. The dump below is that picture — 100 vCPU allocated across a cluster, 12 in use at the moment of measurement — and reading it correctly requires two things the headline number does not contain.
The first is *when* it was measured. Utilization at 03:00 and utilization at the daily peak are different facts, and a decision made from the average of the two is a decision made from a number that describes no actual moment. The second is *what the capacity is reserved against*. Kubernetes requests, autoscaling group minimums and database standby instances are all allocations made deliberately, for reasons, by people who may no longer be on the team.
Note the last two lines of the dump. Peak utilization was 61%, not 12%, and a single-zone failure would push the survivors to 91%. Both numbers change the conversation entirely. The instantaneous 12% was real and it was the least useful thing on the screen.
cluster capacity 100 vCPU 400 GiB requested by workloads 74 vCPU 300 GiB (what the scheduler reserved) actually used right now 12 vCPU 96 GiB (what the processes consume) headline reading: "88 idle vCPU — we are wasting 88% of the cluster" scheduler reading: "26 vCPU unschedulable — we cannot fit another replica" usage over the last 14 days p50 14 vCPU p95 47 vCPU peak 61 vCPU <- daily digest send, every day at 09:00 zones 3, evenly balanced if one zone fails, the surviving two absorb everything: peak on survivors 61 x 3/2 = 91 vCPU -> 91% of cluster capacity verdict: the gap between requested (74) and used (12) is a requests-accuracy problem. The gap between used (12) and capacity (100) is mostly the reliability budget, and it is nearly spent at peak.
Four buckets, only one of which is waste
Unused capacity divides into four categories and they have completely different verdicts. Failover reserve is the capacity that absorbs traffic when a zone or an instance dies. If you run across three zones and any one may fail, you are deliberately buying roughly 50% more capacity than steady-state peak requires, and that is not waste — it is the price of Multi-Zone Deployment stated in units. Burst headroom is the gap between current load and what the system must absorb before new capacity arrives; its size is a function of how fast you can scale, which is why Startup Time & Cold Start logic matters here. If instances take four minutes to become ready, you need four minutes of traffic growth already provisioned.
Scheduling overhead is the capacity that exists but cannot be used, because bin-packing never fits perfectly: a node with 300 millicores free cannot host a pod requesting 500, so that capacity is stranded. Larger nodes strand less proportionally and concentrate more failure per node; smaller nodes strand more and fail more gracefully. This bucket is reducible through better sizing and never to zero.
Genuine waste is what is left: over-stated resource requests that reserve capacity nothing ever uses, environments running outside working hours, instances left behind by a migration, a scale-in policy so conservative that capacity added for one spike is still running a week later. This is the bucket to cut, and it is usually large enough to be worth the work without touching anything load-bearing.
| Bucket | Why it exists | Verdict | How to size it honestly |
|---|---|---|---|
| Failover reserve | A zone or instance failure moves its load onto the survivors | Keep — this is the reliability requirement, priced | Peak load × (zones ÷ (zones − 1)). Verify by draining a zone in a test. |
| Burst headroom | New capacity takes minutes; traffic does not wait | Keep, sized to your actual scaling latency | Measure time-to-ready, then provision for the traffic growth that occurs in that window |
| Scheduling overhead | Bin-packing strands fragments too small to place | Reduce, never eliminate | Right-size node shapes against pod shapes; watch unschedulable events |
| Over-stated requests | A request copied from another service, or padded for safety | Cut — this reserves capacity nothing ever consumes | Set requests from observed p95 with a margin; see Requests vs Limits: Two Numbers That Do Different Jobs |
| Off-hours environments | Non-production running 168 hours to serve about 40 | Cut — the largest easy saving in most organizations | Scheduled shutdown; make restart fast enough that nobody works around it |
| Scale-in that never happens | Conservative cooldowns, or a policy that only scales out | Cut carefully | Verify capacity added for a spike is released after it, and alarm if it is not |
| Abandoned resources | Migrations, experiments, forgotten test clusters | Cut — free money | Tag at creation and report untagged and idle resources weekly |
What cutting too far actually costs
Utilization targets are seductive because they are easy to measure and easy to put in a slide. "Get the cluster to 80%" sounds like operational maturity. It is also, applied naively, a description of a system with no room to absorb anything — and the failure it produces is not gradual. Capacity runs out, requests queue, latency rises, health checks start failing, the platform kills instances it believes are unhealthy, and the remaining capacity absorbs their share too. The bill went down 30% and the incident cost more than the year's saving.
So the sequence matters. Cut waste first — over-stated requests, off-hours environments, abandoned resources, broken scale-in — because none of that reduces the system's ability to survive anything. Only then consider reducing failover reserve or burst headroom, and only with an explicit statement of what you are giving up: "we accept degraded service for the four minutes a zone failure takes to be absorbed" is a decision; silently trimming headroom is not.
The cost panel below prices the same cluster with the buckets separated. The useful thing about the separation is that it makes the conversation specific. "We are wasting 88%" is unarguable and unactionable. "23% of capacity is over-stated requests and off-hours environments, and 40% is failover and burst reserve that the zone-failure requirement demands" is a conversation where somebody can decide something.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- The gap between allocated and used splits into failover reserve, burst headroom, scheduling overhead and genuine waste. Only the last is free to cut.
- Instantaneous utilization is the least useful number available. Peak, p95 and post-failover projected utilization are the ones that decide anything.
- Reserved-but-unused capacity from over-stated requests looks identical to headroom on a dashboard and protects nothing.
- Burst headroom is a function of scaling latency: faster startup buys back the capacity you were holding to cover the gap.
- Cut waste before reserve, and state explicitly what reliability you are giving up if you touch reserve at all.
- A cluster at a 90% utilization target has no room to absorb a zone failure, and the failure mode is a cascade, not a slowdown.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • The scheduler reserves capacity based on declared requests, not on observed usage, so a workload requesting 1 vCPU and using 0.1 removes a full vCPU from the schedulable pool.
- • Autoscaling groups hold a configured minimum regardless of demand, which is a deliberate floor that outlives the reason it was set.
- • Bin-packing strands fragments: capacity smaller than the smallest pending workload cannot be allocated to anything.
- • Failover reserve is implicit — it is whatever spare capacity happens to exist when a zone disappears, which is why it must be sized on purpose rather than discovered.
- • Billing meters allocated capacity, not consumed capacity, for every fixed-shape resource, which is why the gap costs full rate.
- • Measure utilization at peak, at p95 and after a simulated zone loss. Never make a capacity decision from a single instantaneous reading.
- • Set resource requests from observed p95 plus a stated margin, and re-derive them quarterly as the workload changes.
- • Test the failover assumption by actually draining a zone in a controlled window. An untested reserve is an estimate.
- • Schedule non-production environments off, and make the restart fast enough that people do not disable the schedule.
- • Alarm on capacity added by a scale-out event that is still present days later — that is a scale-in policy failure with a monthly cost.
- • Track the four buckets separately so the cost conversation stays specific.
- • Utilization target applied bluntly: headroom removed, a spike arrives, requests queue, health checks fail, the platform kills instances, load concentrates on the rest, and the cascade completes.
- • Failover reserve consumed by an unrelated workload that was scheduled into it, discovered during the zone failure it was meant to cover.
- • Requests tuned down to observed averages, so the workload is throttled or OOM-killed at peak — see OOM Kills and CPU Throttling.
- • Scheduling deadlock: plenty of total free capacity, no single node with a contiguous free block large enough, so pods stay pending while the dashboard shows 40% idle.
- • Off-hours shutdown applied to something with a slow start, so the team disables the schedule within a week and the saving evaporates.
- • Failover reserve as a fraction shrinks as zones increase: two zones need 100% extra, three need 50%, four need 33%. More zones is a genuine efficiency argument as well as a reliability one.
- • Burst headroom shrinks as startup time shrinks, which makes image size and warm pools a cost lever, not only a latency one.
- • Scheduling overhead falls as workloads become more uniform and rises as they diversify.
- • The dimension that runs out first is usually memory rather than CPU, because memory cannot be over-committed the way CPU can — an exhausted memory allocation is an OOM kill, not a slowdown.
- • Capacity is a security control at the edge: headroom is what absorbs a volumetric attack long enough for rate limiting to engage.
- • A cluster running at maximum utilization cannot schedule emergency workloads — a patched replacement, a forensic collector — precisely when you need them.
- • Idle non-production capacity is attractive to attackers looking for compute; abandoned resources are both a cost item and an unmonitored attack surface.
- • Allocated capacity bills at the same rate as used capacity for every fixed-shape resource. That is the entire premise of the lesson.
- • Over-stated requests are usually the largest recoverable bucket, and they are invisible unless requests and usage are graphed together.
- • Off-hours non-production is the cheapest large saving available in most organizations and requires no architectural change.
- • Failover reserve is a real, ongoing, correct cost. Present it as the price of the availability requirement, not as an inefficiency.
- • Requested versus used, per workload, on one graph. The gap between the two lines is the waste bucket, visible at a glance.
- • Peak utilization and projected post-failover utilization, tracked as first-class capacity metrics.
- • Unschedulable and pending events, which detect scheduling overhead that raw utilization hides.
- • Time from scale-out to scale-in, which reveals whether temporary capacity is actually temporary.
- • The signal that lies: average utilization. It averages the 03:00 trough with the 09:00 peak and describes neither, and it is the number most often used to justify a cut.
- • Before optimizing utilization, delete what is abandoned and shut down what runs outside working hours. Both are risk-free and usually larger than the clever work.
- • Faster startup instead of more headroom: a smaller image and a warm pool let you hold less spare capacity for the same absorption.
- • Interruptible capacity for batch and non-urgent work fills the gaps cheaply without touching the reserve.
- • Fewer, larger nodes reduce stranded fragments; more, smaller nodes reduce blast radius. Pick against your failure requirements, not against the utilization graph.
- • For a small system, do nothing. Idle capacity on two small instances is not worth an engineer's week — see No Cargo-Cult Infrastructure.
- • Higher utilization buys a lower bill and costs the buffer that absorbs spikes and failures. That trade is legitimate and must be made explicitly.
- • Accurate requests improve packing and require ongoing measurement; requests set once and never revisited drift back into waste.
- • Bigger nodes pack better and concentrate more failure; smaller nodes waste more and fail more gracefully.
- • Aggressive scale-in saves money and increases the chance of scaling out again immediately, which costs startup latency at exactly the wrong moment.
Where the bill actually comes from
fixed weight is committed at provision time; usage weight follows the workload. idle = 100% − 35% used → headroom 25% (chosen) + waste 40% (not chosen)
What people believe, and what is true
Idle capacity is waste.
Some of it is the failover reserve and burst headroom that keep the system available. Cutting it to zero is how a traffic spike becomes an outage.
Target 80% utilization and you are done.
A target with no reference to failure domains or scaling latency is a number, not a plan. At 80% across three zones, one zone failure puts the survivors past 100%.
Low utilization means we over-provisioned the hardware.
Often it means the workloads over-stated their requests. The scheduler is reserving capacity nothing uses, which is a configuration problem, not a sizing one.
The average utilization graph tells us what to cut.
Averages hide the peak, and the peak is the only number that determines whether the system survives. Use p95, peak and projected post-failover.