The question this answers
How does replica count follow demand automatically, and how late is the capacity when it finally arrives?
Traffic is four times higher between 09:00 and 11:00 than overnight, with unpredictable spikes when a marketing email goes out. Paying for the peak all night is waste; being sized for the night during the peak is an outage.
A control loop that adjusts replica count from an observed metric, within bounds you set — trading a continuous over-provisioning cost for a delayed, approximate match between capacity and demand.
The loop, and the arithmetic inside it
The autoscaler is another controller with the same shape as everything else in this module: observe, compute a gap, act. It reads a metric, compares it against a target, and computes a desired replica count with essentially one formula — current replicas multiplied by the ratio of current metric to target metric, rounded up. Six replicas at 90% of a 60% target gives nine.
Two subtleties change behaviour more than the formula does. First, CPU-based scaling is measured as a percentage of the request, not of the node or of the limit — so a workload with a request set far below its real usage appears permanently over target and scales up forever, and one with an inflated request never scales at all. Wrong requests produce wrong autoscaling, which is why Requests vs Limits: Two Numbers That Do Different Jobs comes first.
Second, scale-down is deliberately asymmetric: it is delayed by a stabilization window, typically several minutes, so that a brief dip does not remove capacity that is about to be needed again. Scale-up is fast; scale-down is cautious. That asymmetry is correct and it means your average replica count is meaningfully higher than a naive reading of the traffic curve suggests.
Add up the delay before you trust it
This is the part that turns autoscaling from a feature into an engineering decision. The capacity does not appear when the traffic does; it appears after a chain of independent delays, and the total is routinely one to three minutes even when nothing is wrong. If your traffic spike lasts ninety seconds, autoscaling will finish arriving shortly after it ends — and then scale back down, having contributed nothing but a bill.
Worse, the delays compound in the one case where you most need them not to. If the cluster has no spare capacity, the new pods are Pending until a *node* is provisioned, which adds machine boot time to the chain — commonly several minutes more. So the cluster that is most efficiently packed is also the one that responds slowest, which is a genuine, unavoidable trade between utilization and elasticity.
The practical consequences follow directly. Keep headroom: run enough replicas that the existing fleet absorbs the first minutes of a spike unaided. Attack the chain: a smaller image and a faster warm-up shorten the two largest terms — see Why Image Size Is an Infrastructure Problem and Startup Time & Cold Start. Scale on the right signal: CPU is a proxy, and for a queue worker the queue depth is the real signal — see Autoscaling Signals. And pre-scale what you can predict: a marketing send at 10:00 is a scheduled scale-up, not an autoscaling problem.
- 1t=0 · traffic rises
Requests per second quadruple. Existing replicas absorb it by queueing, and latency starts climbing immediately.
This is where the user impact begins. Everything below is recovery time.
- 2Metric window15–60s
The metrics pipeline must observe and aggregate the new load before it is visible to the controller.
A longer window smooths noise and delays every reaction. This term is often the largest and the least examined.
- 3Controller evaluation0–15s
The autoscaler runs on its interval, computes the new desired count and writes it to the Deployment.
Scale-up is prompt; scale-down waits out a stabilization window of several minutes.
- 4Scheduling<1s, or minutes
New pods are created and the scheduler places them on nodes with free capacity.
With no spare capacity they stay Pending until a node is added — add several minutes of machine boot.
- 5Image pull and start2–60s cached, longer cold
The node pulls the image if it is not cached, then starts the containers.
A large image on a cold node is frequently the single biggest term in the whole chain.
- 6Application warm-up5–120s
Connection pools open, caches fill, JIT warms, readiness finally passes.
A pod admitted before it is genuinely warm serves slow requests and can make the incident worse.
- 7Serving
The pod joins the Service endpoints and takes its share of traffic.
Total elapsed: commonly 1–3 minutes. A 90-second spike was over before this line was reached.
Choosing the signal, and the bounds that protect you
CPU is the default metric because it is always available, not because it is usually right. It is a reasonable proxy for a CPU-bound service and a poor one for almost everything else. A worker that is slow because a queue is backing up shows modest CPU while the backlog grows; scaling on queue depth or on age-of-oldest-message reacts to the actual problem. An API bound by an external dependency shows low CPU and high latency, and scaling it out adds concurrency against a dependency that is already the bottleneck — which makes things worse, not better.
That last case is the one worth stating plainly: autoscaling amplifies whatever the workload does. More replicas means more database connections, more calls to a rate-limited third party, more pressure on the thing that was actually saturated. An autoscaler pointed at the wrong signal is an efficient way to convert a slow dependency into a complete outage.
Bounds are not a formality. The minimum protects you against scaling down into a spike and against the stampede when everything restarts at once. The maximum is your cost ceiling and your protection against a metric bug: an autoscaler with no sensible maximum, given a broken metric, will scale to whatever the cluster will hold and bill you for it. Set both, and treat the maximum as a budget decision rather than a technical one.
| Workload | Default signal | Why it misleads | Better signal |
|---|---|---|---|
| CPU-bound API (rendering, parsing) | CPU % of request | Genuinely fine here. | CPU is the right answer. |
| Queue worker | CPU % | A worker blocked on a slow downstream shows low CPU while the backlog grows. | Queue depth, or age of the oldest message. |
| API bound by a slow dependency | CPU % | Low CPU, high latency. Scaling adds concurrency against the real bottleneck. | Nothing — fix the dependency. Scaling here makes it worse. |
| Memory-heavy service | CPU % | Memory pressure does not move CPU until it is far too late. | Working set, or a custom saturation metric. |
| Connection-bound service | CPU % | Idle connections consume sockets and memory, not cycles. | Active connections per replica. |
| Predictable daily peak | Any reactive metric | Reactive scaling is always late for a spike you could have foreseen. | A scheduled scale-up before the peak. |
Key points
- The autoscaler computes desired replicas as current × (current metric / target), rounded up, and writes it to the Deployment.
- CPU-based scaling is measured against the *request*, so wrong requests produce wrong autoscaling in both directions.
- Capacity is late by the sum of metric window, controller interval, scheduling, image pull and application warm-up — commonly one to three minutes.
- A tightly packed cluster responds slowest, because new pods must wait for a new node; utilization and elasticity trade directly against each other.
- Scaling on the wrong signal amplifies the real bottleneck: more replicas means more connections to the dependency that was already saturated.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A metrics pipeline collects per-pod resource or custom metrics and aggregates them over a window.
- • The autoscaling controller reads the aggregated metric on its interval and compares it against the configured target.
- • It computes a desired replica count from the ratio and writes it to the workload's replica field, clamped to the configured minimum and maximum.
- • The workload controller creates the new pods; the scheduler places them, or leaves them Pending if no node has room.
- • Each new pod pulls its image, starts, warms up and joins the Service only once its readiness probe passes.
- • Scale-down applies a stabilization window so a brief dip does not remove capacity that is about to be needed.
- • Correct resource requests, since they are the denominator of the most common scaling metric.
- • Choosing and maintaining the signal that actually represents pressure for each workload.
- • Minimum and maximum bounds, treated as a reliability floor and a cost ceiling respectively.
- • Startup time as an ongoing engineering concern — it is the term you can most directly shorten.
- • Cluster capacity or node autoscaling underneath, because pod autoscaling without somewhere to put the pods accomplishes nothing.
- • Interaction with rollouts: an autoscaler and a deploy competing for capacity produce stalls that look like scheduler bugs.
- • Capacity arriving after the spike ended, having cost money and helped nobody.
- • Pending pods because the cluster is full: the autoscaler did its job and nothing happened.
- • Flapping between counts when the target is close to steady-state usage, causing continuous churn and cold starts.
- • Scaling out against a saturated dependency, turning a slow database into an exhausted connection pool and a full outage.
- • A metric pipeline outage leaving the autoscaler blind; replica count freezes at its last value and no alert fires.
- • A missing maximum plus a broken metric, producing a scale-up that fills the cluster and the invoice.
- • Replica count scales throughput only while the shared dependency behind it has room — the database connection pool is the usual real ceiling.
- • Time-to-capacity degrades as image size and warm-up time grow, so autoscaling quality is downstream of build and startup engineering.
- • At larger replica counts each scaling step is proportionally larger, which makes overshoot more expensive and stabilization windows more important.
- • A maximum replica count is a denial-of-wallet control: without it, an attacker who can generate load can generate spend.
- • Each new replica opens connections and holds credentials, so scaling multiplies both your dependency load and your credential footprint.
- • Custom metrics adapters run with cluster access and are an often-overlooked privileged component in the platform.
- • Rapid scaling can trip rate limits and anomaly detection on downstream services, which presents as an outage rather than as a security event.
- • Autoscaling converts a fixed peak-sized cost into a usage-shaped one, which is the entire financial argument for it.
- • The asymmetric scale-down window means you pay for capacity after you stop needing it, by design.
- • Headroom kept for responsiveness is capacity billed continuously and used rarely — the direct price of elasticity.
- • Overshoot from an aggressive target costs real money at every spike, and a missing maximum makes that unbounded.
- • Replica count over time, overlaid with the scaling metric and with request rate — the only view that shows whether scaling is tracking reality.
- • Time from metric breach to new pods Ready, which is the number that tells you whether autoscaling is fast enough to matter.
- • Pending pod count during scale-up events, which distinguishes "the autoscaler is not firing" from "there is nowhere to put the pods".
- • Scaling event history, including when the maximum was reached — hitting the ceiling is a capacity conversation, not a metric blip.
- • The signal that lies: "the autoscaler is configured and the target looks reasonable". It says nothing about whether capacity arrives before the users notice.
- • Scheduled scaling for predictable patterns — a known 10:00 peak is a calendar problem, and pre-scaling always beats reacting.
- • Static over-provisioning: for a small service, running enough replicas for the peak all day is cheaper than the complexity, and it is never late.
- • A queue with backpressure, which absorbs a spike without adding capacity at all and is often the better architecture — see Autoscaling.
- • Serverless, where scaling is per-request and the platform owns the delay — you trade it for cold starts, which is a different version of the same problem.
- • Vertical scaling, when the workload cannot be parallelized and one bigger instance is the honest answer.
- • Buys capacity that follows demand; costs a delay chain that makes it always late, and most late during the sharpest spikes.
- • Buys lower spend at low traffic; costs headroom, overshoot and a scale-down window you pay for.
- • Buys automatic response; costs the risk of amplifying a dependency bottleneck into an outage when the signal is wrong.
- • Buys elasticity; costs utilization, because a tightly packed cluster has nowhere to put the new pods.
Traffic rises. When does capacity arrive?
traffic ▁▁▁▁▃▄▅▆▇███████████████████████████████████████████████████ 900 rps replicas ▁▁▁▁▁▁▂▂▂▂▂▂▂▂▂▂▂▂▃▃▃▃▃▃▃▃▃▃▃▃▄▄▄▄▄▄▄▄▄▄▄▄▇▇▇▇▇▇▇▇▇▇▇▇██████ 19 (15 ready) metric ▄▄▄▄███████████████████████████████████████████████▅▅▅▅▅▅▅▅▅ 60% reported, target 60% shed ▁▁▁▁▁▂▄▅▇██████▇▇▇▇▇▇▇▇▇▇▇▇▅▅▅▅▅▅▅▅▅▅▅▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 0 rps over capacity now
What people believe, and what is true
Autoscaling handles traffic spikes.
It handles sustained traffic changes. A ninety-second spike is over before the new capacity is Ready, which is why headroom exists.
CPU is a reasonable default scaling metric.
It is a reasonable metric for CPU-bound work. For queue workers and dependency-bound services it is actively misleading, and scaling on it can make the incident worse.
Autoscaling saves money.
It converts fixed cost into usage-shaped cost. With headroom, overshoot and a delayed scale-down, a small steady service is often cheaper statically provisioned.