Autoscaling
A control loop from a metric to a policy to more or fewer instances — with lag and warm-up as first-class properties rather than details.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
What is autoscaling actually doing, and why is it never as immediate as it looks on a graph?
Load varies by hour, by day and by event, and a fleet sized for the peak is idle most of the time while a fleet sized for the average fails at the peak.
Point the autoscaler at CPU, set a target, and let it handle capacity. That is what it is for, and the graphs afterwards show it adding instances when load rises.
Scaling is never instant. Between the load rising and a new instance serving traffic there is metric delay, a decision interval, provisioning time, process start, application warm-up and load balancer registration. During all of it, the existing fleet is carrying the excess (Headroom).
- Scaling is never instant. Between the load rising and a new instance serving traffic there is metric delay, a decision interval, provisioning time, process start, application warm-up and load balancer registration. During all of it, the existing fleet is carrying the excess (Headroom).
- CPU reflects the constraint only for compute-bound work. A service that waits on a database looks idle by CPU while its connection pool is full, so the autoscaler sees no reason to act (Choosing the Scaling Signal).
- Scaling the stateless tier multiplies pressure on everything behind it. More instances means more connections, more queries and more calls into services that did not scale (The Connection Budget).
- The graph that shows the autoscaler working also shows it working late. Instances arrive after the peak they were meant to serve, which looks like success and is not.
- Scale-in is a change too, and an aggressive one removes capacity in front of the next burst, or kills instances with work in flight (Graceful Shutdown).
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- Autoscaling is a control loop with four parts: a signal that reflects load, a target you want that signal to sit at, a policy that maps the gap to a desired size, and an actuator that provisions or removes units.
- The loop runs continuously against a system that responds slowly, which makes it a delayed feedback controller — and delayed feedback controllers oscillate unless they are damped.
- The delay has named parts, and they add up: metric collection and aggregation, the controller evaluation interval, the provisioning call, boot and image pull, application start, warm-up (JIT, caches, pools), health check passes, and registration with the load balancer.
- Because of that delay, autoscaling handles load that ramps over a timescale longer than the delay. It cannot handle a step change, and it cannot handle a burst that lasts less time than a new instance takes to appear.
- Every autoscaler needs a floor and a ceiling. The floor is the standing capacity that carries load while scaling happens; the ceiling is what stops a runaway loop from scaling into a downstream failure or a quota.
- Damping is what makes it stable: cooldown or stabilisation windows, asymmetric behaviour (scale out fast, scale in slowly), and a tolerance band so small deviations do not trigger action.
- Autoscaling changes how much capacity you have. It does not change what the capacity is limited by, so a fleet that scales into a shared constraint scales into a wall (Building a Capacity Model).
The loop, with the delay drawn in
Drawn as a loop it looks instantaneous. The delay is not a detail of the implementation; it is the property that determines what workloads autoscaling can serve at all.
Read the diagram as a cycle time. Whatever traffic arrives inside one lap of it is served by the fleet you already had.
The delay budget
Every phase is measurable and most are reducible. Measuring your own is a half-day of work and changes the floor you choose, which is the setting that actually determines whether bursts are served.
- 1Metric collection
The signal is scraped or reported and aggregated into a value the controller can read.
fails by A long scrape interval or a wide aggregation window makes the controller act on stale load.
evidence Age of the metric the controller used, compared with the current value.
- 2Evaluation interval
The controller wakes, compares signal to target, computes a desired size.
fails by A long interval delays every decision; a short one amplifies noise into churn.
evidence Time between scale decisions in the controller log.
- 3Provisioning
The platform allocates a unit — a container slot, a virtual machine, an execution environment.
fails by Capacity unavailable in the zone, or a quota reached, so the request simply does not succeed.
evidence Provisioning failures counted separately from scaling events.
- 4Image and start
The image is pulled if not cached, and the process starts.
fails by A large image or a cold node registry cache dominates the whole budget (What Image Size Actually Costs).
evidence Time from unit created to process running.
- 5Warm-up
Pools connect, caches fill, runtimes reach steady-state performance.
fails by The instance is healthy and slow, so it serves traffic badly (Scale to Zero hits this hardest).
evidence Per-instance latency during the first minutes after start.
- 6Health and registration
Readiness passes and the load balancer begins routing to the unit.
fails by A readiness check that passes before warm-up finishes, sending traffic to a cold instance (Probes: Readiness, Liveness and Startup).
evidence Time from process start to first request served.
Sum these and you have the answer to the only question that matters for the floor: how much traffic growth must the existing fleet absorb unaided?
When autoscaling is the wrong tool
Autoscaling is one option for handling variable load, and it is the default answer far more often than it is the right one. The alternatives are not exotic; they are frequently cheaper and always more predictable.
A service sees load vary substantially through the day. What handles the variation?
when Load ramps over a timescale comfortably longer than the measured scaling delay, and the constraint is per-instance.
cost A control loop to tune and debug, plus lag on every change.
when The pattern is predictable — daily peak, known campaign, batch window.
cost Requires the pattern to hold; an unexpected event finds you at the scheduled size.
when Peak-to-average ratio is small, or the service is critical enough that lag is unacceptable.
cost You pay for the peak continuously (Overprovisioning is the failure mode when nobody re-checks).
when Work is asynchronous and delay is acceptable (Queue-Based Autoscaling).
cost Latency becomes the release valve; queue age becomes the signal you must watch.
when The burst is short, the load is beyond any scaling response, or some traffic is genuinely lower value.
cost Some requests fail by design, which is a product decision (Load Shedding).
when Per-request cost has grown and capacity is compensating for a regression.
cost Slower to deliver than scaling, and usually the highest-return option available.
How to do it properly
Most important first.
- Choose the signal from the actual constraint before touching anything else. This is the decision that determines whether autoscaling works (Choosing the Scaling Signal).
- Measure your own end-to-end scaling delay — from metric change to serving traffic — and hold enough standing capacity to bridge it (Headroom).
- Set a floor high enough to survive the delay and a node loss, not the smallest number that works at 3am.
- Set a ceiling derived from what the downstream can take, not from what the account allows (Capacity During Failover).
- Scale out aggressively and in conservatively. The costs are asymmetric: scaling out too early costs money, scaling in too early costs an incident.
- Make instance start-up fast and warm-up short. Every second removed from that path is a second of lag removed from the control loop (What Image Size Actually Costs).
- For predictable patterns — a daily peak, a scheduled campaign — pre-scale on a schedule rather than reacting. Reactive scaling to a known event is a choice to be late.
- Test scaling behaviour under a load shape that ramps, and under one that steps. They produce completely different results.
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained by the floor, the ceiling and the ability to pin the fleet at a fixed size — an autoscaler with no ceiling has no containment and can take a shared dependency down.
What can go wrong
- Scaling that arrives after the burst ended, so the fleet is largest when demand is smallest.
- Oscillation: scale out, load per instance drops, scale in, load per instance rises, repeat — with each cycle paying start-up and cold-cache costs (How Autoscaling Fails).
- A ceiling hit silently, so the system is saturated while the autoscaler reports it is doing its job.
- Scaling into a downstream constraint, turning a service capacity problem into a database incident.
- Scale-in terminating instances holding long-running requests or consuming messages, producing errors and duplicate work (Draining: Stopping Without Dropping).
- The mitigation failing: a floor set for cost reasons that is below what the scaling delay requires, so every burst is served by a fleet that is still starting.
- "Autoscaling means we do not need capacity planning." It means the fleet size varies. You still need to know the binding constraint, the ceiling and the floor — all of which come from a capacity model (Building a Capacity Model).
- "Autoscaling protects us from traffic spikes." It protects against ramps slower than its own delay. A spike is precisely the case it cannot serve (Load Shedding is what handles that).
- "More aggressive thresholds mean better responsiveness." They mean more oscillation. Responsiveness is limited by provisioning and warm-up time, not by the threshold.
- "It scaled, so it worked." Check whether the capacity arrived before or after the error rate rose. Arriving afterwards is the common case and it is not success.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A measured end-to-end scaling delay, recorded, with the breakdown by phase.
- Fleet size plotted against the scaling signal and against traffic, so lead and lag are visible rather than assumed.
- A load test with a ramp and a step, showing what the fleet did in each and what the error rate did.
- Scale events annotated on the operator dashboard, so an incident timeline shows them alongside deploys (Deploys on the Same Timeline as the Symptom).
- Ceiling-reached and floor-reached events counted, since both are silent by default.
- Autoscaling policy is configuration and should be revertible without a deploy — the first time you need to change it will be during an incident (A Config Change Is a Production Change).
- The most useful rollback is pinning: set minimum equal to maximum at a known-good size and take the loop out of the incident entirely. Every autoscaled service should have a documented way to do that (Runbooks).
- A raised ceiling is easy to set and easy to forget. Record it, because a temporary ceiling left in place is how a later runaway reaches the database.
- Automate the loop itself — that is what it is — and automate scheduled pre-scaling for known events.
- Automate the alert for ceiling reached and for sustained scaling activity, both of which mean the capacity conversation has been deferred rather than answered.
- Keep the ceiling and the floor human. They encode what the downstream can survive and how much standing capacity you are buying, and neither is derivable from the service's own metrics (The Automation Trap).
- Autoscaling trades cost for a slower response to load. A statically provisioned fleet at peak size responds instantly and costs continuously.
- It adds a control loop to your production system, which means the system now has dynamics of its own to debug — a genuine increase in operational complexity.
- Aggressive scale-in saves money and increases the chance of being caught short; conservative scale-in costs money and is nearly always the right default.
- A high floor undermines much of the saving, and is frequently still the correct choice for a service on the critical path.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALSignal, target, policy, actuator and lag describe every autoscaler. The magnitude of the lag varies enormously: a container on a warm node may be serving in seconds, a new virtual machine takes minutes, and a new node in a cluster takes both.
- PLATFORM-SPECIFICWhat the actuator does differs by platform. A VM autoscaling group launches instances from an image; a container orchestrator schedules replicas onto existing nodes and only sometimes triggers a node addition; a serverless platform creates execution environments per concurrent request and never exposes a replica count at all.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — proving a control loop behaves under load shapes it was not tuned for.