Choosing the Scaling Signal
The metric you scale on decides whether autoscaling works at all — and CPU is the wrong one for most APIs.
The question, the obvious approach, and why it breaks
Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.
Why might CPU-based autoscaling fail for an API, and what should it scale on instead?
Every platform offers CPU as the default scaling signal, and it is the right signal for a minority of the services that use it.
Scale on CPU utilisation. It is available everywhere, it needs no instrumentation, and high CPU is what being busy looks like.
A service that spends its time waiting on a database, a cache or another service is I/O-bound: it holds a request, a thread and a connection while consuming almost no CPU. Every worker can be occupied and CPU can look comfortable (Capacity Management).
- A service that spends its time waiting on a database, a cache or another service is I/O-bound: it holds a request, a thread and a connection while consuming almost no CPU. Every worker can be occupied and CPU can look comfortable (Capacity Management).
- That is the interview answer and it is also the common production reality. When the dependency slows down, per-request CPU stays flat while concurrency rises — so the constraint saturates without moving the signal at all.
- CPU is bounded above. Once the service is fully occupied it cannot rise further, so it saturates as a metric exactly when demand keeps growing.
- On a container platform, CPU is measured against a quota that may be throttling the process, so the utilisation you read is capped by the same limit that is causing the latency (CPU Throttling: The Latency With No Error).
- Garbage collection, sidecars and background work all contribute CPU that has nothing to do with request load, adding noise in both directions.
- Scaling on latency instead looks appealing and is worse in a specific way: latency is a symptom that is affected by scaling, so the loop is chasing a signal it changes.
What is actually happening
Underneath the tooling, which is the part that survives a change of tool.
- A good scaling signal has three properties: it moves monotonically with load, it saturates when the real constraint saturates, and it is not materially affected by the scaling action in a way that feeds back.
- CPU has the first property for compute-bound work and fails the second for everything else. That single distinction explains most autoscaling misconfiguration.
- In-flight requests per instance — concurrency — is the best general-purpose signal for a request-serving service. By Little's Law, concurrency equals arrival rate times service time, so it rises both when more requests arrive and when the dependency slows down. It captures both failure directions with one number.
- Request rate per instance is an open-loop signal: it reflects demand but not cost per request, so it works well when per-request cost is stable and misleads badly after a regression makes requests more expensive.
- Queue depth and age are the right signals for pull-based workers, where there is no request rate to observe (Queue-Based Autoscaling).
- Pool or resource saturation — connection pool utilisation, semaphore occupancy — scales directly on the thing that actually runs out, which is the most precise option when the constraint is known and shared (The Connection Budget).
- Latency is a symptom, is affected by the scaling action, and lags. It belongs in alerting and in canary analysis, not as a scaling target.
- Memory is rarely a good scaling signal because most runtimes do not return memory to the system, so the metric ratchets and never falls — which produces scale-out that never reverses.
The signals, and how each one lies
Every row is used in production somewhere and is correct somewhere. The fourth column is the one to read first — it tells you the workload this signal will quietly fail on.
| Signal | Reflects | Right when | How it fails |
|---|---|---|---|
| CPU utilisation | Compute work per instance | The service is genuinely compute-bound | Flat while an I/O-bound service saturates |
| In-flight requests per instance | Occupancy — arrival rate times service time | Most request-serving services | Needs a concurrency limit to be meaningful |
| Request rate per instance | Demand | Per-request cost is stable and known | Blind to a regression that made requests expensive |
| Queue depth | Outstanding work | Pull-based workers, with rate of change | Steady depth is healthy; the number alone is ambiguous |
| Backlog age | The wait a new item faces | Pull-based workers, as the default | Not exposed by every broker (Queue-Based Autoscaling) |
| Connection pool saturation | The actual shared constraint | The constraint is known and is the pool | Misleads once the constraint moves elsewhere |
| Memory utilisation | Allocated bytes, not demand | Almost never | Ratchets up and never falls on most runtimes |
| Latency | A symptom of saturation | Never as a scaling target | Lags, and is changed by the scaling action itself |
| A business metric | Demand in domain terms | Predictable, known-cost workloads | Needs its own pipeline, which becomes a dependency |
Why CPU fails for an I/O-bound API
This is the interview question, and the answer is worth being able to state precisely rather than approximately. The service is not idle — every worker is occupied — but occupancy and CPU consumption have come apart.
Little's Law gives the clean version: concurrency equals arrival rate times service time. When a dependency slows, service time rises and concurrency rises with it, at unchanged arrival rate and unchanged CPU per request. A signal that tracks concurrency sees it; a signal that tracks CPU does not.
database latency rises
each request holds a worker longer
in-flight requests climb
every worker occupied
CPU per request: unchanged
CPU utilisation: flat and comfortable
-> no scale-out
-> queueing at the entry point
-> timeouts, then client retries
-> the graph shows an idle-looking fleet
failing every requestdatabase latency rises
each request holds a worker longer
in-flight per instance climbs
signal crosses the target
-> scale out
-> more workers to hold requests
-> queueing bounded
and, crucially:
the same signal also rises when
arrival rate rises - one metric,
both failure directionsConcurrency is arrival rate times service time, so it responds to both more traffic and slower dependencies. CPU responds to neither once the work is dominated by waiting. Scaling out does not fix a slow database, but it does keep the service accepting and bounding work rather than collapsing — and the signal at least tells you something is wrong.
Choosing, for a specific service
The choice is short once the constraint is known, which is why the capacity model comes first. Answer that, and the signal usually picks itself.
You are configuring autoscaling for one service. Which signal?
when A profile shows the service is genuinely compute-bound.
cost Silently useless the moment the workload becomes dependency-bound, with no signal that it has.
when A request-serving service that calls dependencies — the common case.
cost Requires a concurrency limit and, on most platforms, a custom metrics path.
when Per-request cost is stable and well understood.
cost Blind to per-request cost regressions, which are exactly what capacity incidents are made of.
when A pull-based worker pool (Queue-Based Autoscaling).
cost Needs broker support or derivation from consumer offsets.
when The constraint is a known shared pool and you want to scale on it directly.
cost Tightly coupled to today's architecture; needs review whenever the constraint moves.
when Load is flat, or the constraint is shared and scaling would make it worse.
cost You pay for the peak, and you keep a simple system (Overprovisioning is the failure mode if nobody revisits it).
How to do it properly
Most important first.
- Start from the binding constraint, then pick the signal that saturates with it. This is the whole method (Building a Capacity Model).
- For request-serving services, prefer in-flight requests per instance. It is the closest available proxy for "how occupied is this instance" and it responds to dependency slowdowns.
- For pull-based workers, use backlog age (Queue-Based Autoscaling).
- Use CPU when the service is genuinely compute-bound, and verify that claim with a profile rather than assuming it.
- Verify the signal by load testing to saturation and checking that the metric actually moved when things broke.
- Prefer per-instance normalised signals over fleet totals, so the target does not have to be re-tuned every time the fleet size changes.
- Combine signals with care: most controllers take the maximum of the desired replica counts across metrics, which makes the loop as eager as its most eager signal.
- Keep latency out of the scaling policy and firmly inside alerting, where its being a symptom is exactly what you want (Alert on Symptoms, Not on Causes).
How much can this affect
Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.
Contained only by the floor: a service scaling on a signal that never rises is protected by whatever standing capacity you decided to keep, and by nothing else.
What can go wrong
- CPU-based scaling on an I/O-bound service, which produces no scale events during the incident it was meant to prevent.
- Request-rate scaling that keeps the fleet at its old size after a change made every request twice as expensive.
- Concurrency-based scaling with no upper bound on concurrency per instance, so an instance accepts more work than it can serve rather than exerting backpressure (Load Shedding).
- Memory-based scaling on a runtime with a lazy collector, producing a fleet that only ever grows.
- A custom metric pipeline that becomes a dependency of the scaling loop and fails silently, freezing the fleet at whatever size it was.
- The mitigation failing: multiple metrics configured for safety, with the noisiest one dominating every decision.
- "CPU is a bad metric." CPU is an excellent metric for compute-bound work. It is the wrong metric for services that spend their time waiting, which happens to be most services.
- "We should scale on latency, because that is what users feel." Latency is what you alert on. Scaling on it creates a loop chasing a signal it is itself changing, with lag in between.
- "More metrics means a more robust policy." Most controllers take the maximum across metrics, so adding a noisy metric makes the loop noisier, not safer.
- "The signal only matters for scale-out." A bad signal is equally wrong on the way down, and scaling in on a metric that does not reflect load is how capacity disappears before a peak.
Operating it
Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.
- A load test to saturation showing the chosen signal moving before the error rate did.
- An incident where the dependency slowed and the signal responded — the specific case CPU fails.
- The signal plotted against the binding constraint's utilisation, showing they track each other.
- Scale events correlated with traffic events on the same timeline, with capacity arriving before impact (Deploys on the Same Timeline as the Symptom).
- A record of which signal was chosen and why, in the service's runbook (Runbooks).
- Changing the scaling signal changes the fleet size the loop wants, sometimes dramatically. Set an explicit floor before switching, and watch the first full traffic cycle.
- Keep the old signal recorded and reverting to it easy — a signal change is a policy change and should be revertible without a deploy (A Config Change Is a Production Change).
- If a custom metric pipeline is the new dependency, know what the controller does when it is unavailable: on most platforms it holds the current size, which is safe but silent.
- Automate the loop and the metric pipeline, and automate an alert on the metric pipeline itself, since a scaling loop with no input fails quietly.
- Automate the verification: a periodic load test that confirms the signal still saturates with the constraint after the service has changed.
- Keep the choice of signal human. It follows from an understanding of what the service waits on, which is exactly the thing a controller cannot infer.
- Better signals usually need instrumentation and a custom metric pipeline, which is real work and a new dependency in the scaling path.
- Concurrency-based scaling requires a concurrency limit to be meaningful, which means adopting backpressure — a good thing that is nonetheless a change to how the service behaves under load.
- Precise signals such as pool saturation are excellent while that pool is the constraint and become misleading the moment the constraint moves, so they need review when the architecture changes.
Where this applies
This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.
- GENERALThe three properties of a good signal, and the I/O-bound failure of CPU, hold on every platform. Which signals are available differs: some platforms expose in-flight request counts natively while others require application instrumentation and a metrics adapter.
- PLATFORM-SPECIFICServerless platforms scale on concurrency by construction and never expose CPU as a target, which is why this failure mode is largely absent there. A VM autoscaling group can track a load balancer request-count or active-connection metric without any application instrumentation; a container platform generally needs a custom metrics adapter to do the same.
Where the depth lives
This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.
- — Testing & Reliability Engineering — proving a signal saturates with the constraint, which is the only way to know a scaling policy will act when it matters.