AutoscalingGENERALPLATFORM-SPECIFIC

Queue-Based Autoscaling

For workers, queue depth and message age describe the constraint far better than CPU ever will.

What actually happens How to do it properly Evidence and rollback

The question, the obvious approach, and why it breaks

Every lesson starts where the work starts: an operational problem, a first attempt that is entirely reasonable, and the way production disagrees with it.

The production question

What should a pool of background workers scale on?

The problem

Workers do not receive requests; they pull work. Utilisation of a worker tells you what it is doing, and nothing about how much work is waiting.

What teams do first

Scale workers on CPU, the same way the API tier scales. It is the metric the platform offers by default and the tooling is already in place.

How it breaks

A worker blocked on a network call, a database write or an external API uses almost no CPU while being completely unavailable for other work. A saturated worker pool can look idle (Choosing the Scaling Signal).

How it breaks in production

A worker blocked on a network call, a database write or an external API uses almost no CPU while being completely unavailable for other work. A saturated worker pool can look idle (Choosing the Scaling Signal).
CPU is bounded above and backlog is not. Once every worker is busy, CPU stops rising no matter how much work accumulates, so the metric goes flat exactly when the problem starts growing.
What users of an asynchronous system experience is delay — how long work waits before it is picked up. CPU has no relationship to that at all.
Consumers with a fixed prefetch or concurrency setting cap their own CPU usage, making the metric even less responsive to backlog.
By the time CPU reflects a problem, the backlog has been growing for a while, and draining it needs more capacity than serving the arrival rate would have.

Code→Build→Test→Artifact→Release→Deploy→Run→Observe→Operate→Incident→Recover→Learn→Improve

What is actually happening

Underneath the tooling, which is the part that survives a change of tool.

For a worker pool the constraint is throughput against arrival rate, and the observable that reflects it is the queue: how much is waiting, and how long the oldest item has been waiting.
Depth and age answer different questions. Depth is how much work is outstanding. Age — the wait time of the oldest unprocessed item — is the delay a new item can expect, which is the thing that maps to a user-visible promise.
Depth alone is ambiguous. A constant depth with high throughput is a healthy pipeline running at a steady offered load; a small depth that never drains is a stalled consumer. The rate of change of depth distinguishes them.
Scaling on age gives you a control target that means something: keep the oldest item younger than the latency objective. That is a service-level statement rather than a resource-utilisation statement.
Little's Law relates the three: with a stable arrival rate, the number waiting equals the arrival rate times the wait time. Given two of them the third follows, which is what makes backlog-based capacity arithmetic tractable — Performance owns the derivation.
The drain-time framing is the most operationally useful: at the current consumption rate, how long until the backlog is empty? That single number tells an operator whether to wait, to scale, or to escalate.
Workers scale the same way everything else does, and multiply downstream pressure the same way. Doubling consumers doubles the write rate into whatever they write to (The Connection Budget).

Two signals for the same pool

The comparison is stark because the two metrics are measuring different things entirely: one measures what a worker is doing, the other measures what is waiting for a worker.

Scaling a worker pool

On CPU utilisation

workers wait on I/O most of the time
  CPU stays low while all workers are busy
  backlog grows, CPU does not
  metric saturates at "all busy"
    -> no scale-out signal
    -> delay grows unbounded
    -> first alert is a customer noticing
       work never completed

On backlog age

signal = age of the oldest unprocessed item
  target = the latency objective for this work
  age rises the moment arrivals exceed drain
    -> scale-out before the promise breaks
    -> the target means something to a user
    -> drain time is directly computable
       from arrival and consumption rates

CPU describes the worker; age describes the customer's wait. For asynchronous work, the wait is the thing you promised, and it is the only one of the two that keeps rising when the pool is fully occupied.

The signals a queue offers, and what each is good for

Most brokers expose several of these and teams reach for the first one they find. They are not interchangeable, and the alerting choice matters as much as the scaling choice.

Signal	What it says	Good as a scaling target	How it misleads
Depth	How much work is outstanding	Only with rate of change alongside	A steady depth is healthy; a small stuck depth is not
Rate of change of depth	Whether you are draining or falling behind	Yes, as a trend input	Noisy at low volumes
Oldest message age	The wait a new item can expect	The best default target	Needs a broker that exposes it, or derivation from offsets
Consumer lag	How far behind the consumer is in a log	Yes, for partitioned logs	Measured in offsets, not time — a big message and a small one count the same
Arrival rate	How much work is coming in	For predictive or scheduled scaling	Says nothing about whether you are keeping up
Consumption rate	How fast you are draining	As the denominator of drain time	Falls when consumers are stuck, which looks like less work
Dead-letter depth	How much work has failed permanently	Never — it is a correctness signal	Inflates total depth and cannot be drained by scaling (Dead Letter Queues Are an Operation)

Where the backlog actually goes

GENERALThe topology is the same on any broker. The narrowest downstream point differs by workload — a database write path, a third-party rate limit, or a lock the consumers contend on — and that point, not the queue, is what should set the consumer maximum.

Scaling consumers moves the constraint downstream, exactly as it does for a request-serving tier. The diagram is worth drawing before setting a maximum, because the maximum should come from the narrowest point in it.

Consumers scale; the write path does not

UserLLMAgentToolDataDecisionHumanGuardrail

How to do it properly

Most important first.

Scale on backlog age where the platform exposes it, and on depth plus its rate of change where it does not (Operating Queues and Scheduled Work).
Express the target as a service-level statement — oldest item younger than some bound — so the policy has a reason attached.
Set the maximum consumer count from what the downstream can absorb, not from how fast you would like to drain (How Autoscaling Fails).
Exclude the dead-letter queue from the scaling signal. Poison messages inflate depth and adding consumers cannot reduce it (Dead Letter Queues Are an Operation).
Handle scale-in by letting consumers finish or return in-flight messages rather than being killed mid-processing (Graceful Shutdown).
Make the work idempotent, because scaling events plus at-least-once delivery means retries and duplicates are normal operation, not an exception.
Alert on backlog age, not on depth. Age is the signal that maps to a promise; depth is the one that changes meaning with throughput.
Separate queues by priority or by workload class so a bulk backlog cannot delay interactive work behind it.

How much can this affect

Every production change has a blast radius. Stated as a scale so it is comparable between changes rather than adjectival — and paired with what actually contains it, because a wide scope with a real containment mechanism is a different situation from a wide scope with none.

Blast radius if this is wrongEveryone

One testEveryone

What contains it

The queue is the containment — backlog turns overload into delay rather than failure, up to the point where retention or a downstream limit is reached.

What can go wrong

Failure modes, including of the mitigation

Scaling consumers into a database that cannot take the write rate, converting a backlog into an outage (Operating a Production Database).
A poison message at the head of an ordered partition blocking progress: depth rises, age rises, and no amount of scaling helps (Operating Queues and Scheduled Work).
Scale-in terminating a consumer mid-message, so the message is redelivered and processed twice.
Depth measured across all queues together, so one large batch backlog triggers scaling for consumers of an entirely different workload.
Scaling on depth alone with a slow consumer bug: the fleet grows, each new consumer is equally slow, and the backlog keeps growing while cost rises.
The mitigation failing: a maximum consumer count set to protect the database, reached silently, with the backlog growing behind a healthy-looking fleet (Load Shedding).

Misreads this invites

"A non-empty queue means we are behind." A queue with stable depth and healthy age is doing exactly its job — absorbing variation so consumers can run at a steady rate.
"Scale until the queue is empty." An empty queue means you are paying for consumers that are idle. The target is bounded age, not zero depth.
"Depth is the signal." Depth without a rate of change cannot distinguish a healthy pipeline from a stalled one, and age is closer to what anyone actually promised.
"More consumers always drains faster." Only until the downstream becomes the constraint, after which more consumers drain slower and break something else.

Operating it

Evidence is the signal, not the intention. Rollback is sometimes 'you cannot, and that is the point'.

How you know it worked

Backlog age plotted against the objective, per queue.
Depth, arrival rate and consumption rate on one graph — the three together tell you whether you are draining, holding, or losing.
Drain-time estimate at the current consumption rate, on the operator dashboard.
Consumer count plotted against downstream write rate and connection count.
Dead-letter queue depth tracked separately, with its own alert.

How you get back

Pinning the consumer count is the standard incident action here too, and it is more often used to scale down — when the workers are the thing overwhelming a dependency.
Pausing consumption entirely is a legitimate mitigation for an asynchronous workload in a way it never is for a synchronous one: the work waits in the queue, which is what the queue is for, provided retention outlasts the pause (Operating Queues and Scheduled Work).
Reprocessing after a bad deploy means replaying from the queue or the dead-letter queue, which requires the consumer to be idempotent and requires the messages to still exist (Dead Letter Queues Are an Operation).

What to automate, and what stays human

Automate the scaling loop on backlog age, and automate the drain-time calculation onto the dashboard.
Automate the dead-letter alert, since a filling dead-letter queue is a correctness signal rather than a capacity one.
Keep the decision to purge or replay a queue human, always. It is destructive or duplicative and both outcomes are irreversible in different ways (The Automation Trap).

What this costs

Queue-based scaling gives up the platform default and requires a custom metric pipeline, which is real integration work on most stacks.
Draining a backlog fast means running more consumers than steady state needs, which pushes load onto downstream systems that were sized for the steady rate.
Allowing a backlog at all is a decision to trade latency for capacity — correct for asynchronous work and completely wrong for anything a user is waiting on (Load Shedding).

Where this applies

This domain is unusually tool- and organisation-dependent. These labels say what each claim is specific to, and what a different platform, provider or organisation does instead.

GENERALBacklog age as the scaling signal for consumers holds across brokers and platforms. What differs is availability: some systems expose oldest-message age directly, others expose only depth and consumer offset lag, from which age has to be derived.
PLATFORM-SPECIFICPartitioned logs cap useful consumer count at the partition count, so scaling beyond it adds idle consumers; a competing-consumer queue has no such cap. That difference changes the maximum in the policy and is a property of the broker, not of autoscaling.

Where the depth lives

This domain teaches delivery and operation, and hands the mechanism off to the domain that owns it.

Backendjob-queues worker-scaling worker-processes dead-letter-queues queue-semantics

Observability & Performancequeue-backlog queue-age queue-performance-signals littles-law queueing worker-pool-saturation

Architecturemessage-queues background-jobs

Concurrencybounded-vs-unbounded-queues worker-pools

Domains that do not exist yet

— System Design — choosing between synchronous and asynchronous processing, which decides whether a backlog is an option at all.