Queue Backlog
The queue is growing. Four possible causes, and the recovery that is right for one of them makes two of the others worse.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
The backlog is climbing. What is actually wrong, and what should I do in the next ten minutes?
When background work falls behind, the cause should be identifiable from signals that already exist, and the recovery should not cause a second incident.
The queue is backing up, so add workers. That is what workers are for and it is the fastest lever available.
If the cause is a slow downstream dependency, more workers means more concurrent pressure on the thing that is already struggling — the queue drains no faster and the dependency gets worse (Cascading Failure).
- If the cause is a slow downstream dependency, more workers means more concurrent pressure on the thing that is already struggling — the queue drains no faster and the dependency gets worse (Cascading Failure).
- If the cause is a poison message being retried forever, more workers means more capacity spent on a message that will never succeed (Dead-Letter Queues).
- If the cause is a genuine increase in arrival rate, more workers is correct — but only up to the shared constraint, and past it you have converted a queue problem into a database problem (Connection Pool Exhaustion).
- If the workers are not running at all — crashlooping, wrong configuration, no consumers connected — adding more of the same thing adds nothing, and the depth chart looks identical to the other three cases.
- And by the time depth is visibly high, the oldest items may be so stale that processing them is worse than dropping them: the export nobody is waiting for any more, the notification about an event that has passed.
What is actually happening
- A backlog means one thing: over some interval, arrivals exceeded completions. Every cause reduces to one of four — arrivals rose, per-job duration rose, worker capacity fell, or capacity is being consumed by work that cannot succeed.
- Depth and age are different signals and they answer different questions. Depth says how much work is waiting; age says how long the oldest waiter has waited, which is what a user experiences. A stable depth with a rising age means the queue is being processed out of order or a subset is stuck (Depth Is Not an Emergency; Age Is).
- Little's law relates them: the average time an item spends waiting is proportional to the queue length divided by the completion rate. It gives you time-to-drain from two numbers you already have, and it shows why halving job duration is exactly as effective as doubling the fleet (Little's Law as Working Intuition).
- Recovery is not always "drain it". Work has a shelf life, and a backlog of stale items can be worth discarding, or worth processing newest-first so that current work is timely while the old work catches up behind it.
- A backlog on a shared queue starves everything on that queue, so a bulk import can delay a password reset. That is a design property of the queue layout, and it is only fixable before the incident (Bulkheads).
- Backlogs are self-reinforcing when clients retry. Slow completion produces timeouts, timeouts produce re-enqueues, and the arrival rate rises because the queue is behind (Retry Storms).
Four causes, one chart
A growing depth chart looks the same in all four cases, which is why the instinctive response is right only a quarter of the time. Two additional signals separate them completely: completion rate (is work getting done at all?) and worker utilisation (are the workers busy or waiting?).
Run the table top to bottom during an incident. It takes under a minute and it prevents the most common recovery mistake, which is scaling a fleet into a dependency that is already the bottleneck.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Arrival rate up, completion rate up, workers busy | Depth and age both climbing during a peak | Genuine demand increase — the fleet is undersized | Scale workers, up to the downstream limit and no further (Worker Scaling) |
| Arrival rate flat, completion rate down, workers busy | Job duration has grown; utilisation is high | A downstream dependency slowed, or a deploy made the job more expensive | Do NOT scale. Fix or bound the dependency; check the last release (Deploys Are the First Suspect) |
| Arrival rate flat, completion rate near zero, workers idle | Depth climbs linearly; no errors | Consumers are not running: crashloop, bad config, no connection | Check consumer health and connection count before touching anything else (Health Checks: Startup, Readiness, Liveness) |
| Completion rate down, attempt counts climbing | Errors flat, retries rising, capacity consumed | Poison messages retried without a cap | Cap attempts and dead-letter; the fleet frees itself (Dead-Letter Queues) |
| Depth falling, oldest age still rising | Looks like recovery; users still complaining | Newest-first processing starving the oldest items | Drain the old items on a separate fleet so both progress |
| Depth drops sharply with no matching completions | The problem "resolved itself" | Broker retention expired the oldest messages | Treat as data loss: identify the window and re-derive the work from source |
Recovery, in order
Recovery has a sequence, and the sequence matters more than any individual action. Stop the bleeding, then identify, then choose the drain strategy, then prevent the drain from causing a second incident.
The step people skip is the fourth. A fleet released at full speed into a dependency that has just come back is a well-documented way to have the same outage twice in twenty minutes (Backpressure).
- 11. Stop the growth
Rate-limit or pause the producer if the arrival rate is the cause.
fails by Draining while arrivals continue — the fleet never catches up.
- 22. Identify the cause
Arrival versus completion rate, plus worker utilisation.
fails by Jumping to "add workers" and deepening a dependency outage.
- 33. Free stuck capacity
Dead-letter poison messages; restart crashlooping consumers.
fails by Leaving a poison loop consuming the fleet while you scale around it.
- 44. Decide what the backlog is worth
Drain, prioritise newest-first, or shed the stale items.
fails by Processing hours of work nobody is waiting for while fresh work queues behind it.
- 55. Drain at a controlled rate
Scale up to the downstream limit; rate-limit toward recovering dependencies.
fails by Full-speed drain re-breaking the dependency that just recovered.
- 66. Verify by age, not depth
Watch oldest-message age return to normal.
fails by Declaring recovery on a falling depth chart while the oldest items are still stale.
Steps 1 and 4 are the ones that need a decision made in advance. Who may pause a producer, and who may approve discarding work, are questions with no good answer at 3am.
Arithmetic, not intuition
Two numbers you already have — current depth and current completion rate — give an estimated time to drain. Two more — arrival rate and completion rate — say whether it will drain at all. Neither requires a benchmark or a guess about absolute performance; both are ratios of quantities your broker already reports.
This is Little's law used as a triage tool rather than as theory. It also makes the equivalence explicit: halving job duration doubles the completion rate, which has exactly the same effect on drain time as doubling the worker count — and it does not consume a single extra connection (Little's Law as Working Intuition).
| What you observe | What it means | What follows |
|---|---|---|
| Completion rate > arrival rate | The queue is draining | Time to drain is roughly depth divided by the difference between the two rates |
| Completion rate = arrival rate | Depth is stable, age is not falling | You are keeping up and never catching up — capacity must rise or arrivals must fall |
| Completion rate < arrival rate | The backlog grows without bound | No amount of waiting helps; something must change (Backpressure) |
| Completion rate fell, workers busy | Per-job duration rose | The constraint is downstream or in the code — scaling adds pressure, not throughput |
| Completion rate fell, workers idle | Workers are blocked or absent | Check consumers, connections and configuration before scaling (Health Checks: Startup, Readiness, Liveness) |
| Depth falling, age rising | Newest-first processing | The oldest work is starving; drain it separately |
| Halve job duration | Completion rate doubles | Identical drain effect to doubling the fleet, with no extra connections (The N+1 Query Problem) |
How to build it
Most important first.
- Alert on oldest-message age, not depth, with a threshold set at the point where the delay becomes user-visible. This is the alert that catches all four causes (Alerts Worth Waking Someone For).
- Separate queues by job class and by latency expectation, so that a backlog in bulk work cannot delay interactive work (Bulkheads).
- Cap attempts and dead-letter, so poison messages cannot consume capacity indefinitely (Dead-Letter Queues).
- Bound worker concurrency toward each downstream dependency, so recovery cannot become a second outage (Resource Limits).
- Make jobs cheaper before making the fleet bigger. Profiling a hot job pays back permanently and costs no infrastructure (Why Is My API Slow?).
- Give jobs a deadline or a freshness check. A job that is now pointless should be able to say so and exit rather than consuming a worker.
- Have a drain plan written down before you need it: what to shed, what to prioritise, what the maximum safe worker count is, and who can approve dropping work.
- Instrument arrival rate and completion rate as separate series. The gap between them is the earliest possible signal and it precedes any threshold on depth (Six Queue Signals, Two That Wake You Up).
What can go wrong
- Scaling workers into a struggling dependency and deepening the outage.
- A poison message consuming the fleet while depth grows, with the error rate looking flat because the failure is inside a retry loop.
- A silent consumer outage: workers crashlooping or disconnected, so completion rate is zero and depth grows linearly with arrivals.
- A backlog of stale work processed in order, so every completed item is already worthless while fresh items wait behind them.
- A shared queue starving latency-sensitive work behind a bulk job.
- Recovery draining at full speed into a dependency that has just recovered, knocking it over again (Backpressure).
- Messages expiring or being evicted at a broker retention limit, so the backlog resolves itself by silently discarding work.
- Alerting on depth only, so a slow-growing backlog crosses no threshold until it is enormous.
- Recovery scaling racing a dependency's own recovery, so the drain arrives before the dependency is ready and knocks it over again.
- A poison message redelivered across many workers simultaneously, multiplying its capacity cost (Queue Semantics).
- Two operators taking recovery actions at once — one scaling up, one purging — with no coordination.
- Broker retention expiring messages while a drain is in progress, so the backlog and the drain race and the backlog wins silently.
- An attacker who can enqueue can create a backlog cheaply. Rate limit and quota the enqueue path per tenant (Rate Limiting).
- A backlog is a denial of service against everything on that queue, including security-relevant work: audit writes, revocation propagation, fraud checks (Multi-Tenancy).
- Shedding under backlog must not drop security-relevant work first. Audit and revocation belong in a priority class that is never shed (Defence in Depth).
- Do not expose backlog depth publicly. It tells an attacker precisely how much pressure produces degradation (Not Leaking Your Internals).
- "Add workers." Correct for one of the four causes and harmful for two of them. Identify the cause first — it takes a minute with the right two charts.
- "The queue is deep, so we are behind." Depth without job duration says nothing. A deep queue of fast jobs may be draining in seconds (Little's Law as Working Intuition).
- "The backlog is shrinking, so we recovered." Check age. Newest-first processing shrinks depth while the oldest items get older.
- "It resolved itself." Verify that the messages were processed rather than expired at a retention limit. Those look identical on a depth chart and are opposite outcomes.
- "Queues prevent overload." Queues *absorb bursts*. Under sustained overload they convert an error into a delay, and a long enough delay is an outage with a different name (Backpressure).
Operating it
- Arrival rate and completion rate on one chart, plus their difference. This single view distinguishes "more work arrived" from "we complete less" immediately (Six Queue Signals, Two That Wake You Up).
- Oldest-message age with an alert threshold. Depth alone has no natural threshold because its meaning depends on job duration (Depth Is Not an Emergency; Age Is).
- Attempt-count distribution: a rising tail means work is being retried rather than completed, which points at poison messages or a failing dependency.
- Worker utilisation and count: low utilisation with a growing queue means the constraint is downstream and scaling will not help (Twenty Workers, All Busy, Five Hundred Waiting).
- Downstream latency and saturation alongside the queue chart. A dependency slowdown and a backlog have the same queue signature and different causes (From Symptom to Root Cause).
- Deploy markers on the queue charts. A backlog that starts at a release is a different investigation from one that starts at a traffic peak (Deploys Are the First Suspect).
- At 10x, backlogs become routine daily events at peak, and the design has to make them boring: separate queues, priority classes, and a drain that is a normal operating mode rather than an incident.
- At 10x, per-tenant fairness matters, because one large customer's bulk work can otherwise occupy the whole fleet.
- At 100x, "process everything eventually" stops being achievable during a spike and shedding policy becomes a product decision that someone must own (Backpressure).
- Broker retention becomes a hard limit at scale: a backlog that exceeds it discards the oldest work silently, which is a data-loss event disguised as a recovery.
- Adding workers is the fastest lever and the one most likely to convert a queue problem into a database problem.
- Shedding recovers immediately and permanently discards work someone may be waiting for.
- Processing newest-first keeps current work timely and starves the oldest items, sometimes indefinitely.
- Separate queues prevent starvation and multiply the number of things to monitor, scale and alert on.
- Rate-limited drain protects the recovering dependency and takes longer, which means a longer period of user-visible delay.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALArrivals exceeding completions, and the four causes, apply to any queue on any broker.
- CLOUD-SPECIFICWhat a backlog does at its limit differs. SQS retains messages up to a configurable maximum (14 days at the documented ceiling) and then deletes them. Pub/Sub retains unacknowledged messages for a configurable window. Kafka retains by time or size and drops the oldest segments regardless of whether consumers have read them, so a consumer that falls behind retention loses data and must reset its offset. RabbitMQ can be configured to reject publishes or drop the oldest messages when a queue length limit is reached. "The backlog cleared" therefore means different things on different products.
- SIMPLIFIEDTreats the queue as FIFO with uniform jobs. Priority queues, per-tenant fairness and mixed job durations all change what depth and age mean, and a real triage has to account for them.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.