Depth Is Not an Emergency; Age Is
A million tiny jobs and ten thousand hour-long jobs produce wildly different dashboards from the same word, "backlog". Depth measures accumulation; oldest-message age measures how long a human has been waiting. Only one of them belongs on a pager.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Same number, two different days
Two queues report a depth of 10,000. The first processes thumbnail resizes at 2,000 jobs/s; its backlog represents five seconds of work and the oldest item has been waiting under a second. The second processes nightly report generation at 0.5 jobs/s; its backlog represents five and a half *hours* of work and the oldest item was enqueued before lunch.
Depth cannot distinguish these because depth has no time in it. It is a count of items, and the thing users care about is duration. The conversion factor between them is per-job cost, which varies by orders of magnitude across queues in the same system — and often varies within a single queue, if the job types are mixed.
Age is already in the user's units. "The oldest unprocessed job has been waiting 4 minutes 10 seconds" can be compared directly against "we promise export emails within 60 seconds" without any arithmetic or workload knowledge. That is the property that makes it alertable: the threshold comes from a product promise rather than from someone guessing a round number for a count.
1alert: QueueBacklogHigh2 expr: queue_depth > 500003 for: 5m4 5# Where did 50,000 come from? Someone eyeballed a graph in 2023.6#7# Fires at 3am for the thumbnail queue: 25 seconds of work. Nobody is hurt.8# -> on-call learns to ignore it ([[alert-fatigue]])9#10# Silent for the reconciliation queue at depth 9,000: 5 hours of work,11# finance is blocked, no page.12#13# Silent when consumers crash and depth *falls* to zero.1alert: QueueTooOld2 expr: queue_head_age_seconds{priority="user_facing"} > 603 for: 2m4 annotations:5 slo: "Export emails delivered within 60s of request"6 7alert: QueueTooOld8 expr: queue_head_age_seconds{priority="batch"} > 18009 for: 10m10 annotations:11 slo: "Nightly reports available by 06:00"12 13# Same metric, different thresholds, each traceable to a commitment.14# Fires when someone is actually waiting. Keeps firing if consumers die.The depth threshold is unanchored — it cannot be defended in review and it is wrong for every queue except the one it was tuned against. The age threshold is derived from a promise, so it is defensible, portable across queues, and it stays meaningful when consumers stop consuming entirely.
Age is a distribution too
Head-of-queue age is the maximum: the single unluckiest job. That makes it the right alerting signal, because it is the first thing to breach a promise and the last thing to recover. But it is a poor summary of typical experience — one poison message stuck at the head can report an age of two hours while every other job flows through in milliseconds.
So publish both. Head age answers "is anyone badly hurt?" and belongs on the pager. A percentile of *completed* wait time — p50 and p95 of dequeued_at − enqueued_at — answers "what is the normal experience?" and belongs on the dashboard beside it. When head age is high and p95 wait is low, you are looking at a stuck head, not a capacity problem, and the fix is entirely different: find and dead-letter the poison message rather than scaling workers.
This is the same reasoning as Percentiles: Which One, and How Many Users Is That? and The Average Was Fine and Users Were Not applied to waiting rather than serving. The queue does not have "a" latency; it has a distribution of waits, and the two ends of it fail in different ways for different reasons.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| queue depth | 11,400 | Moderate and stable. Not growing, so arrivals and processing are balanced. | normal |
| head-of-queue age | 2h 14m | The oldest item has not moved in over two hours while the queue keeps flowing around it. | smoking gun |
| p50 wait (completed) | 180 ms | Typical jobs are being served promptly — capacity is fine. | normal |
| p95 wait (completed) | 900 ms | Tail is unremarkable. Nothing suggests a broad slowdown. | normal |
| redelivery count (head msg) | 47 | One message has been delivered and un-acked 47 times — it is failing and returning to the head. | smoking gun |
| dead-letter rate | 0/s | Nothing reaches the DLQ, so the redelivery limit is unset or unreachable. The poison message will retry forever. | suspect |
What the number should be compared against
An age threshold is only as good as the promise behind it, so the useful work is upstream of the alert: writing down, per job class, how late is too late. "Password reset emails: 30 seconds." "Export ready notifications: 60 seconds." "Search index updates: 5 minutes." "Weekly analytics rollup: by 06:00." Each of those is a sentence someone in product can confirm or correct, which is exactly what an unanchored depth threshold can never be.
Those sentences also do double duty as the SLIs: Measuring What the User Actually Feels definitions for asynchronous work. Synchronous request latency has an obvious SLI; async work is where teams typically have none, which is why async lateness so often goes unmeasured until a customer notices. The queue is the measurement point, and head age per class is the metric.
Where the queue is partitioned, age must be per partition for the same reason lag must be: a single stalled partition has an age climbing linearly while the aggregate looks unremarkable. Where jobs are mixed priorities in one queue, the honest options are to split the queue or to accept that your age metric describes the worst class only.
| Head age | p95 completed wait | Depth trend | Diagnosis |
|---|---|---|---|
| Low | Low | Flat | Healthy. Depth value is irrelevant. |
| High | Low | Flat | Stuck head — poison message or a partition stall. Do not scale workers; find the message. |
| High | High | Rising | Genuine capacity shortfall — the The Backlog Arithmetic: Four Levers and a Drain Time arithmetic applies. |
| Low | High | Flat | Bursty arrivals: waits spike within bursts but drain fast. Look at arrival burstiness, not average rate. |
| High | No data | Falling | Consumers are gone. Depth falling is not recovery — nothing is being dequeued to measure. |
Key points
- Depth counts items; users experience duration. The conversion factor is per-job cost, which varies by orders of magnitude between queues.
- Age is already in user units, so its threshold comes from a product promise rather than from a guessed round number.
- Head-of-queue age is the maximum and belongs on the pager; percentiles of completed wait describe the typical experience and belong beside it.
- High head age with low p95 wait means a stuck head — a poison message or stalled partition — and scaling workers will not touch it.
- Falling depth with no wait samples means consumers are gone, not that the incident is over.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Poison message → head: a job fails mid-processing and is redelivered without being acknowledged, returning to the head of the queue each time.
- 2Head → age metric: head-of-queue age climbs linearly and never resets, because the same message keeps occupying position zero.
- 3Queue → other jobs: on a non-ordered queue the rest flow past normally, so p50 and p95 completed waits stay flat and depth stays stable.
- 4Redelivery limit → DLQ: with no maximum-delivery limit configured, the message never dead-letters, so the DLQ rate stays at zero and no alert fires there.
- 5Age → users: whichever user owns that job waits indefinitely, while every dashboard except head age reports a healthy queue.
- • "Depth is only 11,000, we are fine" — depth is irrelevant when one message at the head has been stuck for two hours.
- • "Head age is climbing, add workers" — if p95 completed wait is flat, capacity is not the constraint and workers change nothing.
- • "Zero dead letters means nothing is failing" — it can equally mean the redelivery limit is unset and failures retry forever.
- • "Depth dropped, incident over" — depth also drops when consumers die; check that wait samples are still arriving.
- • "Our async work has no SLO because it is background" — the queue is exactly where an async SLI can be measured; the absence is a choice, not a constraint.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Head-of-queue age per priority class and per partition, sampled on a timer so it survives a total stall.
- • p50 and p95 of completed wait (`dequeued_at − enqueued_at`) alongside head age, to distinguish a stuck head from a broad shortfall.
- • Redelivery count on the head message — a high count with zero dead letters is a poison message with no escape hatch.
- • A written maximum acceptable age per job class, kept next to the alert definition as its justification.
- • Alert on head-of-queue age per class against a written promise, and demote depth to a supporting graph.
- • Configure a maximum delivery count so poison messages dead-letter instead of occupying the head forever, and alert on DLQ arrivals.
- • Publish completed-wait percentiles beside head age so a stuck head is distinguishable from a capacity shortfall at a glance.
- • Split mixed-priority queues, or accept explicitly that the age metric describes only the strictest class in the queue.
- • Write the maximum acceptable age per job class into the alert annotation, so the next responder knows what promise is being broken.
- • Inject a deliberately failing message in staging: head age must climb while p95 wait stays flat, and the message must dead-letter at the configured limit.
- • Stop all consumers: head age must keep climbing (proving the timer-based sampler works) rather than reporting no data.
- • Confirm each age alert fires before its documented promise is breached, by comparing fire time to threshold on one timeline.
- • Timer-based head sampling adds broker load proportional to partition count, which on high-partition topics needs its own budget.
- • Per-class and per-partition age labels multiply metric cardinality — the [[cardinality]] cost applies here as everywhere.
- • Dead-lettering poison messages moves the problem rather than solving it: someone must own draining and diagnosing the DLQ, or failures accumulate unnoticed.
- • Age alerts per priority class, each annotated with the promise it defends, reviewed whenever that promise changes.
- • A maximum-delivery-count setting enforced on every queue by configuration policy, so a new queue cannot ship without a poison-message escape hatch.
- • A DLQ age alert, because a dead-letter queue nobody drains is silent data loss.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEDepths, ages and thresholds here are chosen to contrast two shapes. Your thresholds must come from your own per-class promises, not from these figures.
- WORKLOAD-SPECIFICWhether a stuck head blocks the rest of the queue depends on the broker: strictly ordered partitions block everything behind the stuck message, while unordered queues flow around it. Both cases appear in the matrix above.