Queue & Worker Performance
Arrival rate against service rate, why depth alone is the wrong alarm, oldest-message age as the honest signal, retry storms that feed themselves, and worker pools that saturate quietly.
Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.
Arrival rate, processing rate, depth, oldest-message age, retry volume and dead-letter volume. Depth is the number everyone graphs and the number that explains the least; the rate pair tells you whether you are falling behind, and age tells you whether a human is already suffering.
10,000 jobs/s arriving, 8,000/s processed, backlog growing at 2,000/s. The gap is arithmetic, not opinion — and there are exactly four things you can do about it. Time-to-drain is the number to put in the incident channel.
A million tiny jobs and ten thousand hour-long jobs produce wildly different dashboards from the same word, "backlog". Depth measures accumulation; oldest-message age measures how long a human has been waiting. Only one of them belongs on a pager.
A dependency gets slower, clients retry, the retries become load, the dependency gets slower still. The feedback loop is what turns a 5% error rate into an outage — and it is the one source of traffic you can switch off yourself.
Every worker is occupied and the queue is 500 deep, so the obvious move is more workers. Whether that helps depends entirely on what the workers are busy *doing* — and if they are waiting on a shared dependency, adding workers makes things worse.