The question this answers
Should this request be let in at all, given what I already owe?
Bounded in-flight work and therefore bounded queueing delay: with a concurrency limit L and service rate μ, no admitted request waits longer than roughly L/μ. It does not guarantee that admitted requests succeed, and the bound holds only while μ is what you assumed — a dependency slowdown lowers μ and the same limit now admits far too many.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
The admitting node knows how many requests it currently has in flight, how long recent ones took, and how long the oldest queued item has waited. It does not know the offered load it is about to receive, the true service rate of a downstream that is degrading, or how many sibling instances are making the same decision against the same shared database.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
What an unbounded queue actually converts overload into
The seductive property of a queue is that it never says no. That is also the problem. At an arrival rate above the service rate, the queue length grows without bound and, by Little’s Law, so does the waiting time: W = L / λ. Nothing in the system objects until either the deadline or the heap gives out.
Run the numbers. A service handles 100 requests per second and callers time out at 2 seconds. Anything past position 200 in the queue is already worthless when it is dequeued. If arrivals run at 150/s for one minute, the queue holds 3,000 items — a 30-second wait — and 93% of the work the service does is for callers who left. Throughput looks fine. Goodput is near zero.
Then the second failure arrives: each queued item holds a request body, a connection, and a context. At 3,000 items of 100 KB, that is 300 MB of live objects the collector cannot free. The service OOMs, drops every in-flight request including the ones it could have served, restarts cold, and the queue that was waiting on it now points at fewer instances. This is the standard progression from overload to outage, and its root cause is one missing bound.
unbounded queue limit L = 200 peak queue length 3,000 items 200 items peak wait (p50) 15.0 s 1.0 s requests served 6,000 6,000 served within deadline ~420 6,000 <- goodput rejected at the door 0 3,000 peak heap from queued bodies 300 MB 20 MB outcome OOM, restart steady, degraded
Three signals to admit on, in increasing order of honesty
A fixed concurrency limit is the simplest: allow L requests in flight, reject or briefly queue the rest. It is a semaphore, it is local, and it costs nothing to evaluate. Its weakness is that L is a guess about a service rate that changes — the right limit when the database is warm is far too high when the database is failing over.
A latency-driven limit fixes that by treating the limit as a control variable. Measure a short-window RTT against a long-window minimum; if the ratio degrades, shrink the limit multiplicatively; if it is healthy, grow it additively. This is the same AIMD shape as TCP congestion control, and it discovers capacity rather than assuming it. The cost is a control loop with a time constant you must now reason about.
A queue-age limit is the most honest of the three because it admits on the only thing that actually matters: whether the work can still be delivered in time. Drop anything that has waited longer than its remaining budget. Combined with Pass the Remaining Budget Down, Not a Fresh One this needs no tuning at all — the deadline the caller sent *is* the threshold.
One counter-intuitive companion trick: under overload, serve the queue LIFO rather than FIFO. FIFO under overload maximises the number of requests that time out, because everyone waits the full backlog. LIFO serves the newest — which still has budget — and lets the oldest expire, so a smaller fraction of requests miss their deadline. It feels unfair, and it is; it is also the difference between some users being served and none.
1// Additive-increase / multiplicative-decrease on observed latency.2// Same shape as TCP congestion control, for the same reason: no participant3// can observe the shared bottleneck directly, only its own delay.4class AdaptiveLimit {5 private limit = 206 private inFlight = 07 private minRttMs = Infinity // long-window best case ~ the no-queue latency8 9 tryAcquire(): boolean {10 if (this.inFlight >= this.limit) return false // reject at the door11 this.inFlight++12 return true13 }14 15 release(rttMs: number, dropped: boolean) {16 this.inFlight--17 this.minRttMs = Math.min(this.minRttMs, rttMs)18 19 if (dropped) { // timeout or 5xx: back off hard20 this.limit = Math.max(4, this.limit * 0.7)21 return22 }23 // Queueing delay, inferred: how much of this RTT was waiting?24 const queueRatio = rttMs / this.minRttMs25 if (queueRatio < 1.5 && this.inFlight >= this.limit - 1) {26 this.limit += 1 // only grow when actually saturated27 } else if (queueRatio > 3) {28 this.limit = Math.max(4, this.limit * 0.9)29 }30 }31}Admission control is a local decision about a shared resource
Every instance limits itself, but the thing being protected is usually shared — one database, one downstream service, one connection pool on the other side. Fifty instances each admitting 20 concurrent requests present 1,000 concurrent requests to a database sized for 200. Each instance is behaving correctly and the aggregate is an overload, which is the signature distributed-systems failure: correct local decisions composing into a wrong global outcome.
You have three ways out, and all of them cost something. Divide the global budget by the instance count — cheap, and wrong the moment the fleet scales or load balancing is uneven. Put a shared counter on the path — accurate, and it adds a dependency that must survive the overload it is regulating. Or let the shared resource push back and have callers adapt, which is where the latency-driven limit shines: it converges on the *right* local number without anyone computing it, because congestion at the shared resource shows up as latency at every caller simultaneously.
The last option is the one that generalises, and the reason is worth naming: it needs no agreement. Each caller reacts to a signal it can measure locally, and the aggregate settles near capacity. That is Coordination Avoidance: Restructuring the Problem Instead of Paying for It applied to load, and it is why the AIMD shape keeps reappearing wherever many independent senders share one bottleneck.
Key points
- An unbounded queue converts overload into latency collapse, then into a memory failure — it does not absorb anything.
- By Little’s Law, bounding in-flight work bounds waiting time; that is the whole guarantee, and it is the one that saves goodput.
- Fixed limits assume a service rate. Latency-driven (AIMD) limits discover it. Deadline-driven admission needs no tuning at all.
- Under overload, LIFO service order maximises the number of requests that still meet their deadline — unfair, and better than serving nobody.
- Local limits at many instances compose into a global overload of a shared resource; latency-driven limits converge without coordination.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • A request arrives and the node compares its current in-flight count against a limit.
- • If under the limit, admit and increment; if over, either reject immediately or place in a small bounded queue with an expiry.
- • On completion, record the observed latency and outcome, and adjust the limit if it is adaptive.
- • Queued items are checked against their remaining deadline at dequeue time and dropped if expired, before any work is done.
- • Rejections return an explicit retryable status, distinct from a failure of the work itself.
- • The downstream service rate collapses, so a fixed limit that was sized for health now admits far too much.
- • The adaptive limit’s control loop oscillates: it shrinks on a latency spike, recovers, grows, and re-triggers the spike.
- • The bounded queue is bounded in count but not in bytes, and large request bodies exhaust memory well before the count limit.
- • The limit is per-process while the resource is per-host, so a container with eight workers presents eight times the intended concurrency.
- • The rejection path is not counted separately and the incident looks like a spike in server errors.
- • Latency collapse with full success rate: p99 sitting exactly at the client timeout, throughput at capacity, and a queue-depth chart that has been climbing for twenty minutes. Every request is served; almost none in time.
- • OOM under load: the process dies with a heap dominated by buffered request bodies and pending contexts. Restart, cold caches, reduced fleet, and the queue that survived now targets fewer instances.
- • Limit-induced brownout: an adaptive limit backs off on latency caused by a slow *dependency*, throttling all traffic including requests that never touch it. Operator sees rejections on endpoints with a healthy downstream.
- • Aggregate overload from correct local limits: every service instance reports in-flight well under its limit, and the database reports connection saturation. Nobody is over their limit and the resource is over capacity.
- • Purely local admission requires no coordination and provides no aggregate guarantee — that is the trade, and it is usually the right one.
- • A global concurrency budget requires a shared counter with a round trip per admission, which adds latency to every request and a dependency that must outlive the overload.
- • Adaptive limits reach an approximately correct global outcome with no coordination at all, by having every participant react to a shared congestion signal. Convergence is not agreement, and it can be unfair between fast and slow callers.
- • Admitted work keeps its full correctness and latency guarantees; that is precisely what the bound buys.
- • Rejected work is rejected explicitly and immediately, so the caller can fail over, degrade, or shed rather than wait.
- • The process stays alive, which preserves warm caches, established connections, and the ability to recover without a cold start.
- • Detect: queue depth and queue age together — depth alone cannot distinguish a healthy buffer from a collapse in progress.
- • Contain: bound in-flight count and total queued bytes. A count limit without a byte limit still lets one large-payload endpoint exhaust the heap.
- • Recover: an adaptive limit reopens automatically as latency normalises; a fixed limit needs nothing at all, which is a genuine operational advantage.
- • Reconcile: rejected requests that mattered must be re-driven by their owner. Admission control assumes the caller has a plan for a "no", and that assumption is often false.
- • Verify: load-test past capacity and confirm that p99 for admitted requests stays flat while rejections rise. A rising p99 under load means the bound is not binding.
- • In-flight count against the current limit, and the limit itself as a time series — an adaptive limit that never moves is not adapting.
- • Queue age at dequeue, and the count of items dropped because their deadline had already passed.
- • Rejections at admission, counted separately from downstream failures, so an incident graph distinguishes "we said no" from "it broke".
- • Goodput: responses returned inside the caller’s deadline. Admission control is defensible only if this number goes up.
- • Any synchronous service in front of a resource with a hard concurrency ceiling — a database, a connection pool, a thread pool, a licensed downstream.
- • Systems where requests have deadlines, because expired work can then be discarded at zero judgement cost.
- • Fleets large enough that aggregate concurrency, not per-instance CPU, is the binding constraint.
- • Long-lived streaming or WebSocket connections, where "in flight" is not a meaningful unit and a concurrency limit measures the wrong thing.
- • Highly heterogeneous workloads where one request costs a thousand times another — a count-based limit is nearly meaningless and you need a cost-weighted budget.
- • When the real problem is a dependency’s latency: throttling your own traffic masks the symptom, and the adaptive limit will happily throttle you to nothing while the actual fault goes uninvestigated.
- • A fixed concurrency semaphore, tuned once from a load test. Crude, stable, has no control loop to misbehave, and is right far more often than its reputation suggests.
- • Rate limiting at the edge: bounds arrivals rather than in-flight work. Simpler to reason about across a fleet, but blind to how expensive each request turns out to be.
- • Load shedding by priority, which chooses *which* work to drop instead of just how much. The two compose well — see Rejecting Work on Purpose — and Rejecting It Cheaply Enough to Help.
- • Asynchronous acceptance with a durable queue and an explicit lag budget, for work whose value does not expire in seconds.
Admission control: find out at the door
| Admit on | Bound | Completed | Goodput | Refused | Steady wait |
|---|---|---|---|---|---|
| no admission control | ∞ | 9,000 | 150 | 0 | ∞ |
| queue-depth bound | 400 | 9,000 | 150 | 8,750 | 2.67 s |
| concurrency limit | 6 | 360 | 360 | 17,640 | 40 ms |
| latency / deadline bound | 150 | 9,000 | 9,000 | 9,000 | 1.00 s |
What people believe, and what is true
A queue protects the service from spikes.
A queue protects against a *burst* — arrivals above capacity for a bounded time with an idle period after. Against sustained overload it protects nothing and destroys goodput.
We are fine, throughput is at capacity.
Throughput at capacity with a deep queue means you are doing full-rate work for callers who have already timed out. Measure goodput.
FIFO is the fair order.
Under overload FIFO maximises deadline misses, because every request waits the full backlog. LIFO with expiry serves more people, at the cost of the oldest requests never being served at all.
Our concurrency limit is 20, so the database sees 20.
It sees 20 × instance count × workers per instance. Local limits are local; the resource is shared.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
Check at the door whether you can actually do the work. If you cannot, say no now — saying no in two seconds is much worse than saying no immediately.
Practical
Bound in-flight work with a semaphore, bound the queue in both items and bytes, drop anything at dequeue whose deadline has passed, and report rejections separately from errors. Verify with a load test that admitted-request p99 stays flat as offered load doubles.
Advanced
Make the limit adaptive on queueing delay inferred from the ratio of current RTT to observed minimum RTT — this is Vegas-style congestion control applied to RPC. Then reason about it as a control loop: the increase must be slower than the decrease, and the loop period must be shorter than the client retry interval, or your limiter and their retries will phase-lock into oscillation.
Internals
Under sustained overload the service rate μ is not constant: deeper queues mean more resident memory, worse cache locality, more GC pressure and more context switching, so μ falls as the queue grows. That positive feedback is why latency collapse is so abrupt — the system is not gliding down a linear curve, it is crossing a knee where the thing you are queueing for gets slower because you queued for it.
Apply it
- 💬 Your service has an unbounded work queue and no errors during an incident, but users report failures. Explain what happened.
- 💬 Fifty instances each cap concurrency at 20 against a database sized for 200. Every instance is under its limit and the database is saturated. What do you change?
- 💬 Why might LIFO be the right queue order under overload?