Concurrency Limits: An Unbounded Server Is a Slower Server
Accepting every request that arrives feels generous and produces the worst possible outcome: everything is slow, everything times out, and the capacity is spent on work nobody is still waiting for. A limit is a latency control, not just a safety valve.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
What unbounded acceptance actually buys
A server with no concurrency limit does not serve more requests; it serves the same number more slowly, and spends the difference on requests that will be abandoned. The arithmetic is unforgiving: if capacity is 500 concurrent requests worth of work and 5,000 arrive, each of the 5,000 gets roughly a tenth of the service rate. Every one of them crosses the client timeout. Zero useful responses are delivered, and the machine was busy the entire time.
Bound the same system at 500 and the outcome inverts: 500 requests are served at normal latency and succeed, while 4,500 receive an immediate rejection. Nine out of ten users get an error instead of a timeout — but one in ten gets a working product, and, crucially, the error arrives in milliseconds rather than after a 30-second wait. Fast failure also lets clients fail over, retry elsewhere, or degrade gracefully; a slow failure denies them that option too.
This is the counter-intuitive core of admission control: rejecting work is how you preserve the ability to do work. It is the same insight as Backpressure viewed from the server's side, and the reason Queueing: Why Systems Get Slow Before They Get Broken warns against solving overload with a bigger queue.
1capacity ~500 concurrent requests2arrivals 5,000 concurrent3limit none — every request accepted4 5result:6 in flight 5,0007 service rate/req ~1/10th of normal8 p99 latency 32 s9 client timeout 5 s10 useful responses ~0 <-- all work discarded11 CPU 100% busy the whole time12 13# The machine worked as hard as it could and produced nothing.1capacity ~500 concurrent requests2arrivals 5,000 concurrent3limit 500 in flight, reject beyond4 5result:6 in flight 5007 service rate/req normal8 p99 latency 240 ms9 rejections 4,500 (immediate 503 + Retry-After)10 useful responses ~500/interval, all delivered11 CPU 100% busy doing work that lands12 13# Same hardware. Same load. The limit is the only difference.Nothing about the capacity changed — only what happens to work beyond it. Unbounded acceptance converts a capacity problem into a total outage; a limit converts it into partial, fast, honest degradation that clients can respond to.
Choosing the number
The starting point is Little's Law: the concurrency you can support at your target latency is throughput × target_latency (Little's Law as Working Intuition). If the service handles 2,000 rps at a 100 ms target, that is 200 concurrent requests. Measure rather than assume the throughput term — it is the capacity at the latency objective, not the peak from a load test (Throughput: The Number That Means Nothing Without a Latency Bound).
Static limits are simple and adequate for many services, but they are wrong whenever capacity changes: a slower dependency reduces effective capacity, so a limit that was right yesterday admits too much today. Adaptive schemes address this by inferring the limit from observed latency — increasing it while latency stays flat and decreasing it when latency rises — which is conceptually the same control loop that TCP congestion control runs on a network path (Congestion Control: Protecting the Network).
Whichever you choose, the limit belongs at more than one level. A global limit protects the process; per-dependency limits stop one slow downstream from consuming every worker; per-tenant limits stop one customer from crowding out the rest (Rate Limiting). A single global limit lets one saturated dependency absorb the entire budget while everything else starves.
| Level | Bounds | Protects against | Cost of omitting it |
|---|---|---|---|
| Global in-flight | Total concurrent requests in the process | Overload turning into total collapse | Everything slow, nothing delivered |
| Per-dependency | Concurrent calls to one downstream | One slow dependency consuming all workers | A degraded downstream takes the whole service with it |
| Per-tenant / per-key | Concurrency or rate per caller | One customer crowding out all others | A single client's burst is everyone's outage |
| Per-endpoint class | Expensive operations separately from cheap ones | Slow exports starving fast reads | One heavy endpoint sets the latency for all of them |
| Queue bound | How much may wait, not just how much runs | Unbounded waiting that guarantees timeouts | Work accepted that can never be served in time |
Shedding well
Once you shed, *what* you shed becomes a design decision. Uniform random rejection is the simple default and treats all work as equally valuable, which it rarely is. Prioritised shedding — dropping background refreshes and prefetches before interactive requests, or protecting checkout while degrading recommendations — preserves far more user value at the same capacity. This requires the request to carry enough information to classify it, which is a contract question as much as a runtime one (API Performance: The Levers You Actually Own).
The rejection itself should be a good citizen: a clear status, a Retry-After that spreads returning clients rather than synchronising them, and a response cheap enough that shedding does not itself become expensive. A rejection path that allocates, logs verbosely and serialises a large error body can consume the very capacity it was meant to protect.
Finally, deadline propagation makes shedding smarter: if a request arrives already carrying a deadline that has nearly expired, the correct action is to reject it immediately rather than start work that cannot finish in time. This is the same principle as not serving abandoned work in Queueing: Why Systems Get Slow Before They Get Broken, applied at admission rather than at completion (Timeouts: The Latency Contract Nobody Writes Down).
- Shed by priority, not at random — background and prefetch traffic before interactive requests.
- Make rejection cheap — a costly error path spends the capacity that shedding was protecting.
- Return `Retry-After` with jitter so rejected clients do not return in a synchronised wave (Retry Storms: The Load You Generated Yourself).
- Reject on expired deadlines at admission rather than starting work that cannot land in time.
- Distinguish shed from failed in metrics — deliberate rejection and genuine errors need separate lines, or your error-rate SLI becomes unreadable (SLIs: Measuring What the User Actually Feels).
Key points
- An unbounded server under overload does the same amount of work and delivers almost none of it, because everything crosses the client timeout.
- A concurrency limit is a latency control: it decides how many requests share capacity, and therefore how fast each is served.
- Size the limit from
throughput × target_latency, using capacity at the latency objective rather than peak throughput. - Bound at several levels — global, per-dependency, per-tenant, per-endpoint-class — because one global limit lets a single slow dependency absorb everything.
- Shed by priority, make rejection cheap, add jitter to
Retry-After, and count shed traffic separately from errors.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Spike → service: arrivals jump to roughly ten times sustainable concurrency; no limit exists so all are accepted.
- 2Acceptance → sharing: capacity is divided across ten times as many in-flight requests, so each proceeds at roughly a tenth of normal speed.
- 3Sharing → timeouts: every request exceeds the 5 s client timeout; clients disconnect while the server continues processing.
- 4Timeouts → waste: completed responses are written to closed connections; useful throughput approaches zero while CPU stays pinned.
- 5Root cause → team: absence of an admission bound converted a partial-capacity event into a complete outage.
- • "We are not rejecting anything, so we are handling the load" — accepting is not serving; measure responses delivered before the client gave up.
- • "CPU is at 100%, so we are at capacity and doing our best" — the machine is busy producing responses nobody receives.
- • "Rejecting requests will make the outage worse" — it converts a total outage into partial service, which is strictly better for users and for recovery.
- • "One global limit is enough" — a single slow dependency will consume the entire global budget while every other endpoint starves.
- • "Shedding is an error, so it belongs in the error rate" — mixing deliberate rejection into the error SLI makes both numbers uninterpretable.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Track accepted, completed, and *delivered-before-client-timeout* separately — the gap between the last two is wasted capacity.
- • Record in-flight concurrency as a gauge against the configured limit, so time spent at the bound is visible ([[littles-law]]).
- • Measure rejection rate and reason, split from genuine error rate.
- • Instrument per-dependency concurrency to see whether one downstream is consuming a disproportionate share of workers.
- • Track deadline-expired-at-admission counts, which quantify work you correctly refused to start.
- • Add a global in-flight limit sized from `throughput × target_latency`, with fast rejection beyond it.
- • Add per-dependency limits so one degraded downstream cannot occupy every worker ([[circuit-breaker]]).
- • Bound the wait queue as well as the running set, so work that cannot be served in time is never admitted.
- • Implement priority-aware shedding and deadline checks at admission.
- • Consider an adaptive limit that infers capacity from latency, for services whose effective capacity varies with dependency health.
- • Re-run the overload test and confirm delivered-before-timeout throughput is now substantially above zero, and p99 for admitted requests is near baseline.
- • Verify rejections appear as rejections in telemetry and are excluded from the error SLI.
- • Check that per-dependency limits engage during a simulated slow dependency and that unrelated endpoints stay healthy.
- • Confirm the rejection path itself is cheap under load, by measuring CPU cost per rejection.
- • Shedding means some users see failures during overload; that is a deliberate product trade, and it needs to be an explicit decision rather than an accident.
- • Limits set too low waste capacity and reject work the system could have served — under-utilization is a real cost.
- • Adaptive limits add a control loop that can oscillate or mis-infer capacity, and are harder to reason about during an incident than a static number.
- • Multi-level limits multiply configuration and the number of ways a request can be refused, which complicates debugging.
- • Alert on sustained time-at-limit, which indicates the limit is now binding routinely and capacity planning needs revisiting.
- • Add an overload scenario to load testing that asserts non-zero useful throughput and bounded latency for admitted work (Load Test Shapes: The Shape Is the Hypothesis).
- • Review the limit whenever dependency latency changes materially, since effective capacity moves with it.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe bounded/unbounded comparison uses invented numbers chosen to make the mechanism legible. The qualitative outcome — unbounded overload delivering near-zero useful throughput — is well established; the specific latencies are not measurements.
- WORKLOAD-SPECIFICThe right limit depends on service-time distribution, dependency latency and the client timeout. A limit derived for one traffic mix can be badly wrong for another.
Misconceptions
Apply it
Where the depth lives
Adaptive concurrency limits are a controller: latency is the measured output, the limit is the actuator, and the loop must be damped to avoid oscillation. The same stability concerns that apply to any feedback controller apply here, which is why aggressive adaptation can be worse than a static bound.