The question this answers
I cannot serve everything. Which requests do I drop, and how do I drop them without spending the capacity I am trying to save?
Core function is preserved above a stated overload factor, at the cost of a defined class of work being rejected with an explicit error. It is not a fairness guarantee — shedding deliberately treats requests unequally — and it holds only while the cost of rejecting stays far below the cost of serving.
Everything below is bought to hold this sentence. "Strongly consistent" with no scope attached is a slogan, not a guarantee — read what it actually covers, and what it explicitly does not.
A shedding node knows its own load signal (in-flight, queue age, CPU, or measured latency) and whatever priority the request carries. It does not know the global overload factor, whether the request it just dropped was the user’s third attempt, or whether the caller will retry — so it cannot know whether shedding reduced offered load or merely relabelled it.
A node knows its own state and the messages that arrived. Everything else is inference from evidence that was already stale. "B has not replied in five seconds" is knowledge; "B is down" is a decision — and usually the bug.
The arithmetic that decides whether shedding works at all
Suppose serving a request costs 1.0 units of a saturated resource and rejecting one costs r. At an offered load of k times capacity, the fraction you can still serve is roughly (1 - r·k) / (1 - r) — and once r ≥ 1/k, the answer is zero: you spend the whole machine saying no.
Put concretely: at 3× overload, a rejection that costs 30% of a success leaves you serving nothing. A rejection costing 1% leaves you serving essentially full capacity. The entire viability of load shedding lives in that ratio, which is why where you shed matters more than how you decide to shed.
This is why the expensive work must happen *after* the shed decision, not before. Authenticating, deserialising a 200 KB body, resolving the tenant, opening a database connection to check a quota — every one of those raises r. The cheapest rejection is a connection refused at the edge; the most expensive is a 503 emitted after the handler already did the query.
| Shed point | Relative cost r | Can distinguish | Cannot distinguish |
|---|---|---|---|
| Connection accept / SYN droptypical | ~0.001 | Source IP | User, endpoint, priority — nothing about the request |
| Edge proxy, pre-auth, on path + headerstypical | ~0.01 | Route, method, a priority header, API key | Anything requiring a lookup |
| Service entry, post-authtypical | ~0.05–0.1 | Tenant, plan, user-level priority | Cost of the specific query |
| Inside the handler, before the DB callassumption | ~0.2–0.4 | Everything | Nothing — but at 3× overload this saves nobody |
Knowing what is low priority means somebody had to say so
The hard part of shedding is not the mechanism, it is the classification. A service under load sees a stream of requests that all look equally urgent, because nothing in an HTTP request says "this one is a background prefetch and that one is a checkout". Priority is not discoverable at the point of shedding; it has to be *carried*, the same way a deadline is carried in Pass the Remaining Budget Down, Not a Fresh One.
The pattern that works is a small, fixed criticality ladder assigned at the edge and propagated on every downstream call: CRITICAL (a user is blocked and money is involved), HIGH (a user is blocked), MEDIUM (a user will notice), SHEDDABLE (prefetch, retry, backfill, analytics, recommendation). Four levels, not twenty, because every level you add is a level nobody will maintain.
Two rules keep the ladder honest. First, a call inherits the criticality of its caller — a recommendation service calling the user service must not present as CRITICAL just because the user service is important. Second, retries are one level lower than the original, which makes shedding and Cap Retries as a Fraction of Traffic, Not as a Count per Request cooperate: under load, first attempts win and retries lose, which is exactly the ordering that helps the system recover.
And note the incentive problem: if each team sets its own criticality, everything is CRITICAL within a quarter. The ladder needs an owner outside the calling teams, or it decays into a no-op.
1type Criticality = 'CRITICAL' | 'HIGH' | 'MEDIUM' | 'SHEDDABLE'2 3// Each level is admitted only while the load signal is below its cut-off.4// Cut-offs are on *queue age* (are people still waiting?), not CPU.5const CUTOFF_MS: Record<Criticality, number> = {6 CRITICAL: 400, // shed only when we are about to miss every deadline anyway7 HIGH: 200,8 MEDIUM: 80,9 SHEDDABLE: 20, // the first thing to go, and it goes early10}11 12// Called at service entry, after routing and before auth, body parse or any I/O.13export function admit(req: { criticality: Criticality; isRetry: boolean }, queueAgeMs: number): boolean {14 // A retry is worth strictly less than a first attempt: demote one level.15 const level = req.isRetry ? demote(req.criticality) : req.criticality16 return queueAgeMs < CUTOFF_MS[level]17}18 19function demote(c: Criticality): Criticality {20 return c === 'CRITICAL' ? 'HIGH' : c === 'HIGH' ? 'MEDIUM' : 'SHEDDABLE'21}Shedding changes the demand curve, and not always downward
On a single machine, dropping work reduces work. In a distributed system the dropped request goes back to a machine, and that machine has an opinion. If the client retries on 503, shedding has converted one unit of expensive work into one cheap rejection *plus one more unit of offered load* — and you have not reduced demand, you have increased request rate while reducing goodput. This is the exact mechanism of One Retry per Tier Is Not One Retry — It Multiplies seen from the other end.
So a shedding design is incomplete without a matching client contract: which status code, whether it is retryable, and how long to wait. 503 with Retry-After, or gRPC UNAVAILABLE, says "come back later"; 429 says "you specifically are over budget". The worst choice is a 500, because every sane client treats it as a transient fault worth retrying immediately.
The one shed that is unambiguously free is the request whose deadline has already passed while it sat in your queue. Serving it cannot help anyone — the caller is gone — and dropping it costs a comparison. Any system with Pass the Remaining Budget Down, Not a Fresh One gets this class of shedding almost for nothing, which is a large part of why deadline propagation is worth the plumbing.
Key points
- Shedding is viable only while a rejection is far cheaper than a success; at 3× overload a rejection costing 30% of a success saves nobody.
- Shed as early in the request path as you can while still knowing enough to choose — cost and discrimination trade directly against each other.
- Priority cannot be discovered at the shed point. It must be assigned at the edge and propagated, with retries ranked below first attempts.
- A shed request that the client retries has not reduced demand; the status code and retry contract are part of the shedding design.
- Requests whose deadline has already expired are free to drop and impossible to help — the cheapest and most defensible shedding there is.
The chain, answered
Every field here is required, which is why no lesson in this domain can recommend a design without naming what an operator sees when it fails, what survives the partition, what repairs it afterwards, and the simpler thing to consider first.
- • The edge assigns a criticality to every request and stamps it into the propagated context alongside the deadline.
- • Each service measures a saturation signal that reflects waiting rather than utilisation — queue age is the usual choice.
- • At entry, before auth or body parsing, the service compares the signal against the cut-off for that criticality level and rejects immediately if it is over.
- • Rejections return an explicit, non-ambiguous status with retry guidance, and are counted separately from failures.
- • Cut-offs are tuned so that the highest level sheds only at the point where the system could not have met its deadlines anyway.
- • Criticality is not propagated across a hop, and everything downstream looks equally important.
- • The load signal lags the actual condition, so shedding starts after the collapse rather than before it.
- • The rejection path itself allocates, logs verbosely, or emits a high-cardinality metric per rejection, making
rlarge exactly when it must be small. - • Clients retry the rejection immediately, converting shed load back into offered load.
- • Health checks and control-plane traffic are shed along with user traffic.
- • Shedding the wrong thing: checkout error rate rises while the recommendation carousel stays perfectly available, because recommendations were never labelled sheddable. The operator sees revenue drop while every service reports "degraded gracefully".
- • Shedding that does not shed: CPU pinned at 100%, success rate falling toward zero, and rejection count in the millions — the machine is fully occupied producing 503s. Look for logging or serialisation on the reject path.
- • Priority inversion on the control plane: liveness probes are shed, the orchestrator concludes the instances are dead and kills them, and remaining capacity drops mid-incident. Signature is a restart storm during, not before, the load spike.
- • Shed-and-retry equilibrium: rejection rate and total request rate both climb together and stay high after offered load falls. The system does not recover on its own because its own retries are now the load.
- • Local shedding needs no coordination, and therefore cannot guarantee anything global: each instance protects itself and the aggregate result emerges.
- • Fair shedding across tenants does need shared state — a per-tenant counter that every instance can see — which puts a store on the request path during overload.
- • The usual compromise is local shedding plus a coarse global rate limit at the edge, accepting that neither is exactly fair and that both are cheap.
- • Admitted requests keep their full correctness guarantees — shedding never returns a partial or wrong answer, only a refusal.
- • The refusal is explicit and attributable, which is strictly better than a timeout: the caller learns something.
- • Core function stays inside its latency objective while non-core function is unavailable, which is the trade you signed up for.
- • Detect: track goodput (successful, within-deadline responses) separately from throughput. Shedding raises the first while lowering the second, and only the first matters.
- • Contain: keep control-plane traffic, health checks and admin endpoints on a path that is never shed — see Bulkheads: Buying Independence by Giving Up Utilisation.
- • Recover: as the load signal falls, cut-offs re-admit levels bottom-up automatically; no manual step should be required to stop shedding.
- • Reconcile: sheddable work that still needs doing (analytics events, prefetches) must be either genuinely droppable or routed to a durable queue — decide which, per class, in advance.
- • Verify: run a load test that exceeds capacity by 3× and confirm that critical goodput is flat. If it is not, your rejection path is too expensive.
- • Shed rate broken down by criticality level — the shape tells you whether the ladder is doing anything or everything is labelled critical.
- • Goodput: responses delivered inside the caller’s deadline, which is the only throughput number that correlates with users being served.
- • Cost per rejection, measured as CPU-time on the reject path. This is the number that decides whether shedding works, and almost nobody measures it.
- • Ratio of shed requests that reappear as retries within the backoff window — the direct measurement of whether shedding reduced demand.
- • Services with a genuine mix of criticality, where a meaningful fraction of traffic is prefetch, retry, analytics or background work.
- • Spikes that are short relative to autoscaling time — shedding is the only control that acts in milliseconds.
- • Any system with propagated deadlines, where expired work can be dropped with no judgement call at all.
- • When all traffic really is equally critical: shedding then just picks victims, and capacity or admission-controlled queueing is the honest answer.
- • When rejections are expensive — a TLS handshake plus auth before the decision means you are paying most of the cost anyway.
- • When clients retry aggressively and you have no way to influence them; shedding without a retry contract converts a latency problem into a request-rate problem.
- • Admission control with a bounded queue: admit fewer, queue the rest briefly, and serve everything you admit — better when work is homogeneous and short. See Decide at the Door Whether the Capacity Exists.
- • Autoscaling: add capacity instead of removing work. Right for sustained load, useless inside the first minute.
- • Degrade rather than reject: serve a cached, stale, or reduced-fidelity response, which keeps the user served at lower cost. Often strictly better than a 503 where it applies.
- • Queue non-critical work asynchronously at the edge so it never competes for the synchronous path in the first place — prevention rather than triage.
Load shedding: the arithmetic that decides whether it works
| Shed point | r | Can distinguish | Served at 3.00× |
|---|---|---|---|
| connection accept / SYN drop | 0.001 | source IP only | 100% |
| edge proxy, pre-auth | 0.010 | route, method, priority header, API key | 98% |
| service entry, post-auth | 0.070 | tenant, plan, user priority | 85% |
| inside the handler, before the DB call | 0.300 | everything — and it saves nobody | 14% |
What people believe, and what is true
Load shedding means returning errors, so it is a last resort.
The alternative is not "no errors" — it is timeouts, which are errors that cost you full capacity to produce and tell the caller less.
Shed based on CPU.
CPU at 100% may be perfectly healthy throughput. The signal you want is whether work is *waiting* — queue age or measured latency against a target.
We shed at the handler, where we know the most about the request.
By then you have paid for auth, parsing and possibly a connection. At high overload factors that rejection cost is the difference between shedding working and not working at all.
Shedding reduces load.
It reduces *work*. Whether it reduces load depends on what the client does with the rejection, and a client that retries immediately has increased your request rate.
Go deeper
Only the levels this lesson can honestly fill — a missing level is a claim nobody had.
Overview
When you cannot serve everything, choose what to drop instead of letting timeouts choose. Drop the least important work, and drop it before you have spent anything on it.
Practical
Assign a four-level criticality at the edge, propagate it with the deadline, and gate admission on queue age with a per-level cut-off. Demote retries one level. Reject with a status the client will not immediately retry, and count rejections separately from errors so your dashboards stay legible during an incident.
Advanced
Treat cut-offs as a control loop and check its stability: shedding must respond faster than clients retry, or the two loops phase-lock into an oscillation where load and rejections rise together. Deadline-based shedding is self-stabilising in a way that threshold-based shedding is not, because expired work is dropped in proportion to how far behind you are — the control gain rises exactly with the error.
Apply it
- 💬 Your service is at 4× capacity and you add a 503 shed path. Throughput of 503s is enormous and successes are still near zero. What is wrong?
- 💬 How does a service know which requests are low priority?
- 💬 Why is dropping a request whose deadline has expired qualitatively different from other shedding?