Backpressure
When producers outrun consumers, something has to give. Backpressure is choosing what, instead of letting memory choose for you.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
Work is arriving faster than it can be processed. What should the system do about it?
A traffic spike or a slow dependency must degrade the service predictably rather than turning it into an out-of-memory crash.
Buffer it. Accept everything, put it in a queue, and let the workers catch up when the spike passes. Queues exist to absorb bursts.
Queues absorb *bursts*, not sustained overload. A burst is a temporary excess with a matching quiet period behind it; sustained overload has no quiet period and the buffer grows without limit.
- Queues absorb *bursts*, not sustained overload. A burst is a temporary excess with a matching quiet period behind it; sustained overload has no quiet period and the buffer grows without limit.
- An unbounded in-memory buffer is a memory leak with a business justification. The process grows until it is OOM-killed, and the kill takes every in-flight item with it (Memory Leaks in Backend Services).
- A durable buffer does not crash, so the failure is worse: it grows silently, and by the time anyone notices, the oldest item is hours old and every consumer is working on stale requests nobody is waiting for (Queue Backlog).
- Latency degrades before throughput does. Time in queue grows in proportion to depth, so users see a system that is slow, not one that is full — and slow has no natural alerting threshold (Little's Law as Working Intuition).
- Retries make it worse. Clients time out on the slow response and retry, so overload increases the arrival rate exactly when it should decrease it (Retry Storms).
What is actually happening
- Backpressure is a signal that travels *backwards* against the flow of work: the consumer tells the producer to slow down, and the producer either slows, buffers within a bound, or refuses.
- Without that signal, the only remaining mechanism is buffering, and an unbounded buffer converts a throughput problem into a latency problem and then into a memory problem. Nothing about it is a decision anybody made.
- A bounded buffer is what makes backpressure possible at all. The bound is where the system is forced to choose between blocking the producer, shedding the work, or degrading it (Bounded vs Unbounded Queues).
- There are four responses when the bound is reached, and they are genuinely different products. Block the producer (natural for pull-based consumers and in-process channels). Reject with a 429 or 503 so the client can back off (Rate Limiting). Shed selectively, dropping low-value work to protect high-value work. Spill to durable storage, converting an in-memory problem into a backlog you can see and drain.
- Pull-based consumers give backpressure for free: workers take work at the rate they can handle, so the queue depth is the signal. Push-based delivery does not — the broker or the caller must respect a concurrency limit you configure, and if it does not, your only defence is rejecting.
- Backpressure has to propagate. A service that rejects at its edge but has an unbounded internal buffer between its stages has simply moved the unbounded queue inside itself (Unbounded Concurrency).
Where the pressure goes when nobody chooses
Follow the arrows in order. Work arrives faster than it leaves, so depth grows. Depth is time — each item now waits behind more items — so latency grows. Slow responses cause client timeouts, timeouts cause retries, and retries increase the arrival rate. The loop closes and reinforces itself.
The single decision that breaks the loop is a bound. Once the buffer has a maximum, the system is forced to answer the question it was previously avoiding: block, reject, shed, or spill. Any explicit answer is better than the implicit one, which is "grow until the kernel intervenes".
Four things to do at the bound
Once a buffer is bounded, overflow is a design decision rather than an accident. These four are the complete set, and real systems combine them: reject at the edge, shed by priority inside, block on internal pull-based stages, and spill only where the work is genuinely valuable and genuinely deferrable.
The criteria are what matter here. "Is the producer something I can safely slow down?" separates blocking from rejecting; "is this work still valuable in an hour?" separates spilling from shedding.
What should happen to the next item?
when The producer is a pull-based consumer of something else — a worker reading from a queue, a stream stage — and slowing it down propagates pressure correctly.
cost Latency moves to the producer. Fatal if the producer is a request handler: backpressure on a background path becomes user-visible timeouts.
when The producer is a client that can back off and retry. The default answer for an HTTP ingress path.
cost Visible failures, and clients that ignore the hint retry immediately and make it worse (Rate Limiting).
when Work has genuinely different value: analytics versus checkout, bulk export versus password reset.
cost Requires priority classes in the design and separate queues to act on them; dropped work is gone (Bulkheads).
when The work is valuable, deferrable, and there is a realistic plan to drain it.
cost Converts a memory problem into a backlog problem: stale work, growing age, and a drain that must be scheduled (Queue Backlog).
when Never, in a long-lived process.
cost The kernel makes the decision instead, at a time of its choosing, and takes every buffered item with it (Memory Leaks in Backend Services).
The bounds you already have and never set
Most services already contain half a dozen queues that nobody thinks of as queues. Each has a bound, a default, and an overflow behaviour — and in almost every case the default was accepted rather than chosen.
Going through this list for a real service is a short exercise with a high hit rate. The rows toward the bottom are the ones that turn a dependency slowdown into an out-of-memory crash, because they have no bound at all unless you write one.
| Implicit queue | What fills it | Bound | What happens when it overflows |
|---|---|---|---|
| TCP accept backlog | Connections arriving faster than accept() | OS-level, per listener | Connections refused or dropped at the kernel (Accepting Connections) |
| In-flight request count | Concurrent requests being handled | Usually none by default | Memory and scheduler pressure until the process degrades (Resource Limits) |
| Connection pool wait queue | Requests waiting for a free connection | Pool size plus an optional wait timeout | Requests block for the timeout, then fail — often with a confusing error (Connection Pool Exhaustion) |
| Worker prefetch buffer | Messages claimed but not yet processed | A broker or client setting | Messages sit invisible and may exceed their lease (Queue Semantics) |
| Outbound HTTP concurrency | Calls to a slow third party | None unless you add a limiter | Unbounded pending promises, socket exhaustion (Unbounded Concurrency) |
| An in-memory batch array | Items pushed in a loop before a flush | None — it is just an array | Heap growth until OOM; looks exactly like a leak |
| The durable job queue | Producers outrunning workers | Broker limits, often very large | Silent backlog growth and stale work (Queue Backlog) |
How to build it
Most important first.
- Bound every queue and every buffer, including the implicit ones: connection pool wait queues, in-flight request counts, worker prefetch, and any in-memory array you push onto in a loop (Resource Limits).
- Decide the response at the bound explicitly, per queue. Blocking, rejecting, shedding and spilling are four different product behaviours and the right one differs by workload.
- Reject early and cheaply. A request rejected at the edge costs almost nothing; one rejected after authentication, validation and a database round trip has already consumed the capacity you were protecting (Authenticate First, or Rate-Limit First?).
- Return 429 or 503 with a
Retry-Afterheader so well-behaved clients back off rather than retrying immediately (Status Codes From the Server's Side). - Shed by value, not by arrival order. Analytics events before checkout requests; a bulk export before a password reset. That requires the queues to be separated in the first place (Bulkheads).
- Bound concurrency toward every downstream dependency, so that a slow dependency causes a bounded queue in front of it rather than an unbounded pile of waiting tasks (Circuit Breakers).
- Make queue depth and oldest-age first-class signals with alerts, because backpressure that is being applied is a load-management event that someone should know about (Six Queue Signals, Two That Wake You Up).
What can go wrong
- Unbounded in-memory buffering ending in an OOM kill that loses everything buffered.
- A bounded queue whose overflow behaviour was never chosen, so the library's default (often: block forever, or throw) becomes the design.
- Blocking the producer in a request handler, so backpressure on a background path turns into request timeouts on the front path.
- Rejecting after the expensive work, so shedding costs almost as much as serving.
- Shedding indiscriminately, dropping the checkout and keeping the analytics event.
- Backpressure applied at one stage and absent at the next, so the unbounded queue simply relocates.
- Clients that ignore 429 and retry immediately, converting rejection into a tighter loop (Retry Storms).
- Spilling to durable storage with no drain plan, which is how a memory problem becomes a permanent backlog.
- Multiple producers filling the last slot of a bounded buffer concurrently — the bound check must be atomic with the insert, not a check followed by a push (The Atomicity Illusion).
- A shed decision racing a scale-up, so work is dropped in the same second that capacity arrives.
- Producer blocking while holding a database connection or a lock, converting a throughput problem into a deadlock (Connection Pool Exhaustion).
- A consumer draining while a producer refills at the same rate, so depth looks stable while age climbs — the state that hides a problem best.
- Absent backpressure is a denial-of-service amplifier: an attacker who can enqueue faster than you can consume can exhaust memory or fill a durable queue for the cost of sending requests (Rate Limiting).
- Rate limit and quota on the *enqueue* path, per tenant. A shared queue with no per-tenant bound lets one tenant consume the capacity of all of them (Multi-Tenancy).
- Load shedding decisions must not be attacker-controllable. If a request can mark itself high-priority, shedding protects the attacker's traffic and drops everyone else's (The Trust Boundary).
- A rejection response should not leak internal state — queue depths, worker counts and backlog sizes tell an attacker exactly how much pressure is needed (Not Leaking Your Internals).
- "The queue will absorb it." A queue absorbs bursts. Sustained overload is arithmetic: if arrivals exceed completions on average, the queue grows without bound regardless of its size (Little's Law as Working Intuition).
- "Backpressure means dropping requests." Dropping is one of four responses. Blocking, rejecting with a retry hint, and spilling to durable storage are the others.
- "We autoscale, so we do not need backpressure." Scaling has lag, and it has ceilings — the database, the third-party quota, the pool. Backpressure is what covers the gap (Autoscaling a Backend).
- "429 is a failure." A 429 is the system correctly telling a client to slow down. A 30-second timeout on a request that was never going to complete is a worse outcome for both sides.
- "Our queue is durable, so nothing is lost." Nothing is lost and everything is late. A durable backlog of stale work is its own outage (Queue Backlog).
Operating it
- Queue depth and oldest-item age for every bounded buffer, including in-process ones. The in-process buffers are the ones with no dashboard by default (Depth Is Not an Emergency; Age Is).
- Rejection and shed counts, by reason and by priority class. A non-zero shed rate is the system working as designed and still needs to be visible.
- Producer wait time where you block, because a blocked producer is a latency cost that appears somewhere it did not originate.
- Arrival rate versus completion rate on one chart. The gap between the lines *is* the backlog growth rate, and it is the earliest signal available (Little's Law as Working Intuition).
- Memory growth correlated with queue depth, which distinguishes an unbounded buffer from an actual leak (Leak or Unbounded Cache? The Question That Picks the Fix).
- Saturation of every finite resource — pool, workers, in-flight limit — because saturation is what backpressure exists to communicate (Saturation: The Reading Utilization Cannot Give You).
- At 10x, backpressure stops being a safety net and becomes a routine operating mode: some fraction of traffic is shed during peaks, deliberately.
- At 100x, priority classes and per-tenant quotas become necessary, because "shed the newest" stops being an acceptable policy when the newest includes the most important customer.
- Autoscaling is not backpressure. Scaling takes time, and during that time something has to give — so the two are complements, not alternatives (Autoscaling Lag: The Gap Where the Outage Lives).
- Backpressure must propagate across service boundaries at scale, or one service's rejection becomes another service's unbounded retry queue (Cascading Failure).
- Rejecting work is visible failure now instead of invisible degradation later. It is almost always the better trade and it is the one that generates support tickets.
- Blocking the producer preserves every item and couples the producer's latency to the consumer's — dangerous when the producer is a request handler.
- Shedding by priority protects what matters and requires you to have decided what matters, in advance, in code.
- Spilling to durable storage never loses work and defers the problem into a backlog that must still be drained.
- Bounded buffers turn a rare catastrophic failure into a routine visible one, which is the entire point and is nonetheless a harder conversation with stakeholders.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALBounded buffers and an explicit overflow policy apply to every system that moves work between components at different rates.
- RUNTIME-SPECIFICSome runtimes give backpressure primitives and some do not. Go channels have a capacity and block the sender when full; Node streams implement it via the return value of write() and the drain event, which is silently ignorable and usually is; a plain array or a raw callback queue in any language has no bound at all. Reactive libraries (RxJS, Reactor) model it explicitly with request-based demand. What is automatic in one is entirely manual in another.
- PROTOCOL-SPECIFICTCP applies backpressure at the transport layer via the receive window, so a slow reader eventually slows the writer. That protects the socket, not your application: bytes read off the socket into an unbounded application-level buffer have escaped TCP's flow control entirely (Request Bodies and Streaming).
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.