Bulkheads
Give each dependency its own bounded slice of your resources, so one slow dependency cannot consume every worker you have.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
Why does a slow recommendation service take down endpoints that never call it?
Checkout, search and recommendations share one service. When recommendations is slow, checkout must keep working.
One worker pool, one connection pool, shared by everything. It is simple and it uses resources efficiently — no capacity sits idle.
Recommendations becomes slow. Each call holds a worker for the whole timeout. Within a minute every worker is waiting on recommendations and checkout has no capacity at all (Connection Pool Exhaustion).
- Recommendations becomes slow. Each call holds a worker for the whole timeout. Within a minute every worker is waiting on recommendations and checkout has no capacity at all (Connection Pool Exhaustion).
- The symptom is "the whole API is down", which points investigation everywhere except at the one dependency that is merely slow (Why Is My API Slow?).
- Adding instances does not help: the new ones also fill with recommendations calls, because the traffic mix has not changed.
- The database connection pool has the same problem — a slow analytical query on one endpoint starves every other endpoint of connections (Connection Pools).
- A breaker eventually opens, but only after the pool has already drained, so the fast failure arrives after the damage (Circuit Breakers).
What is actually happening
- The name is from ships: compartments with walls, so a breach floods one section rather than the hull. The engineering version is partitioned resource limits, so exhausting one partition does not exhaust the others.
- The resource that runs out is rarely CPU. It is the bounded set of things that can be in flight: worker threads, connection-pool slots, or, on an event-loop runtime, memory and the queue of pending work (Resource Limits).
- Little's Law explains the collapse precisely: concurrent calls in flight equals arrival rate times duration. When a dependency's duration rises tenfold and arrival rate is unchanged, in-flight calls rise tenfold, and whatever bounds in-flight work is what breaks (Little's Law as Working Intuition in Performance).
- A bulkhead is a cap on in-flight calls per dependency — a semaphore, a dedicated pool, or a separate process. Once the cap is reached, further calls are rejected or queued briefly rather than consuming the shared resource.
- The rejection is the feature. Refusing the eleventh concurrent recommendations call means checkout still has workers; allowing it means everyone waits.
- Bulkheads compose with the other patterns rather than replacing them. Timeouts bound one call's duration; bulkheads bound how many can be in flight at once; breakers stop calling entirely after sustained failure (Timeouts, Circuit Breakers).
- The partition can be per dependency, per tenant, per endpoint class or per priority. Tenant partitioning is what stops one customer's traffic from degrading everyone else's (Multi-Tenancy).
One pool, one bad dependency, total outage
The diagram below is the mechanism in full. Nothing is broken except that one dependency got slower, and the shared bounded resource converts that local slowness into a global failure.
The important detail is that the requests holding workers are not doing work. They are waiting. Capacity is fully consumed by waiting, which is why adding capacity does not help — the new capacity waits too (Little's Law as Working Intuition in Performance).
A permit per dependency
The smallest useful bulkhead is a counting semaphore around each dependency call. It does not need a library, and the two things that make it correct are visible in the code: acquisition either succeeds quickly or fails, and release happens on every path.
The second point is where these break in practice. A permit leaked on an error path shrinks the effective limit until the bulkhead rejects everything, and the resulting incident looks exactly like the dependency being down.
async function getRecs(userId: string) {
return await recsClient.fetch(userId) // timeout set, at least
}
// 500 concurrent slow calls is fine here and fatal
// everywhere else in the process: they hold workers,
// sockets and memory that checkout needs.const recsPermits = new Semaphore(10) // sized for THIS dependency
async function getRecs(userId: string): Promise<Recs> {
// short bounded wait, then give up: no unbounded queue in front
const permit = await recsPermits.tryAcquire({ waitMs: 50 })
if (!permit) {
metrics.increment('bulkhead.recs.rejected')
return Recs.empty() // degrade: recs are optional
}
try {
return await recsClient.fetch(userId)
} finally {
permit.release() // every path, including timeout and throw
}
}The cap converts an unbounded hold on shared capacity into a bounded one, so a tenfold slowdown in recommendations costs at most ten workers instead of all of them. The short acquire wait matters as much as the cap: an unbounded queue in front of the semaphore would hold the same requests in a different place while pretending to be bounded.
Choosing the walls
Isolation exists on a spectrum from a semaphore in a function to a separate deployment with its own machines. Stronger isolation costs more and shares less, and the right point depends on what you are protecting and from what.
A useful heuristic: isolate along the boundary where you would accept one side failing while the other continues. If you would not accept that, the boundary is somewhere else.
What do you need to survive independently, and what are you willing to pay for it?
when The default. Cheap, local, immediately effective against slow dependencies.
cost Shares memory, CPU and the event loop or GC. Does not isolate a dependency that returns enormous payloads.
when Distinct workload classes against the same backing store — interactive versus analytical queries.
cost More pools to size and monitor; reserved connections idle when their class is quiet (Connection Pools).
when CPU-bound or memory-heavy work that must not affect request serving.
cost Process management, more memory, cross-process communication (Worker Processes).
when A genuinely different availability requirement — checkout must survive everything else.
cost A distributed system, with its network calls, deploys and observability (Microservices).
when Multi-tenant systems where one customer's burst must not degrade others.
cost Per-tenant limits to size and enforce, and idle reserved capacity per partition (Multi-Tenancy).
How to build it
Most important first.
- Bound in-flight calls per dependency with a semaphore, sized from the dependency's normal concurrency plus headroom — not from your total worker count.
- Reject fast when the bulkhead is full. A short bounded wait is acceptable; an unbounded queue in front of a semaphore recreates the problem one layer up (Bounded vs Unbounded Queues in Concurrency).
- Separate the pools that matter: distinct database pools for interactive and analytical work; distinct outbound clients per dependency so connection limits are not shared (Connection Pools).
- Isolate by criticality, not only by dependency. Checkout traffic and recommendation traffic can be separate worker sets even in one service (Resource Limits).
- For hard isolation, use separate processes or deployments. Same-process bulkheads still share memory, the event loop and the garbage collector; separate processes do not (Worker Processes).
- Bound background and job workloads separately from request-serving, so a queue drain cannot starve live traffic (Worker Scaling).
- Size against Little's Law rather than intuition: pick the concurrency you are willing to devote to a dependency, and understand it directly caps the throughput you can sustain given its latency.
What can go wrong
- A bulkhead sized so small it rejects during normal operation, converting a capacity plan into a self-inflicted error rate.
- A bulkhead sized so large it never binds, which is indistinguishable from having none.
- Rejections treated as generic errors and retried, so the rejected load comes straight back (Retries).
- Partitioning the worker pool but not the connection pool — the compartment with no wall is the one that floods.
- Same-process bulkheads assumed to isolate CPU. A dependency that returns huge payloads still blocks the loop or dominates garbage collection for everyone (Blocking the Event Loop).
- Per-instance limits on a large fleet, so the effective concurrency against the dependency is the limit multiplied by instance count — frequently far more than the dependency expects (Stateless Services).
- Reserved capacity sitting idle while another partition queues, which is the visible cost and is often mistaken for a bug.
- Many requests acquiring permits concurrently — the counter must be atomic, or the limit is exceeded under exactly the burst it exists to contain (Atomic Operations).
- Permits released on some paths and leaked on others, so the effective limit shrinks over time until the bulkhead rejects everything — the release belongs in a
finally(Unbounded Concurrency). - A timeout firing while a permit is held, where cancellation and release must be ordered correctly or the permit is lost (Timeouts).
- Without tenant partitioning, one tenant's load is a denial of service against every other tenant, and it need not be malicious to be effective (Tenant Isolation).
- An attacker who finds the slowest dependency can direct traffic at it to exhaust shared capacity; per-dependency caps bound how much of your service any single path can consume.
- Bulkheads are what make graceful degradation a real property rather than an aspiration, which matters when availability is part of a contractual commitment.
- "Bulkheads are just rate limiting." Rate limiting bounds requests per unit time; a bulkhead bounds concurrent in-flight work. A dependency that becomes ten times slower breaks the second bound without touching the first (Rate Limiting).
- "A timeout is enough." A timeout bounds one call. A hundred concurrent calls each waiting the full timeout still exhaust the pool — the timeout sets how long, the bulkhead sets how many.
- "We have autoscaling." Scaling adds instances that fill with the same stalled calls. You cannot outscale an unbounded per-request hold time (Autoscaling a Backend).
- "Isolation means microservices." A semaphore per dependency inside one process gives most of the benefit for none of the operational cost (The Modular Monolith).
- "Rejections mean the bulkhead is too small." Sometimes. During a dependency incident, rejections are the wall working exactly as designed.
Operating it
- In-flight count and the limit, per bulkhead, as a saturation ratio. This is the leading indicator — it rises before anything fails (Saturation: The Reading Utilization Cannot Give You in Performance).
- Rejections per bulkhead, separate from dependency errors. Rejections mean the wall did its job.
- Time spent waiting to acquire a permit, as a histogram. Rising acquisition time with a flat dependency latency means the limit is too small for current traffic.
- Per-endpoint latency during a dependency incident. If unrelated endpoints move, isolation is not working regardless of what the configuration says.
- Connection-pool wait time alongside query time, so "the database is slow" can be separated from "we ran out of connections" (Connection Pool Saturation: Waiting in Front of an Idle Database in Performance).
- At 10x, shared pools are the first thing to break, and the failure looks like a general outage rather than a dependency problem — which is why the isolation must exist before it is needed.
- Per-instance bulkheads multiply across the fleet. Twenty instances with a limit of ten each present two hundred concurrent calls to the dependency, which may be far beyond what it will tolerate (Rate Limiting).
- At large scale the strongest bulkhead is deployment separation: a distinct service, pool and failure domain per critical path (Microservices — with all of that lesson's costs).
- Bulkheads bound how much of your capacity a dependency can consume, and therefore bound your throughput against it. That ceiling is a capacity planning input, not an accident.
- Reserved capacity is idle capacity. A partition sized for checkout sits unused when checkout is quiet, and that waste is the mechanism, not a flaw in it.
- More partitions means more limits to size, monitor and revisit, and each one is a place to be wrong.
- Rejecting under load produces errors that a shared pool would have merely made slow — a deliberate trade of some failures now for not failing everything later.
- Process- or deployment-level isolation is the strongest and costs operational surface: more things to deploy, monitor and pay for (Deployment Models).
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALBounded, partitioned resources apply to every runtime; only the identity of the scarce resource changes.
- RUNTIME-SPECIFICOn a thread-per-request runtime the bounded resource is threads and exhaustion is abrupt and obvious. On an event-loop runtime awaits are cheap, so there is no natural bound at all — unbounded in-flight work grows memory and latency until something else fails, which means the semaphore must be explicit rather than inherited from the pool size (Backend Runtime Models).
- SCALE-SPECIFICA single-instance service with one dependency gains little from partitioning. The pattern earns its complexity when several dependencies of differing criticality share one bounded pool.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Distributed Systems — failure containment and why isolation boundaries decide the blast radius of any single component.