The question this answers
What does issuing N calls concurrently instead of sequentially buy me, and what does it cost the systems on the other end?
A product page handler that needs inventory, pricing, reviews, recommendations and shipping estimates — five independent service calls of 40, 60, 30, 180 and 90ms — assembled into one response.
The aggregate result being filled in by five completions, plus the completion counter or promise-combinator state that decides when the request is done. Downstream, the shared state is everyone else's capacity: connection pools, rate-limit budgets and the databases behind those five services.
The response contains exactly one entry per branch — a value or an explicit failure — no branch's result is lost or double-counted, and the handler returns exactly once, after every branch has either completed, failed or been abandoned by a deadline.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Sum becomes maximum — and load becomes N times
Sequentially the five calls cost 40 + 60 + 30 + 180 + 90 = 400ms. Issued concurrently they cost about 180ms, the slowest branch, plus aggregation overhead. That is the entire promise of fan-out and it is a real one: The Sequential Await Trap is the most common single cause of a slow handler, and turning a chain of awaits into a Promise.all & gather is often the highest-value change in a request path.
The cost is on the other side of the diagram. Each incoming request now produces five outgoing requests, so at 200 requests per second the downstream fleet sees 1000. The recommendations service that was comfortable at 200 rps is now the constraint, and the failure will present as *your* handler timing out rather than as an obvious capacity problem in a service you may not own. Parallelism Moves the Load Downstream is this exact failure, and it is why fan-out is a capacity decision, not a latency trick.
Worse, the amplification interacts badly with retries. Five branches, each retrying twice on failure, is fifteen downstream requests per incoming request during exactly the period when downstream is already struggling. Fan-out plus naive retry is one of the standard ways a partial outage becomes a total one; Circuit Breaker and per-branch budgets exist for that reason.
- Latency: sum -> max. Load on each dependency: x1 -> xN per incoming request.
- The branch that sets the wall clock is the only one worth optimizing; the other four are free.
- Fan-out with per-branch retries multiplies amplification again, precisely when downstream is least able to absorb it.
The fan-in is where the bugs are
Fanning out is easy; fanning in correctly is not. The aggregator must decide when the request is done, and the two hand-rolled implementations of that decision — a shared results object and a completion counter — are two of the most reliable sources of concurrency bugs in ordinary application code.
The schedule below is the counter version. Two branches complete, each does a read-modify-write on done, one increment is lost, the counter never reaches 5, and the request hangs until something else times it out. It is Shared Mutable State in the most boring possible form, and it appears in application code precisely because the author did not think of an aggregator as concurrent code — it is "just counting".
The correct answer is almost never a mutex around the counter. It is to not hand-roll the fan-in: use the runtime's combinator (Promise.all & gather), or give each branch a preallocated slot indexed by branch number so there is no shared mutable structure at all. Where you do need a counter, an atomic decrement whose *return value* triggers completion is the standard correct form — see Atomics: What Is Actually Indivisible and Latches & Countdowns.
| # | Reviews branch | Inventory branch | Handler | State |
|---|---|---|---|---|
| 1 | · | · | issue 5 branches; done = 0 | done=0 responded=no |
| 2 | completes at 30ms; read done -> 0 | · | · | done=0 |
| 3 | · | completes at 40ms; read done -> 0 | · | done=0 |
| 4 | write done = 1; results[reviews] = ... | · | · | done=1 |
| 5 | · | write done = 1; results[inventory] = ... | · | done=1 ✕ Two branches completed but done is 1. The counter no longer equals the number of completions and can never reach 5. |
| 6 | · | · | pricing, shipping, recommendations complete; done = 4 | done=4 responded=no |
| 7 | · | · | waits for done === 5 — forever | done=4 responded=no |
Deciding what a partial failure means
With five branches, the probability that all five succeed is lower than the probability that any one does — a 99.9% branch, five times, is 99.5% for the aggregate. Fan-out therefore *reduces* the availability of your handler unless you decide, explicitly, what a missing branch means. That decision belongs in the design, not in a catch block added later.
The matrix below is the menu. Most product pages want "essential branches are required, decorative branches are best-effort with a deadline" — recommendations missing is a slightly worse page, pricing missing is not a page at all. Expressing that requires per-branch deadlines rather than one overall timeout, because a single global timeout gives the slow decorative branch the power to fail the whole request.
Whichever policy you choose, bound the fan-out. Five is a constant; "one call per item in the cart" is not, and an unbounded fan-out driven by user input is how a single request saturates a connection pool (Unbounded Concurrency). A semaphore around the branches (Bounding Concurrency) turns an unbounded burst into a bounded stream, at the cost of turning some of the parallelism back into sequence.
- Aggregate availability is the product of the branches you declared essential. Shrink that set deliberately.
- Per-branch deadlines, not one global timeout — otherwise the least important branch controls the whole request.
- A fan-out whose width comes from user input is an unbounded fan-out. Bound it with a semaphore before it bounds itself with a pool exhaustion incident.
| Policy | Returns when | Availability | Use when | Cost |
|---|---|---|---|---|
| All-or-nothing | All branches succeed | Product of all branches — the worst | Every branch is essential (a payment, an auth check) | One flaky dependency fails every request |
| Fail-fast | First error, cancelling the rest | Same as all-or-nothing, but fails sooner | Errors are terminal and work is expensive | Must actually cancel, or you leak in-flight work |
| Best-effort / partial | All branches settle, errors recorded | As good as the essential subset | Decorative data (recommendations, badges) | Callers must handle a partial shape — put it in the contract |
| Deadline-bounded | Deadline expires; late branches abandoned | Predictable p99 by construction | A latency budget exists and is enforced | Results discarded after the work was already paid for |
| First-response-wins | Any one branch answers | Best — 1 - (failure rate)^N | Redundant equivalent replicas | N times the load for one answer; needs cancellation |
Key points
- Fan-out converts a sum of latencies into a maximum, and converts one unit of downstream load into N.
- Only the slowest branch matters for latency; optimizing any other branch is invisible in the response time.
- Hand-rolled fan-in is where the bugs live — a lost increment on a completion counter hangs the request while every service reports healthy.
- Aggregate availability is the product of the essential branches, so five 99.9% dependencies give 99.5% unless some are declared optional.
- Per-branch deadlines and a bound on N are what separate a fan-out that survives a bad day from one that amplifies it.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Identify branches with no data dependency on each other; anything that consumes another branch's output is not a branch, it is a second round.
- • Issue all branches, bounded by a semaphore when N is data-dependent, each carrying its own deadline derived from the request budget.
- • Collect results into preallocated per-branch slots, or use the runtime's all/gather/settled combinator rather than a hand-written counter.
- • On the deadline or on a fail-fast error, cancel the outstanding branches and propagate cancellation to their I/O (Cancellation Propagation).
- • Assemble the response from present results plus explicit per-branch failure markers, and emit a metric per branch for success, failure and abandonment.
- • The intended one: all five branches issue, complete in any order, each writes its own slot, the combinator resolves when the fifth settles. Slot writes are disjoint, so every ordering is safe.
- • Lost increment: Reviews reads done (0); Inventory reads done (0); Reviews writes 1; Inventory writes 1 — two completions, counter says one, the request waits for a fifth completion that already happened.
- • Late branch after the deadline: the handler responds at 200ms; Recommendations completes at 340ms and writes into the results object of a request that already ended — a use-after-response that corrupts a cache entry or throws inside a completion callback with no request context.
- • Fail-fast without cancellation: Pricing errors at 20ms, the handler returns an error, and four branches continue running to completion, holding connections and doing work nobody will read (Orphaned Tasks).
- • Amplified retry storm: downstream slows, every branch's timeout fires, every branch retries twice, downstream load triples at the moment it was already saturated, and every handler now takes the full deadline.
- • An all/gather combinator guarantees the aggregate resolves only after every branch has settled, and establishes happens-before between each branch's writes and the aggregation.
- • A settled/allSettled variant additionally guarantees you learn the outcome of every branch, rather than only the first failure.
- • A rejecting all/gather does NOT guarantee the other branches stopped. In most runtimes they keep running; cancellation is a separate thing you must arrange (Cancellation).
- • A deadline guarantees when you *respond*. It does not guarantee the abandoned work stopped, that its connections were released, or that its side effects did not happen.
- • Nothing here guarantees the downstream systems can absorb N times their previous load. That is the assumption fan-out silently makes on your behalf.
- • Client-side: N concurrent branches per request multiply demand on the HTTP connection pool, the DNS resolver and the TLS handshake path; pool exhaustion presents as latency, not as an error (Pool Saturation).
- • Server-side, downstream: each dependency sees N times the request rate, and the first one to saturate becomes the slowest branch and therefore your entire response time.
- • Aggregator-side: a lock around the results object serializes the fan-in and can quietly undo the parallelism, especially when branches complete in a burst.
- • Thread-per-branch designs contend for the pool itself: 200 concurrent requests times 5 branches is 1000 tasks, and if those are threads the machine is oversubscribed (Oversubscription).
- • Lost update on the completion counter: request hangs, all dependencies healthy, resolved only by an outer timeout.
- • Partial-failure blindness: one branch throws, the aggregate rejects, and the user gets an error page because the recommendations service was slow.
- • Orphaned branches after fail-fast or a deadline: work continues, connections stay held, and a completion callback runs against a finished request.
- • Load amplification into a downstream outage, made worse by per-branch retries.
- • Unbounded fan-out from user-supplied N: one request exhausts the connection pool for every other request on the instance.
- • Deadline inversion: a global timeout lets the least important branch decide the request's fate.
- • Independent I/O-bound calls in a request path — the canonical win, and usually a large one relative to the effort.
- • Aggregation endpoints and backends-for-frontends whose whole job is composing several services into one response.
- • Any handler currently written as a chain of awaits over calls that do not consume each other's output (The Sequential Await Trap).
- • Redundant replicas where a first-response-wins policy converts a latency distribution into its minimum, if you can afford the duplicate load.
- • When one branch dominates: 180ms out of 400ms means fan-out gets you to 180ms, and after that the only lever is that branch.
- • When downstream capacity is the binding constraint — you have moved the bottleneck outward and made it harder to see.
- • When N is data-dependent and unbounded; the good case is fast and the bad case takes the instance down.
- • When the branches are CPU-bound in a single-threaded runtime: they cannot overlap at all, and you have added the coordination cost of concurrency for no overlap (Async Is Not Parallelism).
- • When failure semantics have not been decided, so the first flaky dependency turns into a full outage of your endpoint.
- • Per-branch latency distribution, and the fraction of requests in which each branch was the slowest. That single ratio tells you where the only useful optimization is.
- • Aggregate latency against max(branch latencies): a large gap is aggregation or scheduling overhead, not a slow dependency.
- • Downstream request rate divided by your request rate — the amplification factor, measured rather than assumed, including retries.
- • Per-branch error, timeout and abandonment rates, separately. Abandonment is invisible unless you count it explicitly.
- • Connection pool wait time on the client side; it rises before anything else does when the pool is the real constraint.
- • You now own a failure policy per branch, a deadline per branch, and a response shape that can be partial — all of which belong in the API contract, not in an exception handler.
- • Cancellation becomes mandatory rather than optional, and cancellation that actually reaches the underlying socket is more work than it looks.
- • Observability must be per-branch or the aggregate is undebuggable; a single span for "fetch data" tells you nothing about which of five things was slow.
- • Capacity planning now couples your traffic to five other teams' capacity, with an amplification factor that changes whenever someone adds a branch.
- • Testing must cover the partial-failure matrix, which grows combinatorially — pick the essential/optional split deliberately to keep it small.
- • One batch call instead of N calls, when the downstream API supports it. Same latency win, one connection, no amplification — strictly better where it exists (Batch APIs and Partial Failure).
- • Cache or precompute the slow branch. If recommendations are 180ms of a 180ms request and tolerate staleness, a cache removes the branch rather than parallelizing it.
- • Sequential with an early exit, when a cheap branch frequently determines the answer — five concurrent calls to discover the item is out of stock is four wasted calls.
- • Move the aggregation server-side into one service that owns the data, when the fan-out exists only because the data was split across services for organizational reasons.
- • Return the essential response immediately and load decorative parts from the client, converting a fan-in problem into a rendering one.
One request, N downstream calls
One of five fails — what happens to the siblings?
Promise.all rejects on the FIRST rejection; siblings are NOT cancelled and keep running
Promise.allSettled never rejects; resolves at 90 ms with {status, value|reason} for all five
asyncio.gather(...) return_exceptions=False → raises at 40 ms, siblings still NOT cancelled
return_exceptions=True → returns at 90 ms with the exception as a value
asyncio.TaskGroup the structured alternative: on failure it CANCELS the siblings, then raisesBounding concurrency with permits
| permits | goodput | mean latency | timeouts | failed of 10K |
|---|---|---|---|---|
| 1 | 25/s | 43 ms | 0.00% | 0 |
| 5 | 125/s | 43 ms | 0.00% | 0 |
| 10 | 250/s | 43 ms | 0.00% | 0 |
| 25 | 625/s | 43 ms | 0.00% | 0 |
| 50 | 1000/s | 53 ms | 0.00% | 0 |
| 100 | 1000/s | 103 ms | 0.00% | 0 |
| 200 | 1000/s | 203 ms | 0.00% | 0 |
| 350 | 1000/s | 353 ms | 0.00% | 0 |
| 500 | 0/s | 503 ms | 100.0% | 10K |
What people believe, and what is true
Fan-out is a pure win: same work, less time.
Same work for you, N times the arrival rate for every dependency, at the same instant. The cost moved rather than disappeared.
A global timeout protects the request.
It caps your response time and lets the least important branch consume the entire budget. Per-branch deadlines are what actually protect the essential branches.
If the combinator rejected, the other branches stopped.
In most runtimes they run to completion, holding connections and producing side effects. Cancellation is separate and must be arranged.
Concurrent branches are safe because each one just returns a value.
The fan-in is shared mutable state. A counter incremented from five completions is the same lost-update bug as any other counter.