Fan-Out: Waiting for the Slowest of Seven
Call seven services in parallel and wait for all of them, and your latency is not the average — it is the maximum. A dependency that is slow one time in a hundred becomes a request that is slow seven times in a hundred, which is how p99 problems become p93 problems.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The maximum, not the average
When a request fans out to N dependencies and needs all of their answers, its latency is the maximum of N samples, plus merge time. That is a fundamentally different quantity from the average, and it behaves worse the more dependencies you add — which is the opposite of the intuition that parallelising work makes it fast.
In the waterfall below, six dependencies return between 85ms and 130ms and one takes 580ms. The request takes 660ms. Optimising any of the six fast ones changes nothing at all; they are not on the critical path, in exactly the sense The Critical Path Is the Only Path That Pays means. Only the slow one matters, and only until it stops being the slowest.
This is also why fan-out latency is so resistant to dashboard-driven optimisation. Each dependency team looks at their own p99, sees a healthy number, and correctly concludes they are not the problem. The problem is a property of the composition, and it is nobody's dashboard.
Tail amplification: how p99 becomes p93
Suppose each dependency independently exceeds 500ms one time in a hundred. A request that waits on one of them is slow 1% of the time. A request that waits on seven is slow whenever *any* of them is slow: 1 − 0.99⁷, which is about 6.8%. At twenty dependencies it is 18%; at a hundred, 63%.
Those figures are an estimate under an explicit assumption — that the dependencies fail slowly *independently*. That assumption is usually wrong in both directions. Shared infrastructure (the same database, the same network, the same noisy neighbour) correlates slowness, which makes the real number better than the model in quiet periods and dramatically worse during an incident, because the correlated case is precisely when everything is slow at once.
The practical consequence is that the per-service latency target must be *stricter* than the request target, not equal to it. If a page composed of ten dependencies needs a 300ms p99, each dependency needs a p99 well below 300ms and, more importantly, a bounded tail — because it is the tail, not the median, that composes.
What you can actually do about it
The strongest fix is to need fewer answers. Dependencies that are not required for the first meaningful render can be fetched after it, or rendered optimistically, or dropped when slow — a page that degrades to "recommendations unavailable" in 120ms is usually a better product than one that is perfect in 660ms.
The second fix is a per-dependency timeout with a defined fallback, which converts an unbounded tail into a bounded one. This is a real trade: you are choosing to sometimes return incomplete data rather than sometimes return late data, and which is correct is a product decision rather than an engineering one.
Hedging — issuing a duplicate request after a short delay and taking whichever answers first — genuinely cuts the tail, and it costs extra load on the very dependency that is already struggling. It is a reasonable tool with a strict precondition: the operation must be safe to duplicate (see Idempotency), and the hedge must be capped so it cannot amplify an incident into a Retry Storms: The Load You Generated Yourself pattern.
| Approach | Effect on tail | Cost | When it is wrong |
|---|---|---|---|
| Need fewer answers — defer or drop non-essential dependencies | Removes them from the maximum entirely | Product work: what does degraded look like? | When every dependency is genuinely required for correctness |
| Per-dependency timeout + fallback | Bounds the tail at the timeout value | Incomplete responses some of the time | When partial data is misleading rather than merely incomplete |
| Hedged requests | Cuts the tail substantially in the uncorrelated case | Extra load on an already-slow dependency | Non-idempotent operations; or during correlated slowness, where it adds fuel |
| Reduce fan-out width — batch or aggregate upstream | Fewer samples to take the maximum of | Coupling, a new aggregation service to own | When the aggregator becomes a bottleneck of its own |
| Tighten per-dependency p99 | Attacks the actual cause | Slow, distributed across many teams | Never wrong; just rarely fast enough to fix today's incident |
Key points
- Fan-out latency is the maximum of the dependency latencies, not the average — adding parallel dependencies makes the tail worse, not better.
- Under independence, N dependencies each slow 1% of the time produce a request slow 1 − 0.99^N of the time: ~6.8% at seven, ~18% at twenty.
- Independence is an assumption, and shared infrastructure violates it exactly when it matters most — during an incident.
- Per-dependency latency targets must be stricter than the composed request target, because tails compose.
- Optimising a dependency that is not the per-request maximum changes nothing the user can perceive.
Fan-Out Tail Amplification
Change an input and watch which number moves — and which one does not.
Each dependency is slow only 1% of the time — a number any team would call healthy. But the request waits for all 5, so it is slow whenever any of them is: 4.9% of the time.
Assumes the dependencies are slow independently, which real systems violate — they share hosts, networks and databases, so correlated slowness makes this worse, not better.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Client → API: requests the dashboard, which requires data from seven services.
- 2API → dependencies: seven calls issued in parallel at 40ms; six answer between 85ms and 130ms.
- 3recommendations-service → API: takes 580ms because it is doing its own fan-out internally, invisible from here.
- 4API → merge: cannot begin until the last answer arrives at 620ms; the six fast responses have been sitting in memory doing nothing.
- 5API → client: responds at 660ms, and every dependency dashboard shows a healthy p99 for the interval.
- • "Every dependency p99 is under 130ms, so the request should be under 130ms." The request takes the maximum, and the maximum has a much worse tail than any individual.
- • "We parallelised the calls, so latency is solved." Parallelising converts a sum into a maximum, which is a large win once and then stops helping as width grows.
- • "The slow service is only slow 1% of the time." Multiplied across seven dependencies that is nearly 7% of requests.
- • "Hedging is free tail reduction." It doubles load on the dependency you are hedging against, which is counterproductive precisely when it is overloaded.
- • "No single dependency is responsible." Correct — and the composition still is. Look at blame share, not per-service dashboards.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • The distribution of per-request *maximum* dependency latency, which is the quantity that actually determines request latency.
- • For each dependency, how often it is the slowest one in a request — the "blame share" that identifies where optimisation would pay.
- • Fan-out width per endpoint from traces, so growth in the number of dependencies is visible over time.
- • Correlation between dependency slowness events, to test whether the independence assumption holds in your system.
- • Per-dependency timeout and fallback rates, so bounded tails are distinguishable from missing data.
- • Remove non-essential dependencies from the blocking path: render without them, fetch them after first paint, or drop them under load.
- • Give every dependency call a timeout and a defined fallback so the tail is bounded by design rather than by luck.
- • Reduce fan-out width by aggregating related calls, accepting the coupling that introduces.
- • Hedge only idempotent, capped, and uncorrelated calls — and disable hedging automatically when the dependency is already degraded.
- • Set per-dependency latency objectives derived from the composed target, and hold them as tail objectives rather than median ones.
- • Request p99 for the affected endpoint, compared against the same window before — the per-dependency dashboards will not show the improvement.
- • The distribution of per-request maximum dependency latency, which should tighten if the fix worked.
- • Fallback and timeout rates, to confirm a bounded tail was achieved by degrading gracefully rather than by silently dropping data.
- • Downstream load on any hedged dependency, confirming the hedge did not add meaningful pressure.
- • Timeouts with fallbacks trade completeness for predictability — some users see partial data that was previously merely late.
- • Hedging costs extra load and requires idempotency; it makes correlated incidents worse.
- • Aggregating dependencies to reduce width introduces coupling and a new component that can itself become the bottleneck.
- • Stricter per-dependency SLOs push work onto many teams and are slow to land, however correct they are.
- • An alert on fan-out width per endpoint, so a new blocking dependency added to a hot path is visible at the time it is added.
- • A per-dependency timeout requirement enforced in review — an un-timed-out call on a critical path fails the check.
- • An SLO on request p99 for composed endpoints, distinct from the per-service SLOs, so the composition has an owner.
- • A dashboard of blame share by dependency, reviewed periodically rather than only during incidents.
Accuracy
Performance numbers are conditional. These are the conditions.
- ESTIMATEDThe 1 − (1−p)^N amplification figures assume dependency slowness events are statistically independent. Shared infrastructure correlates them, which makes the model optimistic during quiet periods and pessimistic about how independent your incidents will be.
- ILLUSTRATIVEThe seven-dependency waterfall timings are invented to show the shape where one span dominates. Real fan-outs vary in width, timing and correlation.