6 lessons

Distributed Systems Performance

What changes when the work crosses machines: fan-out and tail amplification, sequential versus parallel dependency calls, cross-region propagation delay, jitter, and the cost of coordination.

Symptom→Signal→Measurement→Hypothesis→Evidence→Root Cause→Change→Validation→Regression Check

Every lesson below starts from an observable symptom and ends with the measurement that proves the fix worked. Numbers carry a label saying whether they were measured, estimated, simulated or invented to show a shape.

What Changes When Work Crosses a Machine

An in-process function call costs nanoseconds and either returns or throws. The same call across a network costs milliseconds, serialises both ways, waits in three queues you cannot see, and has a third outcome: no answer at all.

Symptom · The same business operation that took 40ms in one process takes 400ms across services, and the sum of the individual service timings does not account for the difference.
Fan-Out: Waiting for the Slowest of Seven
▶ lab

Call seven services in parallel and wait for all of them, and your latency is not the average — it is the maximum. A dependency that is slow one time in a hundred becomes a request that is slow seven times in a hundred, which is how p99 problems become p93 problems.

Symptom · Aggregating service latency dashboards suggests the page should render in 120ms. Real request p99 is over a second, and no single dependency looks responsible.
Sequential or Parallel: Same Work, Different Latency

Four dependency calls take 740ms in a chain and 300ms fanned out. The parallel version is not simply better: it triples the instantaneous load on everything downstream and turns one failure into four things to reason about at once.

Symptom · A handler makes four downstream calls one after another. Latency is roughly the sum of all four, and each call spends its time waiting rather than computing.
Cross-Region Latency Is Physics, Not Configuration

Light in fibre travels about 200,000 km/s. Frankfurt to Virginia and back is roughly 13,000 km of that, so no amount of tuning gets a round trip under about 65ms — and a request that crosses the Atlantic four times has spent a quarter of a second before doing any work.

Symptom · Latency is strongly bimodal by user geography. The slow group is fine on cached pages and terrible on anything that writes, and no service in the path reports elevated latency.
Packet Loss Buys You a Timeout, Not a Retransmit

A link that drops one packet in a thousand looks almost perfect on an average-latency graph. What it actually does is give one request in a few hundred an extra couple of hundred milliseconds, which is invisible at p50 and dominates p99.

Symptom · p50 is flat and p99 is spiky and irregular. The slow requests are not correlated with any particular endpoint, tenant, or piece of code, and they do not reproduce in testing.
Agreement Costs Round Trips

Every guarantee that several machines agree on something is paid for in round trips. A quorum write is at least one; consensus is more; a distributed lock is two plus however long the holder keeps it. The guarantee is often worth it — the cost is never zero.

Symptom · A logically trivial operation takes tens or hundreds of milliseconds. CPU is idle everywhere, no query is slow, and latency scales with the number of participants rather than with the amount of data.