Latency Budgets: Spending 200 Milliseconds on Purpose

A budget converts an aspiration into arithmetic. Below, a 200 ms target is allocated across the hops a checkout request actually crosses. The allocations sum to 185 ms, leaving 15 ms of slack — which is the number that makes the budget useful. Slack is what absorbs a slower-than-usual GC, a retried packet, a cold connection. A budget with zero slack is a target you will miss whenever anything is even slightly unusual.

Notice what the breakdown does to the discussion. "Make checkout faster" is a wish. "The payment provider has 90 ms of a 200 ms budget, so we cannot add another synchronous call without going over" is an engineering constraint that can be designed against — by moving the provider call out of the request path, by caching, or by explicitly renegotiating the target.

Budgets also expose work nobody counted. Network time, TLS, serialization and the client's own rendering are real milliseconds that rarely appear in a service dashboard. If your budget only covers server-side spans, you are budgeting a fraction of what the user experiences, which is why the user-facing target and the server-side target should be stated separately (The Half of the Budget You Cannot See From the Server).

Once allocations exist, three questions become answerable. First: which hop is over its allowance right now? That is a comparison, not a judgement call. Second: what is the cost of a proposed change? Adding a synchronous call to a new service is no longer "a small addition" — it is a request for a share of a fixed budget, and something else must give it up. Third: what happens when a hop cannot meet its allowance? That is a design conversation about moving work out of the request path, not a performance-tuning conversation.

The comparison below is what makes budgets worth the effort. The naive framing treats a target as a single number checked at the end, which produces a diffuse "everyone optimise" response. The budgeted framing localises the overage and produces a specific decision.

The most common structural answer for an over-budget external dependency is to stop waiting for it: make the call asynchronous, return a pending state and complete out of band (The Async Job Pattern). That converts a latency problem into a state-management problem, which is often the right trade — but it is a trade, and the budget is what makes it visible as one.

A subtle trap: if every hop meets its allowance at p95, the end-to-end p95 is not the sum of the p95s. The end-to-end request is slow if *any* hop is slow, and hops are rarely slow simultaneously, so summing p95s over-estimates. Conversely, budgeting each hop at p50 and expecting the total to hold at p95 under-estimates badly. The practical convention is to allocate at the percentile you actually care about, then validate the composition empirically rather than trusting the arithmetic.

The other trap is treating the budget as fixed for all users. A client 150 ms away by network round trip has already spent most of a 200 ms budget before your code runs. That is not a bug to fix in the application; it is a reason for regional deployment, edge caching, or a different target per region (Cross-Region Latency Is Physics, Not Configuration).

Finally, budgets should be revisited when the system changes shape, and they should be attached to something durable — an SLO, a design document, a dashboard annotation. A budget nobody can find is rediscovered as folklore six months later, usually during an incident.