Cost per Request: The Other Performance Metric
Compute, database, cache, bandwidth, third-party calls and inference add up to a number per request. Track it next to latency and a whole category of "optimizations" reveals itself as buying a small latency win with a large permanent bill.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Adding up one request
Cost per request is a derived number, not a measured one — you allocate each resource's spend across the work that consumed it. Compute cost comes from instance-hours divided by requests served. Database cost from provisioned capacity and I/O attributable to the route. Cache from memory footprint, bandwidth from egress bytes, third-party from per-call pricing, and inference from tokens in and out at the model's rate.
The exercise is worth doing even roughly, because it usually produces a surprise. Teams are often startled to find that egress or a third-party call dominates, while the compute they spend all their optimization effort on is a rounding error. The table below is illustrative arithmetic for a checkout request, not a price list — the shape of the result is the point, not the values.
Once you have the breakdown per route, two things become answerable that were not before: which endpoints deserve optimization attention (the expensive ones, not the slow ones), and whether a proposed change is a good trade. Both questions are invisible when the only number anyone has is a monthly total.
| Component | Driver | Illustrative share | What moves it |
|---|---|---|---|
| Compute | CPU-seconds × instance rate ÷ requests | 18% | Per-request CPU cost; serialization and rendering work |
| Database | Provisioned capacity + I/O ÷ requests | 24% | Queries per request; N+1 patterns (The Comb: N+1 as a Visible Shape) |
| Cache | Memory footprint ÷ requests | 4% | Working-set size and TTL policy |
| Egress bandwidth | Response bytes × egress rate | 11% | Payload size and compression (Payload Size: 20KB, 200KB, 5MB) |
| Third-party API | Per-call price × calls per request | 31% | Calls per request; caching or batching them |
| Model inference | Tokens in/out × model rate | 12% | Context size, model choice, retries (What One Agent Run Costs, and Which Term Dominates) |
The latency fix that tripled the bill
Here is the trade that cost per request exists to expose. A team is asked to improve checkout p99. Option one: fan the read out across eight replicas so no replica is ever hot. Option two: fix the N+1 that issues 41 queries per request. Both are described in the ticket as "improve checkout latency". They are not remotely equivalent.
The replica fan-out is faster to ship and produces a small win — the reads were not the dominant cost of the request, so removing contention on them moves p99 by 15ms. It also triples the database spend, permanently, for as long as the service exists. The N+1 fix takes longer to write, removes 40 queries per request, moves p99 by 400ms, and *reduces* database cost because the work simply stops happening.
Neither the latency graph alone nor the bill alone would have distinguished these. Together they do, immediately. This is the argument for tracking cost per request as a first-class performance metric: it turns "which optimization?" from a matter of taste into a comparison with two axes.
1Change: read fan-out across 8 replicas2 3p99 520 ms -> 505 ms (-15 ms)4DB spend 1.0x -> 3.1x (permanent)5cost/request $0.0041 -> $0.0093 (+127%)6queries/request 41 -> 41 (unchanged)7 8The contention was never the dominant term.9The 41 queries still happen; they now happen10on more hardware.1Change: fix the N+1 (batch the per-item lookup)2 3p99 520 ms -> 120 ms (-400 ms)4DB spend 1.0x -> 0.6x5cost/request $0.0041 -> $0.0026 (-37%)6queries/request 41 -> 27 8Work that does not happen costs nothing and9takes no time. Both axes improve together.Adding capacity moves a bottleneck; removing work eliminates it. Capacity purchases show up on the bill every month forever, which is invisible on a latency dashboard and obvious the moment cost per request sits next to it.
Unit economics as a performance metric
Treat cost per request like latency: measure it per route, track it over releases, and alert on regressions. A release that raises cost per request by 30% is a regression even if latency improved, and it deserves the same conversation a latency regression would get. Without the metric, that change ships silently and is discovered a quarter later during a budget review, when nobody can attribute it to a specific release.
Two caveats keep this honest. First, cost per request naturally falls as traffic grows, because fixed costs spread across more requests — so compare like-for-like traffic levels, or the metric will flatter you during growth and punish you during quiet periods. Second, cheapest is not the goal: a slow, cheap service that loses customers is not an optimization. Cost per request is one axis of a trade, and its value is in making the trade visible.
The most useful place to put it is next to latency in the same view, per route. When an endpoint appears in the top five for both, that is where optimization effort belongs — and when a change moves one axis in the wrong direction, everyone can see the price of the win.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| p99 latency, POST /checkout | 520 ms → 505 ms | A small improvement — the headline in the release notes | normal |
| Cost per request, POST /checkout | $0.0041 → $0.0093 | More than doubled, permanently, for a 3% latency win | smoking gun |
| Queries per request | 41 → 41 | The work was never removed; it was spread over more hardware | smoking gun |
| Monthly infrastructure spend | +8% | Diluted across all routes — invisible without per-route allocation | suspect |
| Requests per month | flat | Rules out growth as the explanation for the spend increase | normal |
Key points
- Cost per request is derived by allocating each resource's spend across the requests that consumed it — per route, not per service.
- The breakdown usually surprises: third-party calls and egress often dominate the compute everyone optimizes.
- Adding capacity buys latency and bills monthly forever; removing work improves latency and cost together.
- Track cost per request next to latency per route and treat a cost regression as a regression.
- Compare at like-for-like traffic — cost per request falls with growth for reasons that have nothing to do with efficiency.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Traffic → spend: the bill rises 8% while request volume stays flat, so growth does not explain it.
- 2Spend → route: allocating spend per route attributes the increase to
POST /checkout, not to the service as a whole. - 3Route → component: the checkout increase is entirely database, and query count per request did not change.
- 4Component → change: the release added read replicas, tripling provisioned database capacity for the same query volume.
- 5Change → trade: p99 improved by 15 ms while cost per request rose 127% — a trade nobody explicitly agreed to.
- • "The bill went up because traffic went up" — check requests per month first; often it is flat and the cause is a release.
- • "Cost per request is falling, so we are getting more efficient" — during growth, fixed costs spread and the metric improves on its own.
- • "The expensive endpoints are the slow ones" — frequently false; a fast endpoint called constantly can dominate spend.
- • "We optimized compute, so cost should drop" — if compute is 18% of the request, halving it moves the total by 9%.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Allocate each resource's spend (compute, database, cache, egress, third-party, inference) across requests using per-route traces and usage metrics.
- • Queries per request, external calls per request and response bytes per request from traces — these are the drivers you can actually change.
- • Cost per request per route over releases, compared at matched traffic levels.
- • Token usage per request for any model-backed path, split by input and output ([[agent-cost]]).
- • Remove work before adding capacity: batch N+1 queries, cache or batch third-party calls, shrink payloads — these improve both axes.
- • Publish cost per request per route next to latency, so every optimization proposal can be evaluated on both.
- • Attack the largest component first; optimizing an 18% share caps your win at 18% no matter how well you do it.
- • For model-backed paths, reduce context size and retries before switching to a cheaper model, since both cut cost without changing capability.
- • Set a cost budget per route the same way you set a latency objective, and review changes that breach it.
- • Compare cost per request before and after the change at matched traffic levels, not across a growth period.
- • Confirm the driver moved, not just the total: queries per request, external calls per request, or bytes per request.
- • Check that latency did not regress in exchange — the point is to see both axes, not to trade blindly in the other direction.
- • Accurate cost allocation takes work to build and maintain, and shared resources never allocate perfectly cleanly.
- • Optimizing for cost can hurt latency and reliability — fewer replicas is cheaper and less resilient.
- • A cost budget per route adds a gate to shipping, which is friction that has to be worth what it prevents.
- • Alert when cost per request for a route rises more than a set percentage over the previous release at comparable traffic.
- • Track queries, external calls and response bytes per request as metrics; they move before the bill does and are attributable to a deploy.
- • Review provisioned-capacity changes (replicas, instance sizes, cache tiers) as cost changes, since they do not show up in code review.
Accuracy
Performance numbers are conditional. These are the conditions.
- ESTIMATEDThe component shares, per-request costs and before/after figures are arithmetic on illustrative unit prices. They demonstrate the method and the shape of the result, not any real service's economics.
- ENVIRONMENT-SPECIFICUnit prices, egress charges and inference rates vary by provider, region and contract, and change over time. Derive your own from your actual billing data rather than transferring these ratios.