Capacity or Efficiency: Which Problem Are You Solving?
Capacity asks how much load the system can take. Efficiency asks how much resource each unit of work consumes. A system can scale beautifully while wasting most of what it buys — and the two problems have different fixes, different costs and different urgency.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Two questions that get the same answer far too often
Capacity and efficiency are independent axes and they get confused constantly, because adding instances resolves both symptoms in the short term. A capacity problem means demand exceeds what the current fleet can serve: the answer is more capacity, or less demand. An efficiency problem means each request consumes more resource than it needs to: the answer is to make the work smaller.
The reason this matters is that scaling out papers over an efficiency problem indefinitely — expensively, and with compounding interest. Every future traffic increase multiplies the waste. Every dependency inherits the inflated per-request cost. The fleet grows in proportion to the inefficiency, and because each individual scale-out is small and justified, nobody notices the trend until someone plots machines per thousand requests over a year.
The diagnostic is straightforward. Plot resource per unit of work over time. If it is flat and traffic grew, you have a capacity problem and scaling is the correct response. If resource per unit of work has been climbing, you have an efficiency problem that scaling will follow you into.
| Capacity problem | Efficiency problem | |
|---|---|---|
| The question | How much load can we serve? | What does one unit of work cost us? |
| Signal | Utilization at peak, latency knee distance | CPU-seconds, queries or bytes per request over time |
| Typical cause | Traffic grew; the fleet did not | Per-request work grew: N+1s, payload bloat, serialization, retries |
| Correct fix | Add capacity, or shed/shape demand | Remove work: batch, cache, trim, profile the hot path |
| Cost of the fix | Continuous — the bill rises with the fleet | One-off engineering, then the bill falls |
| Cost of ignoring it | Immediate: queueing, timeouts, an incident | Compounding: every future scale-out multiplies the waste |
| What scaling out does | Solves it | Hides it, permanently and expensively |
Scaling past a wasteful system
A concrete shape. A service serves 10,000 requests per second on 40 instances, sitting at a comfortable 65% CPU. Nothing is on fire. But a CPU profile shows 55% of on-CPU time in JSON serialization, most of it re-serializing an unchanged configuration blob on every single request (Self Time, Total Time, and Where the CPU Went, Reading a Flame Graph).
That is not a capacity problem. Caching the serialized blob removes over half the CPU work per request, and the same traffic then fits on roughly 18 instances. The bill drops by more than half, latency improves as a side effect, and the headroom picture gets better rather than worse. No amount of scaling out would have produced any of that — it would have preserved the waste at larger and larger scale.
The tell was available the whole time and nobody was looking at it: CPU-seconds per request. Utilization stayed healthy because the fleet grew alongside the waste. Only the per-unit metric exposes the drift, which is the argument for tracking it as a standing metric rather than computing it during an investigation.
BEFORE AFTER (config blob serialized once, cached) traffic 10,000 req/s traffic 10,000 req/s instances 40 instances 18 CPU utilization 65 % CPU utilization 68 % CPU-sec / request 0.104 CPU-sec / request 0.047 p99 240 ms p99 180 ms relative spend 1.00x relative spend 0.45x WHAT THE DASHBOARDS SHOWED utilization 65 % -> 68 % "healthy" in both cases; no signal fleet size 40 -> 18 only visible after the change CPU-sec/request the actual finding — drifting up for three quarters WHY UTILIZATION HID IT The fleet grew as the waste grew, holding utilization constant. A per-unit metric cannot be held constant by adding instances.
When wasteful is the right call
Efficiency is not a virtue to be maximized. It is bought with engineering time, and engineering time has alternative uses. If a service costs $400 a month and the optimization takes two engineer-weeks, buying capacity is straightforwardly the correct decision — and saying so explicitly is better than doing it by default and feeling vaguely guilty.
The calculation changes with scale, growth rate and duration. Waste multiplied by a large fleet, or by a traffic curve that is still climbing, or by a service that will run for five more years, turns a small per-request inefficiency into a real number. The honest version of the decision is: monthly cost of the waste, times expected remaining lifetime, against the engineering cost of removing it — plus whatever latency improvement comes along for free, which often tips it.
Two things push toward fixing it regardless of the arithmetic. Efficiency problems in shared dependencies (the database, the cache) consume capacity that other services need, so the cost is not confined to your budget. And efficiency work usually improves latency at the same time, because the fastest work is work that does not happen — which means it is being counted against the wrong budget when treated purely as a cost exercise.
- Small fleet, low growth, short remaining life — buy capacity; the optimization does not pay back.
- Large fleet or steep growth curve — waste multiplies; fix it before it multiplies further.
- Waste in a shared dependency — you are consuming capacity other services need, so the cost escapes your budget.
- Efficiency fix that also cuts latency — count both benefits; most per-request work removal improves p99 too.
- No per-unit metric at all — start there; you cannot decide between these two problems without it.
Key points
- Capacity is how much load you can serve; efficiency is what each unit of work costs. They are independent.
- Plot resource per unit of work over time: flat means capacity problem, climbing means efficiency problem.
- Utilization cannot reveal efficiency drift, because the fleet grows alongside the waste and holds utilization constant.
- Scaling out hides efficiency problems permanently, and every future traffic increase multiplies the waste.
- Buying capacity is sometimes correct — small fleet, low growth, expensive engineering time — but it should be a decision, not a default.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Traffic → fleet: request volume grew 2×, fleet grew 3.4× — the ratio moved, so growth alone does not explain it.
- 2Fleet → per-unit: CPU-seconds per request rose from 0.061 to 0.104 across three quarters while utilization stayed near 65%.
- 3Per-unit → profile: a CPU profile attributes 55% of on-CPU time to JSON serialization of an unchanged configuration blob.
- 4Profile → work: the blob is re-serialized on every request; the result is identical every time.
- 5Work → fix: caching the serialized form removes the work entirely, and 10,000 req/s then fits on 18 instances instead of 40.
- • "Utilization is healthy, so the system is efficient" — utilization is held constant by the scaling that hides the waste.
- • "We scaled successfully, so there is no problem" — successful scaling is exactly what an efficiency problem looks like from the outside.
- • "Efficiency work is premature optimization" — it is premature before you have a per-unit metric, and overdue once that metric has been climbing for a year.
- • "Cost per request is falling, so efficiency is improving" — during growth, fixed cost dilution produces that curve on its own (Cost per Request: The Other Performance Metric).
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • CPU-seconds per request, memory per concurrent request, queries per request and bytes per request, tracked as standing metrics over months.
- • Machines per thousand requests per second, plotted over the last year — the single clearest picture of efficiency drift.
- • A CPU profile of a production instance, to attribute per-request cost to specific code paths ([[cpu-profiling]]).
- • Share of per-request cost attributable to work that could be eliminated rather than merely made faster.
- • Track resource per unit of work as a standing metric, so the drift is visible before it is expensive.
- • Profile the hot path and look specifically for work that can be eliminated, not merely made faster — repeated identical work is the richest seam.
- • Attack shared-dependency waste first, since it consumes capacity other teams are also paying for.
- • When buying capacity is the right call, record the decision and the number, so the next person can re-evaluate it rather than re-derive it.
- • Set a per-unit budget alongside the latency objective and treat a breach as a defect with an owner.
- • Confirm the per-unit metric fell, not just the total — the fleet can shrink for reasons unrelated to your change.
- • Hold traffic constant in the comparison, or compare like-for-like periods, since per-unit cost moves with load on its own.
- • Verify the fleet can be reduced and that latency and error rate hold at the smaller size before booking the saving.
- • Efficiency work costs engineering time that could go to features; below a certain scale it genuinely does not pay back.
- • Highly optimized code can be harder to read and change, and the optimization can be invalidated by the next requirement.
- • Buying capacity is faster and reversible; the cost is continuous and compounds with every future traffic increase.
- • Alert when CPU-seconds per request or queries per request rises beyond a threshold over a release at comparable traffic.
- • Put machines per thousand RPS on the capacity review, so drift is discussed on a schedule rather than discovered.
- • Add a per-request-cost assertion to the CI benchmark for the hottest routes (Regression or Tuesday? Telling a Real Change from Noise).
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe 40-to-18 instance example, CPU-seconds per request and spend ratios are teaching figures showing the shape of an efficiency win. Real profiles rarely have a single 55% culprit that caches away this cleanly.
- WORKLOAD-SPECIFICWhether efficiency work pays back depends on fleet size, growth rate, remaining service lifetime and the cost of engineering time. The arithmetic is transferable; the conclusion is not.