Capacity Planning: Traffic to Machines
Average traffic, a peak multiplier, per-request cost and a latency target become a instance count through five multiplications — each one an assumption you can name, challenge and re-measure. The output is an estimate, and saying so is what makes it useful.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The chain from traffic to instances
Capacity planning is not a formula, it is a chain of multiplications where every link is an assumption. Average request rate times a peak multiplier gives peak rate. Peak rate times average latency gives concurrent in-flight requests (that is Little's Law as Working Intuition, used as arithmetic). Concurrency divided by what one instance can hold gives instances. Divide by your target utilization and you have the number to provision.
What makes this useful is not the arithmetic — it is that each link is a number someone can argue with. "We assumed a 3× peak multiplier" is a challengeable claim; "we need 40 instances" is not. When the estimate turns out wrong, you find out *which link* was wrong, and the next estimate is better. That feedback loop is the entire value of writing it down.
The per-instance capacity number is the one people guess and should measure. Load the service until latency degrades past your objective, note the request rate, and use that — not a vCPU count, not a vendor benchmark, and not what a similar service does. This is the one input that a single afternoon of Load Testing: What Question Is This Test Answering? converts from a guess into a measurement.
| Link | Where the number comes from | How it goes wrong |
|---|---|---|
| Average RPS | Production metrics over a representative window | A window that excludes the last launch, sale or seasonal peak |
| Peak multiplier | Peak-to-average ratio from your own traffic history | Borrowed from another company; peaks are business-specific, not universal |
| Concurrency = RPS × latency | Little's Law, using p50 or mean service time | Using p50 when the tail is what saturates you — see Tail Latency: Why p50 Being Fine Does Not Help |
| Per-instance capacity | A load test to the latency knee, not a vCPU count | Guessed from instance size; ignores the actual bottleneck resource |
| Target utilization | A headroom decision — see Headroom: The Capacity You Deliberately Do Not Use | Set to 100%, which provisions for the average and fails at the peak |
A worked estimate, with every assumption on the page
Here is the whole calculation for a checkout service. Every line is labelled ESTIMATED because that is what it is: a derivation from stated inputs, not a measurement of a running system. The inputs marked *measured* came from production metrics and a load test; the rest are judgement calls that belong in the document next to the answer.
Notice how much the answer moves when one assumption moves. Change the peak multiplier from 3× to 5× and the fleet goes from 18 to 30 instances. Change target utilization from 60% to 80% and it drops to 14 — which is why "how much headroom" is a capacity decision, not an afterthought. An estimate that does not show its sensitivity invites false confidence.
The output is a range, not a number. Provision for the top of the range if scaling is slow (see Autoscaling Lag: The Gap Where the Outage Lives), and for the middle if you can scale in seconds and shed load safely. Either way, write down which one you chose and why.
INPUTS average rate 1,200 req/s (measured, 28-day p50 of 1m rate) peak multiplier 3.0x (assumption: observed Black Friday ratio) mean service time 120 ms (measured, mean of request duration) per-instance capacity 220 req/s (measured: load test to the latency knee) target utilization 60 % (decision: see headroom) DERIVATION peak rate = 1,200 x 3.0 = 3,600 req/s concurrency = 3,600 x 0.120 s = 432 in-flight requests raw instances = 3,600 / 220 = 16.4 provisioned = 16.4 / 0.60 = 27.3 -> 28 instances plus N+1 for one-instance failure -> 29 instances SENSITIVITY (what happens if one assumption is wrong) peak multiplier 5.0x instead of 3.0x -> 46 instances target utilization 80% instead of 60% -> 21 instances service time 200 ms instead of 120 ms -> 28 instances (concurrency 720) CONFIDENCE Weakest link: the peak multiplier. It is one observation, not a distribution.
What the estimate cannot tell you
The arithmetic assumes the service scales linearly and that nothing downstream notices. Neither is true. Doubling the application tier doubles the connections arriving at the database, and the database was never in the model — this is the most common way a correct capacity plan produces an incident (see Connection Pool Saturation: Waiting in Front of an Idle Database).
Three other places the model breaks. Latency is not constant with load: near saturation it rises sharply, so the concurrency figure you computed from a lightly loaded p50 understates in-flight work exactly when it matters (Queueing: Why Systems Get Slow Before They Get Broken). Cache hit rates change with scale and with traffic mix, and a hit-rate drop multiplies backend load (A 95% Hit Rate Tells You Almost Nothing). And new instances are not immediately useful — cold caches, cold connection pools and JIT warm-up mean the tenth instance serves at reduced capacity for its first minutes (JIT and Warm-Up: The First Thousand Requests Are a Different Program).
So the estimate is a starting point that gets validated by a load test and corrected by production. Treat it as a hypothesis with an expiry date: re-derive it when traffic shape changes, when a dependency is added, or when per-request cost moves. A capacity plan nobody has re-run in a year is a historical document.
- Downstream multipliers: each app request becomes 2.4 database queries — the database plan must be derived from the app plan, not separately.
- Third-party quotas are hard ceilings the arithmetic ignores until a 429 storm arrives (see The Rate-Limit Contract).
- Latency rises with utilization, so concurrency at peak exceeds the figure computed from an idle-system p50.
- New capacity starts cold — a scale-out event temporarily lowers average per-instance capacity.
Key points
- Capacity planning is a chain of named assumptions; the value is in being able to say which link was wrong afterwards.
- Per-instance capacity is the input people guess and should measure — load to the latency knee, not to a vCPU count.
- Always publish the sensitivity: if a 3× peak assumption becoming 5× doubles the fleet, that assumption deserves scrutiny.
- The model covers the tier you sized; downstream databases, caches and third-party quotas need their own derivation.
- The output is an ESTIMATE with an expiry date, validated by a load test and corrected by production reality.
Progressive depth
Overview
Capacity planning answers "how much do we need to buy". Traffic times cost per request, plus room to spare.
Practical
The chain: average rate → peak rate (multiplier) → concurrency (rate × latency) → instances (÷ per-instance capacity) → provisioned (÷ target utilization), plus a spare for failures. Measure per-instance capacity; challenge every other input.
Advanced
Latency is not constant with utilization, so concurrency computed from an idle p50 understates peak in-flight work. Downstream tiers have their own multipliers and their own scaling rules. Publish sensitivity, because the answer often moves more than the input does.
Internals
The arithmetic is Little's Law plus a utilization target chosen from queueing behavior: as utilization approaches one, wait time grows sharply, which is why targets sit at 60–80% rather than 95%. The knee's position depends on service-time variability — highly variable service times push the usable target lower for the same latency objective.
Capacity Planner
Change an input and watch which number moves — and which one does not.
Every output below is derived from the inputs you set. It assumes work distributes evenly, that per-request cost stays constant as load rises (it does not — contention grows), and that CPU is the binding constraint. Use it to size a starting point and to notice when an assumption is absurd, then validate with a load test.
- • At 60% target utilisation you are buying 40% headroom. That is a defensible middle: enough slack to absorb a spike without paying for double the fleet.
- • 288 concurrent requests is the number every pool, thread count and connection limit on this path must exceed. If your database pool is smaller than this, the pool is your real capacity limit, not the CPU.
- • This sizes for a peak of 3× average. Measure that multiplier from a real peak — teams routinely under-estimate it, and the cost of being wrong is asymmetric.
- • Instances are sized on CPU alone. If the service is I/O-bound, CPU will never be the constraint and this number is meaningless — size on concurrency instead.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Traffic history → peak rate: 1,200 req/s average with an observed 3× peak gives 3,600 req/s to plan against.
- 2Peak rate → concurrency: 3,600 req/s × 120 ms mean service time means roughly 432 requests in flight at any instant.
- 3Concurrency → instances: at a measured 220 req/s per instance the raw need is 16.4, before any headroom.
- 4Headroom → provisioned fleet: a 60% utilization target and one spare for instance failure gives 29.
- 5Fleet → downstream: 29 instances at 2.4 queries per request puts 8,640 queries/s on a database that was never in the calculation.
- • "We are at 40% CPU, so we have 60% more capacity" — utilization is not linear with headroom; latency degrades well before 100%.
- • "The instance is 8 vCPU so it handles 8× the small one" — capacity is set by the bottleneck resource, which is often not CPU.
- • "Last year's peak multiplier is this year's peak multiplier" — one observation is not a distribution, and the business changed.
- • "We scaled the app tier, so we are covered" — the database, cache and third-party quotas scale by different rules or not at all.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Average and peak request rate per route from production metrics over at least one full business cycle, not a quiet week.
- • Per-request resource cost: CPU seconds and peak RSS per request, derived from utilization divided by request rate at a known load.
- • Per-instance capacity: the request rate at which p99 crosses your latency objective in a load test with a realistic dataset.
- • Downstream multipliers: queries per request, cache operations per request, external API calls per request — from traces.
- • Measure per-instance capacity with a load test to the latency knee, replacing the largest guess in the chain with a number.
- • Write the derivation down with each assumption labelled and its source, so the next person debugs the assumption rather than the conclusion.
- • Publish the sensitivity analysis alongside the answer; provision toward the pessimistic end when scale-up is slow.
- • Derive downstream capacity (database connections, cache memory, third-party quota) from the app-tier plan rather than in a separate document.
- • Set a re-derivation trigger: a traffic-shape change, a new dependency, or a per-request cost regression.
- • Run a load test at the planned peak rate against the planned fleet size and confirm p99 stays inside the objective with the assumed dataset.
- • Compare predicted per-instance capacity to what production shows at the next real peak; record the delta and correct the model.
- • Check that downstream signals (connection pool waiters, cache eviction rate, third-party 429s) stay flat at planned peak, not just app-tier CPU.
- • Provisioning for the pessimistic end of the range costs money continuously to avoid an event that may not happen this quarter.
- • Measuring per-instance capacity properly costs a load-test environment with realistic data — the cheap version produces confident nonsense.
- • A detailed model is more accurate and more likely to go stale; a simple model is easier to keep current and easier to be wrong with.
- • Track per-request CPU seconds and per-request query count as metrics; alert when either drifts up by more than a set percentage over a release.
- • Re-run the capacity derivation on a schedule and on dependency changes, with the previous estimate and actual outcome shown next to it.
- • Add a load test at planned peak to the release process for services where a scale-out cannot happen within the spike duration.
Accuracy
Performance numbers are conditional. These are the conditions.
- ESTIMATEDEvery figure in the worked example is derived from the stated inputs. Instance counts, concurrency and sensitivity ranges are arithmetic on assumptions, not measurements of any real service.
- WORKLOAD-SPECIFICPeak multipliers, per-request cost and per-instance capacity depend entirely on your traffic mix, dataset size and code path. Borrowing any of these three numbers from another system is the main way capacity plans fail.
Misconceptions
Where the depth lives
The same discipline — name the assumption, do the multiplication, state the confidence — is what separates a defensible design interview answer from a guess.