Load Testing: What Question Is This Test Answering?
Baseline, load, stress, spike and soak are five different tests answering five different questions. Most load tests fail before they start — against a warm cache, an empty database and one hot key, they measure a system that does not exist.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Five tests, five questions
Calling all of these "load testing" is why teams run one test and believe it answered every question. They do not overlap much: a stress test tells you nothing about memory leaks, and a soak test tells you nothing about your saturation point. Pick the test that matches the question you actually have, and be explicit about which questions it leaves unanswered.
The most common mistake is running only a load test — a steady rate at expected peak — and concluding the system is ready. That test answers "can we serve expected peak right now, from this starting state". It does not answer what happens above peak, what happens when peak arrives suddenly, or what happens after six hours of it. Those are three separate tests.
Whichever you run, the output should be a curve, not a pass mark. Throughput and latency across a range of loads tell you where the knee is and how much room you have. A single "passed at 5,000 RPS" hides whether you passed at 4,999 with three seconds to spare or had double the headroom.
| Test | Question it answers | Finds | Blind to |
|---|---|---|---|
| Baseline | What does one user experience on an idle system? | Per-request cost floor, obvious regressions | Everything about contention and scale |
| Load | Can we serve expected peak? | Whether the objective holds at target rate | Behavior above peak; how close the knee is |
| Stress | Where does it break, and how? | The saturation point and the failure mode past it | Whether it degrades gracefully over time |
| Spike | What happens when load arrives suddenly? | Autoscaling lag, cold caches, connection storms (Autoscaling Lag: The Gap Where the Outage Lives) | Slow accumulating problems |
| Soak | What happens after hours at load? | Leaks, fragmentation, disk fill, connection churn, lag drift | Peak capacity and burst behavior |
The test that proves nothing
A load test measures the system you set up, and the default setup is not the system you run. Three defaults do most of the damage: a database seeded with a thousand rows when production has fifty million, a request generator that hits the same handful of ids so every lookup is a cache hit, and a cache pre-warmed by the smoke test that ran first.
Each of those individually inflates the result. Together they can produce a number several times higher than reality, and they do it in a way that looks completely legitimate — the requests are real, the responses are correct, the errors are zero. Nothing in the output says "your working set fits in memory and production's does not" (An Index Scan Is Not Automatically Faster explains why the dataset size changes the plan, not just the timings).
The fix is to make the test environment wrong in the same ways production is: production-scale data, a realistic key distribution including the hot keys and the long tail, a cache started cold or at production hit rate, and the same request mix — including the expensive endpoints that are only 2% of traffic and 40% of load.
1dataset: 1,000 rows # production: 50,000,0002key access: ids 1..100 uniform # production: zipfian, hot keys3cache: warmed by smoke test # production: cold after deploy4request mix: 100% GET /health-ish # production: 2% heavy reports5duration: 3 minutes # production: hours6think time: none # closed loop, see coordinated-omission7 8RESULT: 5,000 req/s, p99 45 ms, 0 errors9REALITY: the whole working set was in memory and the10 expensive endpoints were never exercised.1dataset: production-scale restore or synthetic at scale2key access: sampled from production access logs (keeps the tail)3cache: started cold; report warm and cold separately4request mix: route proportions from production traffic, heavy routes included5duration: long enough to see steady state, plus a separate soak6load model: open (constant arrival), latency from intended send time7 8RESULT: a curve — throughput and p99 across a load range,9 with the knee identified and headroom stated.The bad setup is not a smaller version of production, it is a different system: different query plans, different cache behavior, different bottleneck. Numbers from it are precise and unrelated to what you are about to deploy.
Reading the result: find the knee
A load test's most valuable output is the shape of the curve. As offered load rises, throughput rises with it and latency stays roughly flat — the system is keeping up. At some point throughput stops rising: every additional request joins a queue instead of a free server. Latency, which had been flat, starts climbing steeply. That inflection is the capacity number worth recording (Queueing: Why Systems Get Slow Before They Get Broken, Saturation: The Reading Utilization Cannot Give You).
Past the knee, more offered load produces no more throughput and dramatically more latency. Load tests that report only the peak throughput achieved often report a number from past the knee, where the system was technically completing requests while every user waited eight seconds. Capacity is the knee, not the maximum.
Two things to record alongside it. Which resource saturated first — CPU, connections, a downstream dependency — because that tells you what to fix and what the next bottleneck will be (The Bottleneck Moves After Every Fix). And the error behavior past the knee: a system that sheds load cleanly is in far better shape than one that accepts everything and times out, even though the second looks better in a graph of successful requests.
| Signal | Value | What it tells you | Verdict |
|---|---|---|---|
| 1,200 req/s offered | throughput 1,200 · p99 95 ms | Below the knee: throughput tracks offered load, latency flat | normal |
| 2,000 req/s offered | throughput 1,980 · p99 210 ms | Approaching the knee: latency rising faster than load | suspect |
| 2,400 req/s offered | throughput 2,050 · p99 1,900 ms | Past the knee: throughput flat, latency exploding — this is capacity | smoking gun |
| First resource to saturate | DB connection pool waiters > 0 at 2,050 | Names the constraint and predicts the next one after it is fixed | smoking gun |
| Behavior past the knee | Accepts all, times out at 30s | No shedding: every user waits instead of some failing fast | suspect |
Key points
- Baseline, load, stress, spike and soak answer different questions; running one and generalizing is the standard mistake.
- Test setup decides the result: production-scale data, realistic key distribution and a cold cache change the number by multiples.
- The valuable output is a curve, not a pass mark — throughput against latency across a load range.
- Capacity is the knee, where throughput stops growing and latency starts climbing, not the maximum throughput observed.
- Record which resource saturated first and how the system behaved past the knee; both predict production behavior more than the peak number does.
Progressive depth
Overview
A load test puts artificial traffic through the system to find out what it can handle before real users find out for you.
Practical
Choose the test that matches the question: baseline, load, stress, spike or soak. Fix the setup first — production-scale data, realistic key distribution, honest cache state, real request mix — then run a ramp and report the curve.
Advanced
Capacity is the knee, not peak throughput. Record which resource saturated first and how the system behaved past the knee. Watch for closed-loop load generators hiding latency (Coordinated Omission: When the Load Generator Lies) and for cache warmth inflating everything.
Internals
The knee is queueing behavior: below it, arrivals mostly find a free server; above it, wait time grows without bound as utilization approaches one. Its position depends on service-time variability — the more variable each request's cost, the earlier the knee arrives for a given latency objective, which is why heavy-tailed workloads saturate at lower utilization than uniform ones.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Test setup → dataset: 1,000 rows means the working set fits in memory and query plans differ from production's.
- 2Dataset → cache: uniform access over 100 ids produces a hit rate no production traffic distribution will reproduce.
- 3Cache → throughput: with nearly every read served from memory, the database never becomes the constraint during the test.
- 4Throughput → conclusion: the test reports 5,000 req/s, which is the capacity of a system with an in-memory working set.
- 5Production → reality: at 1,800 req/s the real access distribution misses cache, the database becomes the constraint, and the fleet passes its knee.
- • "We hit target RPS with zero errors, so we are ready" — zero errors says nothing about how close to the knee you were.
- • "Peak throughput was 2,400 req/s" — if p99 was two seconds there, that is past the knee and not usable capacity.
- • "The load test passed, so production will be fine" — only if the dataset, key distribution, cache state and request mix matched.
- • "Latency was flat throughout" — flat latency with throughput below offered load means requests were being dropped or queued upstream.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Throughput and p50/p95/p99 at each step of a ramp, so the knee is visible rather than inferred.
- • Resource utilization on every tier during the test — app CPU, database connections, cache hit rate — to identify what saturated first.
- • Error rate and error type past the knee: timeouts, shed requests, or accepted-and-slow.
- • Cache hit rate during the test compared to production, as the single most common source of inflated results.
- • State the question before designing the test, and pick the test type that answers it; note explicitly what it will not tell you.
- • Fix the setup first: production-scale data, key distribution sampled from real access logs, honest cache state, real request mix.
- • Run a ramp and report the curve with the knee identified, rather than a pass/fail at a single rate.
- • Record the first resource to saturate and the failure mode past the knee, since those determine what to fix next.
- • Add a separate spike and soak test for services where burst behavior or long-run degradation matter — they cannot be inferred from a steady-rate test.
- • Compare the load test's predicted knee to the rate at which production latency degrades at the next real peak; record the gap.
- • Confirm cache hit rate and query plans during the test resemble production before trusting any throughput number.
- • Re-run after fixing the first bottleneck and confirm the knee moved and a different resource now saturates first.
- • A realistic environment costs money and maintenance; a cheap one produces numbers that are worse than having none.
- • Ramp tests take longer than a single-rate test and need someone to interpret a curve rather than read a pass mark.
- • Testing to the knee means deliberately degrading a system, which needs an environment where that is safe and a window where it is allowed.
- • Run the ramp test on a schedule and track the knee over releases; a falling knee is a capacity regression even when nothing failed.
- • Alert if the load-test environment's dataset size or cache hit rate drifts away from production, which silently invalidates every subsequent result.
- • Keep the request mix generated from recent production traffic rather than a fixed script that ages out of relevance.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe ramp figures, throughput values and the 5,000-versus-1,800 discrepancy are teaching examples showing the shape of the failure. Your knee and your inflation factor come from your own comparison of test to production.
- WORKLOAD-SPECIFICHow much an unrealistic dataset or cache state inflates results depends entirely on how much of your working set fits in memory and how skewed your access distribution is.