Performance Testing a Backend
Latency, throughput, concurrency, CPU, memory, database load and dependency load are seven different dimensions — a single "requests per second" number answers almost none of them.
The requirement, the obvious build, and why it breaks
Every lesson starts where the work starts: someone asked for something, and the first implementation that comes to mind has a problem.
How do I find out how this service behaves under load before production finds out for me?
A campaign will multiply traffic on Monday. We need to know whether the service holds, what breaks first, and what to scale — with enough lead time to do something about it.
Point a load tool at the busiest endpoint, ramp up until errors appear, and record the requests per second at that point as our capacity.
One endpoint is not the workload. Real traffic is a mixture, and the mixture is what saturates a shared resource — a pool exhausted by a rare expensive query is invisible when you hammer a cheap one (Connection Pools).
- One endpoint is not the workload. Real traffic is a mixture, and the mixture is what saturates a shared resource — a pool exhausted by a rare expensive query is invisible when you hammer a cheap one (Connection Pools).
- The number found is the point where it *broke*, which is well past the point where it stopped being acceptable. Useful capacity is bounded by latency, not by the error threshold.
- Averaged latency hides everything that matters. A run reporting a healthy mean can have a tail in seconds, and the tail is what users and dependent services experience (Tail Latency: Why p50 Being Fine Does Not Help).
- A closed-loop load generator that waits for each response before sending the next cannot produce the queue that real open-loop traffic produces — so it systematically under-measures latency under saturation (Coordinated Omission: When the Load Generator Lies).
- It tests one dimension. Throughput was measured; memory growth over a sustained run, CPU headroom, database load and dependency saturation were not, and any one of them can be the real limit.
What is actually happening
- Performance is not a scalar. Seven dimensions move independently, and a system is limited by whichever saturates first — which is usually not the one being measured.
- Latency is per-request time, and only meaningful as a distribution. Throughput is completed work per second. They are related through concurrency and are not substitutes (Little's Law as Working Intuition).
- Concurrency is how many requests are in flight. It is the dimension that turns a working system into a queueing one, and it is what most load tests control badly.
- CPU and memory are the process's own resources. CPU saturation shows as rising latency across everything at once; memory problems show as growth over time and as pauses, not as a step change (Memory Leaks in Backend Services).
- Database load is a shared, finite resource behind your service — connections, locks, IO. A backend can be nowhere near its own limits while the database is at its limit (Which Signal Actually Means "The Database Is Slow").
- Dependency load is capacity you do not own. Your throughput can be capped by a third party's rate limit, and the test that finds this out is worth running before the campaign.
- Open-loop versus closed-loop load generation is the distinction that decides whether the numbers mean anything: open-loop sends at a target rate regardless of responses and reproduces real queueing; closed-loop self-throttles when the system slows and hides exactly the behaviour you are testing for (Load Test Shapes: The Shape Is the Hypothesis).
- Different questions need different shapes: a steady load test finds the sustainable rate, a stress test finds the breaking point and the failure mode, a soak test finds leaks and drift, and a spike test finds whether recovery works (Load Testing: What Question Is This Test Answering?).
Seven dimensions, and the one that saturates first
A capacity number is only meaningful alongside the dimension that produced it. The practical value of a performance test is identifying which resource runs out first, because that is the only thing worth changing — improving anything else moves no number at all.
The last column matters most operationally: each dimension has a signal that moves *before* users notice, and watching those during a run tells you more than the final result does.
| Dimension | What it measures | Saturation looks like | Watch during the run |
|---|---|---|---|
| Latency | Per-request time, as a distribution | The tail separates from the median | p50 / p95 / p99 versus arrival rate (Percentiles: Which One, and How Many Users Is That?) |
| Throughput | Completed requests per second | Plateaus while arrival rate keeps climbing | Completed versus offered rate — the gap is failure or queueing |
| Concurrency | Requests in flight | Climbs steadily while throughput is flat | In-flight gauge (Little's Law as Working Intuition) |
| CPU | Compute headroom in the process | Latency rises across every endpoint at once | CPU per instance, and loop lag or scheduler latency (CPU Saturation: When Cores Become the Queue) |
| Memory | Heap and RSS over time | Growth across a soak; longer or more frequent GC pauses | RSS trend, GC pause time (Garbage Collection: Pause, Throughput, Footprint — Pick Two) |
| Database load | Connections, query latency, locks | Pool wait time above zero; query latency rising | Pool in-use and wait, slow-query rate (Connection Pool Saturation: Waiting in Front of an Idle Database) |
| Dependency load | Third-party capacity you do not own | 429s, or their latency rising while yours is idle | Per-dependency latency and error rate (Calling Something You Do Not Control) |
Open loop or closed loop decides whether the numbers mean anything
This is the single most consequential detail in load testing and the one most often left as a tool default. A closed-loop generator holds a fixed number of virtual users, each sending the next request only after the previous response. When the system slows, the generator slows with it — so the offered load drops exactly when you needed it to stay constant.
The result is systematically optimistic percentiles: the slow period is under-sampled because fewer requests were sent during it. Real traffic does not behave that way. Users and upstream services keep arriving at their own rate regardless of how you are doing, and that is what builds a queue (Coordinated Omission: When the Load Generator Lies).
Four shapes, four different questions
Teams say "load test" for four distinct experiments that answer different questions and have different designs. Running the wrong shape is why a test can be executed carefully and still tell you nothing about the risk you were worried about.
Before a traffic event, the useful pairing is usually a steady test at the expected multiple to confirm the target holds, and a stress test to learn what breaks first and how — because knowing the failure mode is what lets you prepare for it.
What do you actually need to know before Monday?
when "Can we sustain the expected traffic within our latency target?" Hold a realistic mixture at a fixed arrival rate.
cost Says nothing about the breaking point or about recovery.
when "What breaks first, and does it fail gracefully or cascade?" Ramp past the target until it degrades.
cost Destructive; needs an isolated environment and a plan for cleanup.
when "Does anything grow?" Leaks, unbounded caches, connection drift, log-disk growth.
cost Slow and expensive to run; easy to skip, and the failures it finds are found in production otherwise (Memory Leaks in Backend Services).
when "Can we absorb a sudden jump, and do we recover after it passes?" Step the rate up sharply, then back down.
cost Recovery behaviour is the interesting half and the half usually not measured (Queue Backlog).
when "What happens when a third party gets slow rather than failing?" Inject latency into a dependency under normal load.
cost Needs a proxy or fault injection; frequently reveals a missing timeout (Timeouts).
when "Did this change make things worse?" A small, fixed workload run on every merge against a baseline.
cost Noise control is the hard part; a flaky gate gets disabled (Regression or Tuesday? Telling a Real Change from Noise).
How to build it
Most important first.
- State the question before the test. "Can we serve 3x Monday traffic with p99 under our target" is testable; "how fast is it" is not.
- Model the real workload mixture, including the expensive rare endpoints, realistic payload sizes and realistic cache hit rates. A test against a warm cache and a tiny dataset measures a system you do not run (A 95% Hit Rate Tells You Almost Nothing).
- Use open-loop generation at a target arrival rate, and record the full latency distribution rather than an average (Percentiles: Which One, and How Many Users Is That?).
- Measure all seven dimensions during the run: latency percentiles, throughput, in-flight count, CPU, memory over time, database connections and query latency, and dependency latency and error rate (The Metrics a Backend Must Emit).
- Test with production-shaped data volume. Query plans change with table size, and a test on ten thousand rows says nothing about ten million (Reading EXPLAIN ANALYZE).
- Run a soak test long enough to reveal growth. Leaks, unbounded caches and fragmenting heaps are invisible in a five-minute run.
- Record what fails first and how it fails. The failure mode is more valuable than the number: graceful shedding and a cascading collapse are different outcomes at the same throughput (Cascading Failure).
- Stub or budget third parties deliberately. Hitting a real payment sandbox with load tests is both misleading and rude — but do test what happens when a dependency is slow.
- Re-run on a schedule and compare against a baseline. A single result is a snapshot; regression detection is where the ongoing value is (Regression or Tuesday? Telling a Real Change from Noise).
What can go wrong
- The load generator saturating before the system does, so the plateau being measured is the client's limit. Always verify the generator has headroom.
- Coordinated omission: a closed-loop generator stops sending while a request is slow, so the slow period is under-sampled and the reported percentiles are optimistic.
- Testing against an unrealistic dataset, so the database serves everything from cache and the plan is nothing like production's.
- A test environment with different instance sizes, different network topology or a smaller database, producing numbers that do not transfer.
- Load-testing production without a plan — and creating the incident you were trying to prevent.
- The mitigation failing: a performance test in CI that is too noisy to gate on, so it is set to warn-only and then ignored (Benchmark Fallacies: Confident Numbers That Are Wrong).
- Measuring only steady state and never recovery, so nobody knows whether the system drains its queue or stays saturated after the spike passes (Queue Backlog).
- Load tests are one of the few practical ways to surface real concurrency bugs: races that need a specific interleaving appear under sustained concurrent load and almost never in a functional suite (Backend Races).
- Contention itself is a performance dimension. Lock waits and deadlock retries show up as latency variance under load, not as errors (Deadlocks in Application Code).
- A load test is indistinguishable from a denial-of-service attempt. Coordinate with anyone who operates infrastructure in the path, and never point one at a third party without permission.
- Never use production personal data in a test environment. Generate synthetic data of realistic shape and volume instead (Sensitive Data Classification).
- Rate limits and quotas are part of the system under test. A load test that bypasses them measures a system that does not exist in production (Rate Limiting).
- Load testing reveals resource-exhaustion vulnerabilities: an endpoint that allocates proportionally to input, a regex with pathological backtracking, an unbounded query. Those are security findings as much as performance ones (Unbounded Concurrency).
- Test with authentication and authorization enabled. Disabling them to simplify the test removes real per-request work and often a real bottleneck.
- "We can do 5000 requests per second." At what latency, with what mixture, with what cache state, against what data volume, with which dependencies real? Without those, the number is not transferable (Benchmarking: Does This Number Answer My Question?).
- "The average latency is fine." Averages hide the tail, and the tail is what users experience and what dependent services time out on (The Average Was Fine and Users Were Not).
- "It held for five minutes, so it holds." Leaks, unbounded caches and connection drift need a soak test to appear.
- "Load testing tells us where the bottleneck is." It tells you the system saturated. Finding *why* is profiling and tracing, and that depth lives in Observability & Performance (Measure Before You Optimize).
- "More concurrency means more throughput." Past the saturation point, more concurrency means more queueing and worse latency at the same throughput (Little's Law as Working Intuition).
- "We tested the API, so we tested the system." Background workers, scheduled jobs and queue consumers have their own capacity limits and often share the same database (Worker Scaling).
Operating it
- During a run, watch the saturation signals rather than the result: in-flight count, pool wait time, queue depth, event-loop lag or thread-pool saturation. They move before latency does (Saturation: The Reading Utilization Cannot Give You).
- Graph latency percentiles against arrival rate. The knee in that curve is the useful capacity number, and it is well below the error threshold.
- Watch memory across a soak run. Flat is healthy; a sawtooth that trends upward is a leak wearing a garbage collector's clothing (Leak or Unbounded Cache? The Question That Picks the Fix).
- Compare your service's latency against your dependencies' latency during the run. If yours rises and theirs does not, the bottleneck is yours.
- Record the environment with every result — instance type, data volume, config, build sha — or results are not comparable across runs ("What Changed?" — Deploy Markers and the Invisible Deploys).
- At 10x, the resource that saturates first usually changes. Capacity found at one scale does not extrapolate, which is why the test is re-run rather than reasoned about (The Bottleneck Moves After Every Fix).
- At 100x, the load generator itself becomes a distributed system with its own coordination and its own bottlenecks.
- Shared dependencies do not scale with your instances: adding application capacity moves the bottleneck to the database or to a third party, often abruptly (Read Replicas From the Application).
- The failure mode changes with scale too. What sheds load gracefully at one size can cascade at another, because retry amplification grows with the number of clients (Retry Storms).
- A realistic test environment is expensive — production-like instances and production-like data volume, for a test that runs occasionally.
- Building a realistic workload model is real work, and it decays as traffic patterns change.
- Testing in production gives the most accurate answer and risks an incident. Testing in staging is safe and gives numbers that need interpretation.
- Performance tests in CI catch regressions early and are noisy. Making them stable enough to gate on is its own engineering effort.
Where this applies
Backend advice is context-sensitive. These labels say what each claim is specific to, and where a different stack or scale would differ.
- GENERALThe seven dimensions and the open/closed-loop distinction apply to any backend on any stack.
- RUNTIME-SPECIFICWhat saturates first is a runtime property: a Node service is usually limited by the single loop thread, so CPU work anywhere blocks everything (Blocking the Event Loop); CPython by the GIL and worker-process count; the JVM by thread-pool sizing and GC pauses, which need warm-up before measurements mean anything (JIT and Warm-Up: The First Thousand Requests Are a Different Program); Go by goroutine scheduling and GC pressure. Measure the runtime-specific saturation signal, not just CPU percentage.
- SIMULATEDNo latency, throughput or capacity figures are given anywhere in this lesson, deliberately. Any number would be a fabrication: real values depend on hardware, data volume, workload mixture, configuration and dependencies. What transfers is the shape — which dimension saturates first, and what the curve does at the knee.
- SCALE-SPECIFICFor a low-traffic internal service, a short load test before a known event is proportionate. Continuous performance testing with regression gating is worth its cost when a latency regression has real business consequences or when many teams change the same service.
Where the depth lives
This domain teaches the application-side mechanism and hands the rest off.
- — Testing & Reliability Engineering — fault injection, chaos experiments and game days: verifying that the failure modes a stress test revealed are actually handled.