Coordinated Omission: When the Load Generator Lies
A load generator that waits for each response before sending the next one stops sending requests exactly when the system stalls. The requests that would have been slowest are never issued, never measured, and the reported p99 can be an order of magnitude better than what users experience.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
The bug is in the measuring instrument
Most simple load generators use a closed loop: each virtual user sends a request, waits for the response, then sends the next. This is convenient and it introduces a systematic error, because the send rate is now coupled to the system's response time. When the server stalls for two seconds, every virtual user is blocked waiting — and the requests they would have sent during that stall are never sent at all.
Those unsent requests are the problem. In production, arrivals do not pause because your server is busy; users keep clicking and the queue keeps growing. The requests that arrive during a stall are exactly the ones that experience the worst latency. A closed-loop generator omits precisely that population, and it omits them in a correlated way — hence *coordinated* omission.
The result is a measurement that is internally consistent, statistically clean, and describing a different system. Every request that *was* measured really did take the time recorded. The distribution is wrong because of what is missing from it, which is why no amount of scrutiny of the reported numbers reveals the problem.
1# CLOSED LOOP — send-after-response2for each virtual_user:3 loop:4 t_start = now()5 send_request()6 await_response() # <-- blocks for the full stall7 record(now() - t_start) # honest for THIS request8 sleep(think_time)9 10# During a 2s server stall with 100 VUs and 50ms normal service time:11# requests that WOULD have been sent: ~4,00012# requests actually sent: 10013# requests recorded as slow: 10014# requests omitted from the sample: ~3,900 <-- the worst ones15 16# OPEN MODEL — send on schedule, measure from INTENDED time17schedule = constant_arrival(rate = 2000/s)18for each scheduled_time in schedule:19 if now() > scheduled_time:20 backlog += 1 # we are already behind; do not skip21 send_request_async()22 record(response_time_from = scheduled_time) # not from actual sendWhat the omission does to the numbers
The distortion concentrates in the tail, which is the part anyone cares about. The median is barely affected — most requests happen when the system is fine, and those are measured correctly in both models. The upper percentiles are where the missing population lived, so p99 and p99.9 can be wrong by an order of magnitude while p50 is nearly exact.
This is a particularly nasty failure because it makes the tail look *tighter* than reality, which is the direction that produces confidence. A report showing p50 48 ms and p99 180 ms describes a well-behaved system with a modest tail. The corrected measurement of the same run shows p99 above two seconds — a system with a serious stall problem, and one that would have failed its objective.
The corrected markers on the distribution below come from the same test run, with latency attributed from intended send time rather than actual send time. Nothing about the system changed; only the instrument was fixed.
Fixing the instrument
There are two practical fixes and one thing to be aware of. The first fix is an open model: schedule sends at a constant arrival rate and issue them regardless of whether previous responses have returned. Arrivals then behave like real users, the queue grows during a stall, and the requests that arrive into that stall are measured. Most serious load tools support this — it is variously called an open model, constant arrival rate, or a rate-based rather than user-based executor.
The second fix is to measure from intended send time. Even in a closed loop, if a request was scheduled for t=1000ms and only issued at t=2400ms because the previous one was stalled, the honest latency is measured from 1000ms, not 2400ms. Some tools apply this correction automatically; others let you reconstruct it from the schedule.
The thing to be aware of: open-model tests can genuinely overwhelm a struggling system, because they do not back off when it slows. That is the point — it is what real traffic does — but it means the test can drive a system into a failure state a closed loop would have gently avoided. Run these where degradation is safe, and treat "the open-model test melted it" as a finding rather than a test failure.
1model: 100 VUs, send-after-response2during stall: VUs blocked, arrival rate -> ~03measures: only requests that got through4reports: p50 48 ms p99 180 ms5implies: a tight, well-behaved tail6 7Self-limiting: the harder the system struggles,8the less load the test applies.1model: constant arrival 2,000/s, async issue2during stall: requests keep arriving and queueing3measures: latency from INTENDED send time4reports: p50 52 ms p99 2,100 ms5implies: a real stall problem worth fixing6 7Not self-limiting: will drive the system past8its knee, which is what production does too.The closed loop is a negative feedback system that reduces load in response to slowness. Production has no such feedback — users and upstream services keep arriving — so the open model is the one that resembles reality.
Key points
- A closed-loop generator couples send rate to response time, so it stops issuing requests exactly during the stalls that matter.
- The omitted requests are the ones that would have been slowest, so the error concentrates in the upper percentiles.
- The median stays roughly correct while p99 can be wrong by an order of magnitude — and wrong in the flattering direction.
- Fix it with an open model (constant arrival rate) and by measuring latency from intended send time, not actual send time.
- Open-model tests do not back off, so they can drive a struggling system past its knee — that is realistic, and it needs a safe environment.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Server stall → virtual users: a 2s stall blocks all 100 VUs, so no new requests are issued for the duration.
- 2Blocked users → arrival rate: intended 2,000 req/s drops to near zero during the stall, while production arrivals would have continued.
- 3Arrival gap → sample: roughly 3,900 requests that would have queued into the stall are never sent and never measured.
- 4Missing sample → distribution: the omitted requests are precisely the slowest population, so the tail is truncated.
- 5Truncated tail → decision: p99 reads 180 ms instead of 2,100 ms, and the team plans capacity against a number that describes a different system.
- • "The percentiles are clean and consistent, so the test is sound" — internal consistency is exactly what makes this failure invisible.
- • "Production is slower because production has more traffic" — check the achieved arrival rate; the test may have applied far less load than requested during stalls.
- • "p50 matches production, so the test is representative" — the median is the part coordinated omission leaves intact.
- • "The open-model test overwhelmed the system, so it is invalid" — that is the system's real behavior under sustained arrivals; the finding is genuine.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Which load model your harness uses: user-based/closed loop, or rate-based/open model with a scheduled arrival process.
- • Whether reported latency is measured from request issue or from intended send time.
- • Achieved arrival rate against intended arrival rate during the test — a gap means the generator was blocked and omitting.
- • The same workload measured both ways, so the size of the correction on your system is a known number rather than a theory.
- • Switch to an open model with a constant arrival rate so the generator does not back off when the system slows.
- • Measure latency from intended send time; correct historical results the same way before comparing them to production.
- • Report achieved arrival rate alongside latency, so any gap between intended and actual load is visible in the result.
- • Re-baseline capacity numbers derived from closed-loop tests, since the knee was likely measured optimistically too.
- • Run open-model tests in an environment where driving past the knee is safe, and treat the resulting degradation as data.
- • Run the identical workload in both models and compare tail percentiles; the gap is the size of the omission on your system.
- • Confirm achieved arrival rate tracks intended arrival rate throughout the run, including during degradation.
- • Compare corrected test percentiles against production percentiles at matched load; they should now be in the same neighbourhood.
- • Open-model tests can push a system into genuine failure, which requires an environment and a window where that is acceptable.
- • Rate-based executors need enough generator capacity to sustain the schedule, which can mean more load-generating machines.
- • Corrected percentiles look worse, which makes comparisons against historical (uncorrected) results awkward until everything is re-baselined.
- • Standardize on open-model executors in the shared load-testing configuration so the closed loop cannot creep back in.
- • Fail a test run when achieved arrival rate falls materially below intended, which is the automatic detector for this problem.
- • Include the load model and the latency reference point in the test report, so a future reader can tell what was measured.
Accuracy
Performance numbers are conditional. These are the conditions.
- ILLUSTRATIVEThe bucket counts, percentile values and the 180 ms versus 2,100 ms gap are teaching figures chosen to show where the distortion lands. The size of the real correction depends on how often and how long your system stalls.
- ENVIRONMENT-SPECIFICWhether your harness omits depends on the executor: user-based/VU-loop executors are susceptible, rate-based/arrival-rate executors are not. Naming differs across tools; check the executor semantics rather than the tool.