JIT and Warm-Up: The First Thousand Requests Are a Different Program
A JIT-compiled runtime starts interpreted and speeds up as it observes what the code actually does. That makes early requests slower, benchmarks without warm-up meaningless, and freshly-scaled instances a source of tail latency nobody attributes correctly.
Frame the diagnosis
Performance work starts from a symptom and a signal — never from a resource dashboard.
Why the same code gets faster over time
A JIT-compiled runtime does not know what your code will do until it runs. It starts by interpreting or quickly compiling with little optimization, counts how often methods are called and which branches are taken, and recompiles hot code with progressively more aggressive optimization based on what it observed. Steady-state performance is therefore reached after some amount of representative work, not at process start.
The optimizations depend on observed behaviour, which means they can be invalidated. A call site that has only ever seen one implementing type gets specialized; the first time it sees a second type, that specialization is discarded and the method is deoptimized back to a slower path before eventually being recompiled. This is why a workload change can make a long-running process temporarily slower — nothing was redeployed, the runtime simply learned that its assumption was wrong.
The practical consequence is a rule that applies to every measurement in this domain: performance numbers must state how much warm-up preceded them. Without that, a benchmark result is uninterpretable, and the disagreement between "our benchmark says 4ms" and "production says 11ms" cannot be resolved because the two are measuring different programs.
Benchmarks that measure the wrong program
A benchmark that runs an operation a thousand times and reports the mean is, on a JIT runtime, reporting a blend of interpreted and compiled execution weighted by how quickly the runtime happened to promote that method. Change the iteration count and the number changes, which is a reliable sign the measurement is not measuring what it claims to.
The discipline is well established: run a warm-up phase whose results are discarded, then measure the steady state across enough iterations to be statistically meaningful, and report variance rather than a single number. Purpose-built harnesses exist precisely because doing this correctly by hand is harder than it looks — dead-code elimination will happily delete the operation you are timing if its result is unused, and constant folding will evaluate it at compile time if the input is a literal.
This is one instance of the broader problem in Benchmark Fallacies: Confident Numbers That Are Wrong and Microbenchmark or End-to-End: Why p99 Did Not Move: a microbenchmark measures a method in isolation, warm, with a hot cache and no competing load, and production runs it cold, contended, with a cold cache and a shared allocator. Both numbers can be correct and neither predicts the other.
1start = now()2for i in 1..1000:3 result = parse(payload) # may be optimized away if unused4print((now() - start) / 1000) # "average" of three different programs5 6# Change 1000 to 100 and the number gets worse.7# Change it to 100_000 and it gets better.8# A measurement whose result depends on how long you measure9# is not measuring the thing you named.1# 1. Warm up until the runtime has stopped improving2for i in 1..20_000:3 sink(parse(payload)) # sink() prevents dead-code elimination4 5# 2. Measure the steady state, in batches, keeping the distribution6samples = []7for batch in 1..30:8 t0 = now()9 for i in 1..1_000: sink(parse(payload))10 samples.append((now() - t0) / 1_000)11 12report(median(samples), p95(samples), stddev(samples))13# Report the spread. A single number hides whether the result is stable.The first version produces a number that changes with iteration count, which makes it unfalsifiable. The second separates warm-up from measurement, defeats dead-code elimination, and reports variance — so a later comparison can distinguish a real regression from noise.
What warm-up means for autoscaling and deploys
A freshly-started instance is slower than a warm one, so any event that creates instances — a deploy, a scale-out, an instance replacement — injects slow capacity into the fleet exactly when it is needed. Scaling out during a traffic spike is the worst case: new instances arrive cold, serve slowly, and take longer to clear their share of the load, which is one of the mechanisms behind Autoscaling Lag: The Gap Where the Outage Lives.
The mitigations are ordinary and worth naming. Send a synthetic warm-up load to an instance before adding it to the load balancer. Ramp real traffic gradually rather than switching a fraction to it at once. Scale earlier so warm-up completes before capacity is genuinely needed. Where the runtime supports it, ahead-of-time compilation or cached compilation profiles reduce the phase substantially — at the cost of build complexity and, sometimes, lower steady-state peak performance.
For diagnosis, the essential move is to break latency down by instance age. Fleet-wide p99 during a scale-out mixes cold and warm instances and produces a mystery; the same data grouped by instance age produces an obvious decaying curve and an equally obvious explanation. This is the same lesson as Percentiles: Which One, and How Many Users Is That? and per-partition lag: the aggregate hides the population that is actually suffering.
| Approach | Effect | Cost | Best when |
|---|---|---|---|
| Synthetic warm-up before serving | Instance reaches steady state before real traffic | Startup takes longer; the warm-up load must be representative | Deploys and scale-outs are frequent |
| Gradual traffic ramp | Cold instances take a small share while warming | Load balancer must support weighting; slower to reach full capacity | Any fleet with a capable load balancer |
| Scale earlier / more headroom | Warm-up finishes before the capacity is needed | Money — you run capacity you are not yet using | Traffic is predictable enough to anticipate |
| AOT or cached profiles | Shortens or removes the warm-up phase | Build complexity; sometimes lower steady-state peak | Short-lived processes, serverless, fast-scaling fleets |
| Ignore it | Nothing | Recurring unexplained deploy-time tail latency | Instances are long-lived and deploys are rare |
Key points
- JIT runtimes start slow and speed up as they observe the workload, so steady-state performance arrives after representative work, not at start.
- Optimizations built on observed behaviour can be invalidated, so a workload change can slow a long-running process with no deploy involved.
- A benchmark whose result changes with iteration count is measuring warm-up, not the operation it names.
- Every performance number should state its warm-up; without that, benchmark and production numbers cannot be reconciled.
- Latency broken down by instance age turns a scale-out mystery into an obvious decay curve.
Follow the diagnosis
The causal chain, hop by hop — and the readings that invite the wrong conclusion.
- 1Autoscaler → fleet: a traffic spike triggers scale-out, adding several cold instances to the load balancer at once.
- 2Load balancer → cold instances: each receives a full share of traffic immediately, while still executing interpreted or lightly-optimized code.
- 3Cold instances → latency: their p50 is several times steady state, and requests routed to them land in the fleet-wide tail.
- 4Cold instances → capacity: serving more slowly, they clear less load than expected, so the autoscaler adds still more cold capacity.
- 5Fleet-wide p99 → responders: the aggregate shows a spike with no failing dependency, and the cause is invisible until latency is grouped by instance age.
- • "Latency spiked after the deploy, so the new code is slower" — check whether it decays over minutes; warm-up decays, a genuine regression does not.
- • "The benchmark says 4ms, production says 11ms, so production is misconfigured" — the benchmark measured a warm, uncontended, cache-hot process.
- • "We scaled out, so capacity increased" — cold instances contribute less than warm ones for their first minutes.
- • "A long-running process cannot suddenly get slower without a deploy" — deoptimization triggered by a workload change does exactly that.
- • "Fleet p99 is the number to watch" — during scale events it mixes cold and warm populations and describes neither.
Measure, fix, validate
An optimization is not finished until the metric that motivated it has moved.
- • Latency grouped by instance age (time since process start) — the single measurement that makes warm-up visible.
- • Time-to-steady-state per service: how many requests or seconds until p50 flattens, which sets the warm-up budget.
- • JIT compilation activity and deoptimization events where the runtime exposes them, which explain mid-life slowdowns.
- • Benchmark results with warm-up iterations and variance reported alongside, never as a bare mean.
- • Fleet composition during scale events: what fraction of instances are below the steady-state age threshold.
- • Warm instances with representative synthetic traffic before adding them to the load balancer, which removes the cold phase from user-visible latency.
- • Ramp real traffic to new instances gradually where the load balancer supports weighting.
- • Scale earlier, using leading indicators, so warm-up completes before the capacity is actually required ([[autoscaling-lag]]).
- • Use ahead-of-time compilation or cached compilation profiles for short-lived or fast-scaling workloads, accepting the build-complexity cost.
- • Standardize benchmark methodology with an explicit warm-up phase and variance reporting, so results are comparable across time.
- • The latency-by-instance-age curve flattens: new instances should reach steady state before or shortly after taking traffic.
- • Fleet p99 during a scale-out stops spiking, which is the user-visible proof that cold capacity is no longer serving.
- • Benchmark results become stable across iteration counts, confirming the measurement now describes steady state.
- • Deploy-time latency spikes disappear from the deploy-annotated latency graph across several consecutive releases ([[deployment-markers]]).
- • Synthetic warm-up lengthens startup, which slows deploys and delays the arrival of emergency capacity when it is most needed.
- • Gradual traffic ramping means the fleet reaches full capacity later, a real cost during a genuine spike.
- • Scaling earlier costs money continuously for capacity that is idle most of the time ([[headroom]]).
- • AOT compilation shortens or removes warm-up and can lower steady-state peak performance, because it cannot use runtime profile information.
- • Latency-by-instance-age retained as a standard dashboard panel, so warm-up regressions are visible rather than rediscovered.
- • A warm-up gate in the deployment pipeline: instances do not receive traffic until a readiness check reflecting steady state passes.
- • Benchmark methodology enforced in CI (warm-up iterations, variance thresholds), so results stay comparable release to release.
- • An alert comparing p99 of young instances against mature ones, which catches a regression in warm-up time itself.
Accuracy
Performance numbers are conditional. These are the conditions.
- RUNTIME-SPECIFICWarm-up behaviour differs substantially between HotSpot, V8, the CLR and PyPy, and depends on compilation tier configuration. Ahead-of-time compiled languages such as C++, Rust and Go do not have this phase at all.
- ILLUSTRATIVEThe 48ms → 11ms decay curve shows the shape of warm-up, not a measurement. Actual warm-up duration and magnitude depend on runtime, code size, workload and configuration, and must be measured per service.