JITimplementation

Just-in-Time Compilation

Start the program immediately by interpreting it, watch which code actually runs, and compile that code to native instructions while the program is still running — using facts about this execution that no ahead-of-time compiler could have had.

The question

What does a JIT actually do, and at what moment does it do it?

Source→Lexing→Tokens→Parsing→AST→Semantics→Typed→IR→Optimize→Codegen→Machine code→Link→Execute
What the program is here

Two representations of the same function coexist, and a JIT is the machinery that moves between them. One is the bytecode of [[bytecode]] plus accumulated profile data — counters, observed types, branch outcomes — which is what a program *has been doing*. The other is native code specialized to that profile, guarded by explicit checks. The pair exists to answer a question neither form answers alone: which parts of this program are worth the cost of compiling, and what may that compilation assume?

What this phase may assume or do

Compiled code may replace interpreted execution only if it produces the same observable behavior for every execution in which its guards hold, and transfers control back to a lower tier whenever one does not. Concretely, three preconditions: the compiler must insert a check for every fact it took from the profile rather than proved; it must be able to reconstruct the interpreter-level state of [[vm-state-model]] at every point where such a check can fail; and it must be notified and invalidate its code when the runtime changes something the code assumed — a redefined function, a modified class, a newly loaded subclass. Drop any one and the program does not run faster, it runs differently.

Key points

  • A JIT trades compile budget for information: it knows almost nothing when the program starts and everything about how it has run by the time it compiles.
  • It compiles what ran, not what exists, which is what keeps optimizing-compiler cost inside a latency envelope.
  • The advantage over an ahead-of-time compiler is specialization to facts that vary between executions — actual types, actual branch bias, actual call targets.
  • Specialization is only legal because a guard checks each assumption and a fallback path exists when the check fails.
  • Being able to go backwards — reconstructing interpreter state from optimized code — is a standing obligation that constrains what the optimizer may do.
  • The win is a large multiple on hot code and nothing at all on cold code, so the system's real problem is telling the two apart cheaply.
  • "Compiled" and "interpreted" are not properties of a language; a JIT is an implementation strategy that uses both at once.

The asymmetry the whole module is about

An ahead-of-time compiler has unlimited time and no facts. It may run every analysis it likes, and it must produce code correct for every input, every type that could reach a call site, and every branch outcome. A just-in-time compiler has the opposite position: almost no time — its work is on the user's critical path — and every fact it wants, because the program is running in front of it.

A JIT is what you build when you decide the second trade is better for your workload. It is not a faster compiler and it is not a different back end. It is the decision to spend a small compile budget on a small fraction of the program, chosen by observation, and to specialize aggressively because you can check your assumptions instead of proving them.

That choice has a name for a reason: [[execution-strategies]] are implementation choices, not properties of languages. The same language can have an interpreter, a JIT and an ahead-of-time compiler, and several do. What a JIT changes is *when* compilation happens, and therefore *what is known* when it happens.

The pipeline, with compilation moved to run time

implementationThe number of stages and where the hinge sits differ by engine and by version. V8 as of 2024 runs Ignition (bytecode interpreter), Sparkplug (baseline), Maglev and TurboFan; HotSpot runs an interpreter, C1 and C2, with Graal as an alternative top tier; .NET runs a quick JIT, an optimizing JIT and optional ahead-of-time ReadyToRun images that a tier can replace at run time. All of them have added or removed a tier in the last decade. Treat any specific stage list as dated on the day it is written.

The stage rail below is the ahead-of-time pipeline with a hinge in it. Everything up to bytecode looks familiar. What is new is that the last three stages happen while the program is executing, repeatedly, on a subset of the code chosen by measurement — and that one of them runs backwards.

Read the loses column in particular. Each step toward native code discards something, and unlike an ahead-of-time compiler a JIT may need it back: when a guard fails, the runtime has to rebuild interpreter state that the optimized frame no longer explicitly contains. That obligation is why [[deoptimization]] constrains the optimizer rather than merely following it.

One function through a tiered JITtypical
  1. Sourceyou write it
    Text, shipped as-is or as a bundle.
  2. Bytecodeload time
    A linear instruction array plus a constant pool and slot table.
    A countable unit of execution, which is what a profiler needs to exist at all.
    Expression structure; source spans survive only in a side table.
  3. Interpreted runrun time
    The bytecode plus a live profile: invocation counters, loop back-edge counters, observed operand and receiver types per site.
    Facts about *this* execution — which types actually occur, which branches actually run, which call targets are actually reached.
  4. Hotness decisionrun time
    The same bytecode, now labelled worth-compiling.
    A commitment of compile time to one function, made from counters — see [[profiling-and-hotness]].
  5. Optimizing compilerun time
    SSA IR built from the bytecode and specialized to the profile, with a guard at every assumption and a state map at every guard.
    Inlining across sites the profile said were monomorphic, unboxed arithmetic, and hoisted checks.
    Generality. This code is correct only while its guards hold.
  6. Installed native coderun time
    Machine instructions in executable memory, with the function entry patched to reach them.
    No dispatch, values in registers, calls inlined away.
    The materialized interpreter state — frames may not exist, locals may live only in registers, and a stack trace has to be reconstructed.
  7. Deoptimizationrun time
    Back to bytecode plus a rebuilt frame, at the exact bytecode offset the guard was recorded against.
    Correctness when a bet failed, without ever having been wrong in between.
    The compiled code for that path, and the time spent producing it.

Read it asThe rail runs forward seven times and backwards once, and the backwards arrow is what makes the forward ones affordable. Because the system can always fall back, the optimizing compile is allowed to assume things it cannot prove — which is the entire source of its advantage over an ahead-of-time compiler and the entire source of its complexity.

What "just in time" is actually buying

The name suggests the win is scheduling — compiling at the last useful moment. The real win is *information*. At the moment a JIT compiles a function it can read the profile and see that this + has only ever added two small integers, that this call site has only ever reached one implementation, and that this branch has been taken ten thousand times and not-taken zero. An ahead-of-time compiler for the same source must handle all the cases, because it cannot know which will occur — see [[why-runtime-information-helps]].

The second win is scope. A JIT compiles what ran, not what exists. A large application whose hot path is fifty functions pays optimizing-compiler cost on fifty functions instead of on fifty thousand, which is how the compile budget stays inside a user-visible latency envelope at all.

Both wins are paid for, and the bill is itemized in [[jit-costs]]: compilation happens on the critical path, the program is slow until it is warm, compiled code and profiles occupy memory, and the same benchmark run twice does not produce the same numbers. There is no version of this that is free.

The same function, compiled at three different timestypical
CompiledKnowsBudgetWhat it cannot do
Ahead of timeThe source, the target, and whatever the whole-program view reachesMinutes — a build, not a requestSpecialize to types, branch bias or call targets that vary between runs
At load or install timeThe above, plus the exact machine and the set of modules actually presentSeconds — an install step, once per deploymentKnow anything about which inputs the program will receive
Just in timeAll of the above, plus the observed types, counts and targets of this executionMilliseconds, on the user's critical pathAfford an expensive analysis, or optimize code that has not run yet

The loop it replaces

simplifiedOur AtlasLang VM has no JIT and the right-hand listing is a sketch of the shape, not output from any compiler — register names, guard encoding and the loop form differ on every real target, and a real optimizing tier would also apply bounds-check elimination, strength reduction and unrolling here. What is faithful is the structure: hoisted checks, unboxed registers, and a deoptimization edge out of each guard.

It is worth being concrete about what disappears. [[dispatch-loop]] costs a fetch, an indirect branch, a handler, and a back edge for every bytecode instruction; [[interpreter-performance]] adds a type test and a box to that per operation. Compiled native code for the same function has none of them: the operations are machine instructions in sequence, the values are in registers, the types were established once by a guard at the top rather than tested at every use.

That is why the gain is a large multiple rather than a percentage, and also why the gain is concentrated. Code that runs once gains nothing and pays the compile cost; code that runs ten million times gains almost everything. The whole system is an attempt to tell those two apart cheaply, at run time, without being wrong often enough to matter.

A summing loop as bytecode, and the shape optimized native code takes
Bytecode — per iteration, executed by the interpreter
LOAD 1 ; total
LOAD 0 ; n
ADD            ; generic: test both operands, unbox, add, rebox
STORE 1 ; total
LOAD 0 ; n
PUSH 1
ADD            ; generic again
STORE 0 ; n
LOAD 0 ; n
PUSH 100
LT             ; generic compare
JUMPF exit
After an optimizing tier, given a profile that saw only small integers
▸; entry guard, once:
▸guard total is a machine integer, else deoptimize
▸guard n is a machine integer, else deoptimize
▸; loop body, per iteration:
▸add   rTotal, rN
▸add   rN, 1
▸cmp   rN, 100
▸jl    body

Read it asTwelve interpreted instructions with a type test inside three of them become four machine instructions and two checks that execute once. The two guards are not overhead to be removed later — they are the reason the four instructions are allowed to assume machine integers at all. Note also what the guards must carry with them: enough information to rebuild the interpreter's slots and instruction pointer, or the fallback would have nowhere to land.

How it works

The steps, in the order the compiler takes them.

  • Compile source to bytecode at load time, or load bytecode directly, and begin executing it in an interpreter so the program starts immediately.
  • Instrument execution with counters and type feedback: how many times each function was entered, how many times each loop back edge was taken, and which operand and receiver types each site observed.
  • When a counter crosses a threshold, queue the function for compilation — usually on a background thread, so the running program is not stopped for it.
  • Build IR from the bytecode, then annotate it with the profile: replace generic operations with specialized ones and record, for each specialization, the assumption it depends on.
  • Insert a guard for every assumption and attach to each guard a map describing how to rebuild the interpreter state at the corresponding bytecode offset.
  • Optimize the specialized IR — inlining, escape analysis, register allocation — under the constraint that any value a state map still needs must remain recoverable.
  • Emit machine code into executable memory, then patch the function entry so subsequent calls reach it; for a function already running, transfer at a loop header instead — [[on-stack-replacement]].
  • On a guard failure, rebuild the interpreter frame from the state map, discard or mark the compiled code, and resume in a lower tier at the recorded bytecode offset.
  • Invalidate compiled code when the runtime changes an assumption globally — a function redefined, a class loaded, a property made non-constant — rather than waiting for a guard to notice.

How it breaks

What the engineer observes when it goes wrong — not what goes wrong internally.

  • A function is compiled, deoptimizes, is recompiled from a profile that still contains the old assumption, and deoptimizes again. The program spends its time compiling and never gets faster, and the CPU profile shows the compiler rather than the application.
  • The compile queue backs up behind a large function and a latency-sensitive request runs interpreted for far longer than the threshold suggested, producing a tail-latency spike that no single request explains.
  • A guard is missing for an assumption the compiler took from the profile, and the optimized code produces a wrong value with no error — the hardest class of JIT bug, and the reason engines run the interpreted and compiled results against each other in testing.
  • A benchmark reports a large speedup that vanishes in production, because the benchmark ran one shape of input long enough to specialize and the production traffic is polymorphic.
  • Memory grows steadily in a long-running process as compiled code and profile data accumulate for functions that were hot once, and never gets released because nothing evicts code.
  • A stack trace from optimized code names the wrong function or omits frames entirely, because the inlined callees never had frames and the reconstruction metadata was incomplete.

When it helps

  • Long-running processes — servers, browsers, data pipelines — where warmup is amortized over hours and the steady-state speed is what matters.
  • Dynamically typed languages, where the gap between "what could be here" and "what is here" is largest and specialization therefore buys most.
  • Programs whose hot path is a small fraction of a very large codebase, where compiling everything ahead of time would be an enormous cost for almost no return.
  • Workloads whose shape is not knowable at build time — a query engine compiling a plan, a regex engine compiling a pattern, a template engine compiling a template.
  • Polymorphic code where one call site turns out to be monomorphic in practice, which is where [[inline-caches]] and speculative inlining collect most of their win.

When it hurts

  • Short-lived processes. A command-line tool that runs for eighty milliseconds pays the whole compile bill and collects none of the return.
  • Memory-constrained environments, where code caches and profile tables are a real fraction of the budget.
  • Hard real-time and latency-critical paths, where a compilation or a deoptimization landing inside a request is a worse outcome than uniformly slower code.
  • Environments that forbid writable-then-executable memory: locked-down mobile platforms, some embedded and console targets, and some hardened server configurations — see [[jit-costs]].
  • Benchmarking and performance regression work, where the non-determinism a JIT introduces makes small differences genuinely hard to measure.

What it costs

Every one of these is paid by something.

  • Compiling at run time buys profile-specialized code and pays with compile time on the critical path, plus a warmup period during which the program is measurably slower than its own steady state.
  • Compiling only hot code buys a small compile budget and pays with a cold-code cliff: the first requests after a deployment or a cache flush are served by the interpreter.
  • Speculating on profile data buys unboxed arithmetic, inlining and direct calls, and pays a guard on every assumption plus the memory and the optimizer constraints of the state maps that make failure survivable.
  • Keeping deoptimization possible buys correctness under wrong bets and pays by forbidding transformations that would destroy values the state maps still describe — an optimizer inside a JIT is strictly less free than one inside an ahead-of-time compiler.
  • Holding compiled code and profiles buys steady-state speed and pays in resident memory that grows with how much of the program has ever been hot.

What else you could do

What a different compiler or language does instead, and when that is better.

  • Interpret and never compile. Predictable, small, portable, and an order of magnitude slower on hot code — [[interpreter-performance]].
  • Compile everything ahead of time and ship native code. Fastest start, no warmup, no runtime compile surface, and no ability to specialize to this execution — [[aot-compilation]].
  • Compile ahead of time *from* a profile collected earlier, which recovers much of the specialization without any runtime compiler — [[profile-guided-optimization]], at the cost of the profile being from a different run than the one being served.
  • Compile at install time on the target device, so the machine is known but the workload is not, which is what Android's ahead-of-time compilation of DEX and .NET's ReadyToRun images do.
  • A hybrid: ship ahead-of-time code as a baseline and let a JIT replace it for genuinely hot methods. This is now the mainstream answer in .NET and Android, and it makes startup and steady state separately tunable.

See it for yourself

The flag, dump or tool that shows you this directly.

  • V8: node --print-opt-code, --trace-opt and --trace-deopt show what was compiled, when, and why it was thrown away; --allow-natives-syntax plus %GetOptimizationStatus(f) reports a function's current tier.
  • HotSpot: -XX:+PrintCompilation prints one line per compilation with the tier; -XX:+UnlockDiagnosticVMOptions -XX:+PrintInlining shows what was inlined and what was refused and why.
  • JITWatch reads HotSpot's -XX:+LogCompilation output and shows bytecode, inlining decisions and generated assembly side by side — the closest thing to a pipeline explorer for a production JIT.
  • .NET: the DOTNET_TieredCompilation and DOTNET_TieredPGO environment variables switch the behaviour off and on, which is the cleanest way to see what tiering is actually contributing.
  • PyPy: PYPYLOG=jit-log-opt:out writes the traces the tracing JIT produced, which is a genuinely different design from the method JITs above and worth reading once for contrast.
  • Our own tier tracker at /compilers/jit steps through the counter, the threshold, the compile and the deoptimization on a small program, with no engine-specific behaviour claimed.

Plausible wrong readings

Stated the way a confident engineer states them.

  • "A JIT is a compiler that runs later." It is a compiler that runs *with different information and a different budget*, and both halves of that change what it should do. A JIT that behaved like an ahead-of-time compiler would be strictly worse than one.
  • "JIT compilation means the program is compiled every time it runs." Only the hot parts, only after they prove hot, and typically on a background thread. Most of the code in a large program is never compiled at all.
  • "Once it is compiled, it stays compiled." Compiled code is discarded when a guard fails, when the runtime invalidates an assumption, and sometimes when a code cache fills. Falling back to the interpreter is a normal event, not an error.
  • "A JIT makes a dynamic language as fast as a static one." It closes a large part of a large gap, on hot code, when the code is type-stable. Polymorphic, allocation-heavy or megamorphic code collects much less of that.
  • "The interpreter is just there for startup." The interpreter is also the fallback that makes speculation safe, and the definition of correct behaviour that the compiled code is checked against. Removing it removes the ability to be wrong safely.

Misconceptions

The claim, and what is actually true.

A JIT compiles the program the first time each function is called.
That describes one simple design and not the mainstream ones. Real systems interpret first, count, and compile only what crosses a threshold — precisely so that the compile cost is spent where it can be repaid.
Because a JIT compiles at run time, the generated code is worse than an ahead-of-time compiler's.
It is generated under a much tighter time budget and with much better information. On type-stable hot code the information usually wins, which is why engines that added an optimizing tier saw large gains over their own baselines.
JIT and interpretation are alternatives.
Every mainstream JIT contains an interpreter and depends on it — for startup, as the deoptimization target, and as the reference semantics. They are layers of one system, which is what [[tiered-compilation]] describes.

Go deeper

The same idea at increasing depth. Stop wherever it stops being useful.

overview

The program starts by being interpreted, which is slow but instant. While it runs, the system counts which functions and loops execute most, and hands the busiest ones to a compiler that turns them into real machine instructions. Because that compiler can see what the program has actually been doing, it can produce better code than one that ran before the program existed — provided it also inserts checks, so that if the program starts behaving differently the system can go back to interpreting.

practical

What this means for code you write is mostly about stability, not cleverness. Keep the types at a call site consistent, keep object shapes consistent, and hot code will specialize and stay specialized; make a site see many shapes and it goes megamorphic and stops being inlined. Measure only after warmup, and measure warmup separately, because pre-warmup and post-warmup numbers describe two different programs. And when a performance change is mysterious, --trace-deopt or -XX:+PrintCompilation will usually explain it in one line: something deoptimized and never recovered.

advanced

The structural insight is that a JIT is a compiler operating under an obligation no ahead-of-time compiler has: at every point where a speculation might fail, it must be able to materialize the abstract machine state that [[vm-state-model]] describes. That obligation flows backwards through the optimizer. A transformation that would destroy a value some state map still needs is illegal, even when it preserves observable behavior in the ordinary sense — the state map is, in effect, an additional observer. This is why deoptimization metadata is a compiler concern rather than a runtime one, why "how much can I speculate" is really "how much state can I afford to describe", and why engines invest so heavily in making state maps compact. The performance question and the correctness question turn out to be the same question, approached from opposite ends.

How much this depends on

Nothing in this domain is true of every compiler. These say how much.

implementationTier counts, tier names, thresholds and compile-queue policies differ between V8, HotSpot, .NET, JavaScriptCore and PyPy, and change between versions of each — Maglev was added to V8 in 2023, Sparkplug in 2021, and HotSpot's tiered compilation only became the default in Java 8. Anything stated about "how the JIT works" needs an engine and a version attached, and will age.
typicalThe description here is of a method-based tiered JIT, which is what V8, HotSpot and .NET use. Tracing JITs — PyPy, LuaJIT, and the older TraceMonkey — compile hot *loop traces* across function boundaries instead of methods, which changes what the unit of compilation is, what a guard exit means and how deoptimization works. The vocabulary transfers; the mechanics do not.
simplifiedAtlasLang has no JIT. Every listing in this lesson describing native code is a shape, not compiler output, and the interactive tier tracker models counters, thresholds and transitions without generating any machine code. Where a real number is quoted anywhere in this module it comes from our bytecode VM, which does exist.

If you were asked this in an interview

  • What does a JIT know that an ahead-of-time compiler cannot, and what does it give up to know it?
  • Walk me through the life of one function from process start to optimized native code, and then back again.
  • Why does every mainstream JIT still contain an interpreter?
  • You deploy a change and the first thirty seconds of traffic is much slower than before. What are the candidate explanations and how would you tell them apart?

Connections

Computer Architecturebranch-predictioninstruction-cacheregistersspeculative-execution
Performancejit-and-warmupcpu-profilingmeasure-before-optimizingbenchmark-fallacies
OS & Networkingmemory-mapping
Domains that do not exist yet
  • Programming Languages & Runtime Internals — The runtime half of a JIT: code caches, on-stack transfer, invalidation on class loading, and the object model the profile describes
    Everything in this lesson is compiler-side — what is emitted, what is assumed, what metadata is recorded. The machinery that stores compiled code, patches entry points, notices that a class was loaded and tells the compiler to throw work away lives in the runtime, and the two halves are designed against each other.