Where an Interpreter's Time Actually Goes

Start with the honest framing, because it sets what is worth doing. A well-built bytecode interpreter is commonly quoted at somewhere between five and fifty times the runtime of equivalent optimized native code, depending overwhelmingly on the language's dynamism and on what fraction of the work happens inside library routines the interpreter merely calls. That is a constant factor. The interpreter executes the same number of logical operations as the compiled version; each one simply costs more.

Which means the constant factor is the only thing on the table, and it also means the constant factor is worth less than people expect. A quadratic algorithm written in a language with a fast interpreter still loses to a linear one written in a slow interpreter, at every input size that matters. Interpreter engineering buys a multiplier; algorithm choice buys an exponent. Spend the multiplier deliberately, and only after measurement has said which multiplier you actually have.

Our own VM makes the arithmetic concrete without any hardware claim attached to it. A hundred-iteration accumulation loop — six lines of AtlasLang — executes 2,819 bytecode instructions to produce one number. Each of those instructions is a full loop iteration in [[dispatch-loop]]: a fetch, an unpredictable indirect branch, a handler body, a back edge. The compiled equivalent of that loop is a handful of machine instructions per iteration and no dispatch at all.

The per-instruction budget has four large entries and they are not equally sized in every VM, which is exactly why this lesson is about diagnosis before technique. [[dispatch-loop]] covers the first of them in detail; the other three are where a dynamically typed language usually loses.

Dispatch is the indirect branch from the loop to the opcode handler, and its cost is dominated by misprediction rather than by the branch itself. Type dispatch is asking, at run time, what the operands are before you can operate on them — a chain of tests in front of every arithmetic operation that a statically typed compiler resolved once, at compile time, and emitted nothing for. Boxing is that every value is a heap object or a tagged word rather than a machine integer, so an addition is an unbox, an add and possibly an allocation. Memory traffic is the operand stack and the slot array living in memory rather than in registers, so values move to and from cache lines on every operation.

The diagnosis is a measurement, not an argument. perf stat gives you branch misses and instructions retired; an allocation profiler gives you the boxing rate; a cycles-per-instruction figure over your bytecode counter gives you the budget itself. In dynamically typed VMs the answer is very often that dispatch is third on this list, and optimizing it first is the classic wasted week.

If the tax is per instruction, the structural fix is to execute fewer instructions for the same work. A superinstruction is a single opcode whose handler performs what two or more adjacent opcodes would have done, chosen because that adjacency is common in real bytecode. LOAD/LOAD/ADD is one of ours; LOAD_FAST/LOAD_FAST was fused into LOAD_FAST_LOAD_FAST in CPython for the same reason.

The saving is not the arithmetic — that was always going to happen. The saving is the two dispatches and the two intermediate stack writes and reads that vanish between them. The transformation is a peephole pass over the instruction stream, the same shape as [[peephole-optimization]] on machine code, and it has the same precondition: the window being rewritten must not be entered from the middle.

The other structural fix attacks the type test rather than the dispatch. If a particular ADD site has seen two machine integers on every execution so far, rewrite that site — in the running bytecode — into an integer-specialized ADD_INT whose handler does one cheap check that both operands are still integers and then adds. The general handler stays available; the specialized one is a bet with a fallback.

That structure should look familiar, because it is the interpreter-sized version of everything in the next module. The observation is [[profiling-and-hotness]]. The bet is [[speculative-optimization]]. The cheap check is what [[guards]] describes. The fallback is a miniature [[deoptimization]]. Caching the resolved answer at the site rather than re-deriving it is [[inline-caches]]. An adaptive interpreter is a JIT that emits bytecode instead of machine code, and understanding it that way is worth more than any individual technique in it.

The gain is real and bounded. Specialization removes the checks; it does not remove the dispatch, the fetch, or the stack traffic between instructions. Getting those requires generating native code for a whole region at once, which is [[jit-compilation]] and the reason the next module exists.

Stack the techniques and you reach a wall. Threading reduces the cost per dispatch. Superinstructions reduce the number of dispatches. Specialization reduces the work inside a handler. Stack caching reduces the memory traffic between handlers. Each is a fraction of a fraction, each enlarges the interpreter, and the interpreter's size pushes back through the instruction cache. Past this point the remaining gap is not a technique you have not applied; it is the loop itself.

Our VM applies none of this, deliberately, and it is worth saying exactly why rather than implying otherwise. It is written in JavaScript, running on a host engine that is itself dispatching, specializing and JIT-compiling our interpreter one level down. Any timing taken from it measures the host, not the technique. What it can honestly show is the *count* — 2,819 instructions for a hundred-iteration loop, visible at /compilers/vm — and counting is where an interpreter performance investigation should start anyway.