LLM Inference Is a Memory Bandwidth Problem

A request arrives over the network, is tokenized, and becomes a sequence of vectors. Those pass through the model's layers as a series of matrix operations executed on an accelerator, with the weights living in the device's high-bandwidth memory. The output is a distribution over the vocabulary, one token is chosen, appended, and the process repeats for the next token.

The repetition is the crucial structural fact. Generating a hundred tokens is not one pass through the model; it is a hundred passes, each of which needs the weights again. Since the weights do not fit in any on-chip memory, each pass reads them from device memory afresh.

That makes the arithmetic stark. For a single sequence, generating one token performs a modest amount of arithmetic against every weight in the model — which is an extremely low arithmetic intensity, exactly the profile When the Memory Bus Is the Bottleneck describes. The accelerator has enormous arithmetic capacity and spends most of it waiting for weights to arrive.

Processing the prompt — prefill — handles all input tokens simultaneously. That is a wide matrix multiplication with high arithmetic intensity, the shape accelerators are built for, and it is typically compute-bound. It is also why time to first token scales with prompt length in a way that feels like real work, because it is.

Generating each output token — decode — processes a single position. The matrix operations become narrow, arithmetic intensity collapses, and the same full sweep of weights is required. Decode is therefore bandwidth-bound for a single sequence, and per-token latency is governed roughly by how long it takes to stream the model's weights out of memory.

This explains a set of otherwise puzzling observations directly: a long prompt costs a lot up front but subsequent tokens arrive at a steady rate; a model twice the size roughly halves the token rate even on hardware with plenty of spare arithmetic; and running one request at a time wastes most of the accelerator. It also explains why Inside One Model Call: Queue, First Token, Generation separates time-to-first-token from inter-token latency — they are two different hardware regimes, not two parts of one.

If decode is bandwidth-bound because one weight read serves one sequence, the fix follows immediately: make one weight read serve many sequences. Batching several requests so their decode steps proceed together reads the weights once and applies them to every sequence in the batch, so arithmetic intensity rises with batch size and the accelerator moves back towards being compute-bound.

This is why the economics of serving many concurrent users differ so sharply from running a model for one user. The marginal cost of an additional sequence in an existing batch is small, because the dominant cost — streaming the weights — was already paid. It is also why a locally-run model on a single machine gets a small fraction of the tokens per second that a served deployment achieves with the same hardware.

The trade is latency against throughput, the same shape as Throughput Improved, Latency Did Not. Larger batches use the hardware better and increase total tokens per second, while any individual request may wait to be batched and shares the device with others. Continuous batching — adding and removing sequences as they arrive and finish rather than waiting for a fixed batch — is the standard way of getting most of the throughput without most of the latency penalty.