Model Memory, and Why the Naive Number Is Always Too Low
Parameters times bytes per parameter gives a floor, not an answer. Activations, the KV cache that grows with context and concurrency, and runtime overhead all sit on top — which is why a model that "fits in memory" by the simple calculation frequently does not.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Where the memory actually goes
The weights are the part everyone counts: parameter count multiplied by bytes per parameter. That number is a floor and it is usually the largest single term, but treating it as the requirement is what produces deployments that fail under load rather than at startup.
The KV cache is the term that surprises people, because it scales with things the model card does not mention: it grows linearly with context length *and* with the number of concurrent sequences. A deployment sized for the weights alone will start fine and run out of memory once real traffic arrives with long contexts, which is a failure that looks like a load problem and is actually a capacity arithmetic problem.
Activations are transient but real, and their peak depends on batch size and the widest layer. The runtime itself holds workspace, fragmentation and its allocator's reserve. In practice the usable fraction of device memory for weights is meaningfully below the physical capacity, and the gap has to be budgeted rather than discovered.
1// 1. Weights: the floor everyone computes.2weights = parameters * bytes_per_parameter3 4// 2. KV cache: grows with BOTH context and concurrency.5// This is the term that turns a working demo into an6// out-of-memory failure once real traffic arrives.7kv_per_token = 2 * layers * kv_heads * head_dim * bytes_per_element8kv_cache = kv_per_token * context_length * concurrent_sequences9 10// 3. Activations: transient, peaks with batch and widest layer.11activations = f(batch_size, hidden_size, layers)12 13// 4. Runtime: workspace, fragmentation, allocator reserve.14overhead = runtime_workspace + fragmentation15 16total = weights + kv_cache + activations + overhead17 18// The common mistake is stopping after line 1 and concluding19// the model fits. It fits at concurrency 1 and context 0.What lowering precision actually does
Quantization stores parameters in fewer bits. In hardware terms this does two things at once, and both matter. It reduces capacity consumption proportionally, so a larger model fits on the same device. And it reduces bytes moved per weight sweep, which — because decode is bandwidth-bound, as LLM Inference Is a Memory Bandwidth Problem establishes — raises the token rate close to proportionally.
That second effect is the one that is easy to miss. Halving the bytes per parameter does not merely let a bigger model fit; on a bandwidth-bound decode phase it roughly halves the time to stream the weights, so tokens arrive faster. This is why quantization is the highest-leverage single change available for inference performance: it moves capacity and throughput with one decision.
The cost is output quality, and its magnitude is genuinely workload-dependent rather than a fixed penalty. Some tasks tolerate aggressive quantization with no measurable degradation; others degrade noticeably at the same setting. There is also a hardware dependency: a device with native support for a given precision executes it at full rate, while one without may emulate it and lose the benefit — so the right choice is a property of the model, the task *and* the hardware together.
| What | Effect on capacity | Effect on decode rate | Risk |
|---|---|---|---|
| Halving bytes per parameter | Roughly halves weight memory | Roughly halves weight streaming time | Quality loss, workload-dependent |
| Quantizing the KV cache | Cuts the term that grows with context and concurrency | Helps once KV traffic is significant | Quality loss concentrated in long contexts |
| Below native precision support | Still saves capacity | May save nothing if the device emulates it | Check hardware support before assuming a gain |
| Quantizing activations too | Smaller transient peak | Modest additional gain | Usually more damaging than weight-only |
| No quantization | Baseline | Baseline | None, but capacity and rate are what they are |
Capacity and rate are the same lever
It is worth stating the unifying point directly, because the two constraints are usually discussed separately. Whether a model *fits* is a bytes question. How fast it *generates* is also a bytes question, because decode streams the weights. The same reduction improves both, and no other single change does that.
This reframes model selection. "Which model fits on this device" and "which model is fast enough" are not independent questions to be answered in sequence — they are the same question asked about the same number. A smaller model at higher precision and a larger model at lower precision can occupy identical memory and stream at identical rates, and then the choice between them is purely about output quality on your task.
It also explains why the KV cache deserves attention out of proportion to its share at low concurrency. At batch size one and a short context it is a rounding error. At production concurrency with long contexts it can rival or exceed the weights, and at that point it is simultaneously eating the capacity that would have allowed more concurrent sequences and the bandwidth that would have generated tokens.
- Capacity and rate are both bytes — the same reduction moves both, which is what makes precision the highest-leverage lever.
- Budget the KV cache at production concurrency and context, not at the demo's batch size of one.
- Check native precision support before assuming a lower precision will be faster rather than merely smaller.
- Evaluate quality on your own task — the degradation from a given setting is not a published constant.
- Equal-memory choices are quality choices — a small model at high precision and a large one at low precision may be interchangeable on hardware.
Key points
- Parameters times bytes per parameter is a floor; activations, KV cache and runtime overhead sit on top of it.
- The KV cache grows with context length and concurrency, which is why capacity failures appear under load rather than at startup.
- Reducing precision cuts capacity and weight-streaming time together, moving both constraints with one change.
- The quality cost of quantization is workload-dependent and must be evaluated on the actual task.
- A precision the hardware does not natively support may save memory while saving no time at all.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Model → device memory: weights are loaded once and stay resident for the life of the process.
- 2Sequence → KV cache: every token of every concurrent sequence adds state that persists for that sequence.
- 3Decode step → memory bandwidth: the weight sweep plus KV reads set the achievable token rate.
- 4Precision reduction → fewer bytes: both the resident footprint and the per-step traffic shrink together.
- 5Capacity exhaustion → rejected work: concurrency is capped by what remains after weights and cache.
- • "The model is 14 GB and the device has 24 GB, so it fits" — it fits at concurrency one with an empty cache, which is not the deployment.
- • "Quantization only saves memory" — while decode is bandwidth-bound it also raises the token rate, which is often the larger benefit.
- • "Lower precision is always faster" — only where the hardware executes it natively; otherwise it saves capacity and no time.
- • "Quality loss from quantization is a known constant" — it varies by model and by task, and has to be measured on yours.
Consequences, controls and cost
- • Deployments sized on weights alone fail under production concurrency and long contexts.
- • Token rate improves roughly in proportion to reduced bytes per parameter while decode is bandwidth-bound.
- • Maximum concurrency is set by leftover memory after weights, not by arithmetic capacity.
- • Two very different model-and-precision combinations can be indistinguishable on hardware and differ only in quality.
- • Budget all four terms at production context length and concurrency before selecting hardware.
- • Quantize weights to the lowest precision that passes evaluation on your own task, checking native hardware support.
- • Quantize or compress the KV cache when long contexts or high concurrency make it a leading term.
- • Re-evaluate quality after any precision change rather than assuming a published result transfers.
- • Instrument device memory occupancy split by weights, KV cache and activations at realistic concurrency.
- • Measure tokens per second before and after a precision change to confirm the bandwidth benefit materialised.
- • Run a task-specific evaluation at each candidate precision rather than relying on published degradation figures.
- • Push concurrency until allocation fails and record the ceiling; that number is the real capacity limit.
- • Lower precision trades output quality for capacity and speed, at a rate that differs by task.
- • Quantizing the KV cache helps most where quality is most sensitive — long contexts.
- • Reserving memory for high concurrency limits the model size you can host, and vice versa.
Scope
§224 — what these claims are specific to.
- SIMPLIFIEDA budgeting sketch. Real KV cache size depends on the attention variant — grouped-query and multi-query attention reduce it substantially — plus layer count, head dimensions and cache precision.
- GPU-SPECIFICWhich precisions execute at full rate is a hardware property; a device without native support for a format may emulate it, saving capacity but not time.