GPUs & Efficiency

Parallel compute, matrix operations, memory bandwidth and VRAM; quantization from FP32 to INT8; pruning and distillation; and what inference actually costs.

GPU Fundamentals

A GPU is thousands of simple cores doing the same matrix arithmetic in lockstep. It is fast only when there is enough parallel work to fill it, which is why a single small request leaves it mostly idle.

Q · We moved inference to a GPU and per-request latency barely changed while cost went up. What is the GPU actually doing, and when does it help?
Memory Bandwidth & VRAM

What fits on the device is parameters times bytes per parameter, plus activations, plus — for training — optimizer state. What runs fast is bounded by how quickly those bytes can be read, and for large models every token reads all the weights.

Q · Will this model fit on the device, and once it fits, is inference bounded by arithmetic or by reading the weights?
Quantization

Storing weights in fewer bits — FP32 to FP16, BF16, INT8 — shrinks memory and speeds up memory-bound inference. The quality cost is real, concentrated on rare inputs, and only visible if you evaluate on the same slices you used before.

Q · The quantized model is half the size and passes the aggregate evaluation. Where would a quality loss hide, and how do I know it did not?
Model Compression

Quantization, pruning and distillation are three different bargains: fewer bits per weight, fewer weights, or a smaller model taught by the larger one. They trade quality, latency, cost and engineering effort differently, and can be combined.

Q · The model is too slow or too big for where it has to run. Which way of making it smaller fits this constraint, and what does each cost in quality and effort?
Pruning & Distillation

Pruning removes weights, and only speeds things up when it removes them in shapes the hardware can skip. Distillation trains a small model on a large model's outputs, and inherits everything the large model believed.

Q · The pruned model is ninety percent sparse and no faster; the distilled model is fast and wrong where the teacher was uncertain. What did each technique actually do?
Inference Cost
▶ lab

Cost per prediction is hardware cost per hour divided by predictions per hour, plus feature fetch and storage. Utilisation is the lever, and the first question is whether the prediction needs this model at all.

Q · What does one prediction actually cost, which term dominates, and which of the cheaper options — simpler model, batching, caching, quantization, fewer retrains — would move it?