GPUs & Accelerators
Throughput hardware and its price: why GPUs win on wide regular work, why transfers and divergence undo it, and what specialized accelerators trade away to be fast at one thing.
A CPU spends its transistor budget making one instruction stream go fast — speculation, out-of-order issue, large caches. A GPU spends a comparable budget on many simple lanes running the same operation over different data. Neither is faster; the shape of your work decides which bet pays.
A host CPU, a device with many compute units, an unusually large register file, a small block of programmer-managed on-chip memory per unit, and a large pool of high-bandwidth global memory. The register file being large — and being the thing that limits how many lanes stay resident — is the part that surprises people.
Lanes execute in lockstep groups: one instruction, many lanes, different data. Two consequences follow and both are unlike anything on a CPU — a data-dependent branch makes the group execute both sides in sequence, and the memory addresses the lanes request must line up or the traffic multiplies.
Host memory and device memory are separate pools connected by a bus that is slow relative to both. A kernel ten times faster than the CPU loses if you pay two crossings to use it — so the real question is not how fast the kernel is, but at what input size the whole path overtakes staying put.
Dense linear algebra, graphics, model training and inference, image processing, scientific simulation. The list looks unrelated until you notice that every entry is wide, regular and reuses each loaded byte many times — and that the workloads which disappoint share the opposite properties.
GPUs, AI accelerators, NPUs and FPGAs are not four unrelated products. They are points on one axis running from fully general to fully fixed, and each step along it trades away the ability to run arbitrary work in exchange for doing one kind of work with less silicon and less power.
Specialized hardware is more efficient because it does less. That is a genuine gain and a genuine risk: the efficiency comes from decisions frozen at design time, and workloads have a habit of changing shape faster than silicon can be replaced.
The path from prompt to token runs through matrix operations on an accelerator, and the surprise is which resource binds. Generating tokens one at a time reads the entire model from memory per token, so inference is usually bandwidth-bound rather than compute-bound — which is why model size and memory bandwidth dominate the conversation.
Parameters times bytes per parameter gives a floor, not an answer. Activations, the KV cache that grows with context and concurrency, and runtime overhead all sit on top — which is why a model that "fits in memory" by the simple calculation frequently does not.