What Is Actually Inside a GPU
A host CPU, a device with many compute units, an unusually large register file, a small block of programmer-managed on-chip memory per unit, and a large pool of high-bandwidth global memory. The register file being large — and being the thing that limits how many lanes stay resident — is the part that surprises people.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
The parts
The host CPU owns the program and the decisions; the device owns the lanes. Work reaches the device as a kernel launch, which the device scheduler distributes across compute units as many groups of lanes. Each compute unit runs several groups concurrently, switching between them whenever one stalls.
Inside a compute unit there are three resources that matter to a programmer. The lanes perform the arithmetic. The register file is unusually large by CPU standards, because every resident lane needs its own registers simultaneously — this is the resource that decides how many groups can be resident. The on-chip scratchpad is a small, fast, explicitly managed memory shared by a group, and unlike a CPU cache it is something you place data into deliberately.
Below all of it sits global memory — a large pool with high bandwidth and high latency, fronted by a cache that is small relative to the number of lanes competing for it. The whole design assumes that global memory latency will be covered by having other work available, which is why the register file is sized to keep many groups resident rather than to make any one group fast.
The memory hierarchy is explicit, and that is the point
On a CPU the hierarchy is automatic: you access memory, and hardware decides what to cache. You influence it only indirectly, through layout and access order — the subject of Spatial Locality and Memory Moves in Lines, Not Variables. On a GPU the fast on-chip memory is *addressable*. You copy a tile into it, the group works out of it, you write results back. The hierarchy is part of the programming model rather than an optimisation the hardware performs on your behalf.
That is a genuine trade. It means a well-written kernel can guarantee reuse instead of hoping for it, which is exactly what makes tiled matrix multiplication so effective — the same idea as Matrix Tiling: Same Arithmetic, Ten Times Faster, but with the tile placement made explicit rather than left to a cache. It also means the responsibility is yours: forget to stage data and every lane reads global memory independently, which is the most common reason a first kernel underperforms.
The relative costs below are the ones worth internalising. Register access is effectively free; scratchpad is close; global memory is far enough away that a kernel touching it on every operation will be limited by it regardless of how much arithmetic capacity the device has.
Occupancy: why registers limit parallelism
The scheduler hides memory latency by switching to another resident group. That only works if there *are* other resident groups, and residency is bounded by the physical resources each group consumes: registers per lane and scratchpad per group. A kernel that uses many registers per lane allows fewer groups to be resident, which leaves the unit with less to switch to when one group stalls.
This produces a counter-intuitive tuning result that has no CPU equivalent. Adding local variables — or unrolling a loop so aggressively that more values must be held live — can *reduce* throughput, because it raises register pressure and drops occupancy. The kernel got better at instruction-level work and worse at latency hiding, and on this device latency hiding was the resource that mattered.
Occupancy is not a goal in itself, and maximising it is not automatically right: a kernel with low occupancy but excellent reuse from the scratchpad can beat a high-occupancy kernel that keeps going to global memory. The useful framing is that occupancy is the budget for hiding latency, and you only need enough of it to cover the latency you actually incur.
| Limiter | Symptom | Lever |
|---|---|---|
| Registers per lane | Occupancy drops as the kernel grows; adding variables makes it slower | Simplify the kernel, cap unrolling, split into two kernels, or cap registers via compiler flags |
| Scratchpad per group | Fewer groups resident than the register budget alone would allow | Use smaller tiles, or trade scratchpad for recomputation |
| Group size | Units are partly idle because groups do not divide evenly | Size groups to the hardware lane-group granularity |
| Not enough total work | Occupancy is fine but units sit idle | Batch more work per launch — see The Transfer You Forgot to Count |
| Nothing — reuse is high | Low occupancy, near-peak arithmetic | Leave it alone; occupancy was never the constraint |
Key points
- A GPU is a hierarchy: device → compute units → lanes, over a large register file, a small explicit scratchpad and a big high-latency global pool.
- The register file is large because many lane groups must be resident at once; residency is how latency gets hidden.
- Fast on-chip memory is programmer-managed, so reuse is something you guarantee rather than hope for.
- Register pressure limits occupancy, which is why adding local variables or unrolling harder can make a kernel slower.
- Occupancy is a budget for hiding latency, not a score — high reuse can beat high occupancy.
Where the Data Is
Change an input and watch which number moves — and which one refuses to.
The exact ratios vary by machine and the absolute times vary far more, which is why none are shown. What is stable enough to build intuition on is the shape: each level is several times the one above, and the gap between the last cache level and memory is the one that decides most program performance.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Host → device: a kernel launch specifies the work and the group shape; the fixed cost is per launch, not per element.
- 2Scheduler → compute units: groups are assigned to units up to the limit set by register and scratchpad consumption.
- 3Compute unit → lanes: one instruction is issued across the lanes of a group; a stalled group is switched out for a ready one.
- 4Lanes → scratchpad: the kernel explicitly stages a tile so the group can reuse it without returning to global memory.
- 5Compute unit → global memory: whatever was not staged goes to the high-latency pool, covered only if other groups are runnable.
- • "Maximise occupancy" — occupancy only needs to be high enough to cover the latency you actually incur; a high-reuse kernel can be fastest at low occupancy.
- • "The scratchpad is just a faster cache" — it is not automatic. Nothing arrives in it unless the kernel puts it there.
- • "More unrolling is better, as it is on a CPU" — on a GPU it can raise register pressure and reduce the residency that hides latency.
- • "Global memory is fast because bandwidth is high" — bandwidth is high and latency is also high; the design covers latency with concurrency, not with speed.
Consequences, controls and cost
- • Kernels that stage data into the scratchpad can achieve reuse that a CPU cache would only give you by luck of layout.
- • Kernels that use too many registers lose throughput even though their instruction stream got better.
- • Small launches waste the device: the fixed launch cost is amortised over too little work.
- • Two kernels with identical arithmetic can differ by an order of magnitude based on where their data lived.
- • Stage reused data into the on-chip scratchpad explicitly instead of relying on the cache to notice the reuse.
- • Watch register pressure when unrolling or adding locals; check achieved occupancy after any such change rather than assuming.
- • Size lane groups to the hardware granularity so units are not left partly idle.
- • Give the device enough work per launch that the fixed launch cost disappears into the total.
- • Read achieved occupancy from the vendor profiler and compare it against the theoretical limit implied by register and scratchpad use.
- • Check the register count the compiler assigned per lane; most toolchains report it and most allow a cap.
- • Compare achieved global memory throughput against the device peak — near peak with low arithmetic means memory-bound.
- • Vary group size and tile size and measure; the interaction is not reliably predictable from first principles.
- • Explicit memory management is more code and more opportunity for error than relying on a cache.
- • Tile sizes tuned for one device are frequently wrong on the next generation, so the tuning is perishable.
- • Optimising for occupancy can conflict with optimising for reuse; the two must be balanced by measurement.
Scope
§224 — what these claims are specific to.
- SIMPLIFIEDA schematic. Real devices add texture and constant paths, matrix units, more cache levels and multi-die interconnects; the box names differ per vendor.
- GPU-SPECIFICDiscrete GPUs. Integrated parts share memory with the CPU, so the global-memory and host-transfer rows collapse together.