GPUgpucputhroughputlatency hidingparallelism

CPU or GPU: Two Bets About What Work Looks Like

A CPU spends its transistor budget making one instruction stream go fast — speculation, out-of-order issue, large caches. A GPU spends a comparable budget on many simple lanes running the same operation over different data. Neither is faster; the shape of your work decides which bet pays.

▶ Run the labFollow the mechanism

Software view, hardware view

The gap between what you wrote and what the machine does is where this whole domain lives.

The question
What is a CPU doing with all the transistors a GPU spends on lanes, and how do I tell which of the two my workload actually wants?
What you wrote
Two devices that both run code, one described as "massively parallel" and therefore assumed to be the faster choice whenever something is slow.
What the hardware does
Two different answers to one question: *what should the machine do while waiting for memory?* The CPU answers with machinery that keeps a single stream moving — branch prediction, out-of-order execution, deep caches. The GPU answers by having thousands of other lanes ready to run, so a stalled lane costs nothing as long as something else is runnable.
Almost every disappointing GPU port comes from porting work that was never shaped for the bet: not enough independent elements, too much data-dependent branching, or so little arithmetic per byte that the device spends its time waiting on memory exactly like the CPU did. Recognising the shape before writing the kernel saves the port.
Source→Compiler→Instructions→Front End→Execution→Registers→Caches→Memory→I/O→Behavior

Where the transistors go

GPU-SPECIFICThe area split is directional, not a measurement; ratios differ by vendor and generation, and integrated GPUs sharing a memory controller with the CPU behave differently again from discrete parts.

Both devices face the same physical problem: arithmetic is cheap and memory is slow, so most of the time a naive machine would be idle waiting for data. They resolve it in opposite directions, and the resolution is visible in what the silicon is spent on.

A CPU core devotes a large fraction of its area to *latency hiding for one thread*: a branch predictor so the pipeline keeps moving through control flow, an out-of-order window so independent instructions run while one waits, and a cache hierarchy big enough that most accesses never reach DRAM. The arithmetic units are a small part of the picture. That is why a CPU is excellent at code full of unpredictable branches, pointer chasing and dependencies — the machinery exists specifically to survive that.

A GPU devotes its area to *many simple lanes plus the registers to keep them all resident*. There is far less speculation and far less per-lane cache. Instead, when one group of lanes stalls on memory, the scheduler switches to another group that is ready — latency is hidden by having something else to do, not by avoiding the stall. This only works if there genuinely is something else to do, which is the whole condition.

The same problem, two designs
Design questionCPUGPU
What is optimisedTime to finish one instruction streamWork completed per unit time across many streams
How memory latency is hiddenSpeculation, out-of-order window, large cachesSwitching to other resident lanes that are ready
Cost of an unpredictable branchA misprediction: pipeline refill — see Misprediction: What a Wrong Guess CostsDivergence: both paths execute in sequence — see Lanes, Divergence and Coalescing
Cache per unit of workLarge; a working set can live entirely in cacheSmall per lane; bandwidth matters more than capacity
Good atBranchy, dependent, irregular, latency-sensitive workWide, regular, independent work with high arithmetic intensity
Bad atThousands of identical independent operationsLong dependency chains, divergent control flow, small inputs

The condition, stated precisely

The GPU bet pays when the work has three properties at once, and fails if any one is missing. First, width: enough independent elements to keep the lanes occupied, which is thousands rather than dozens. Second, regularity: the lanes should mostly follow the same control path, because divergent branches serialise. Third, arithmetic intensity: enough computation per byte moved that the device is not simply waiting on its own memory.

The third is the one that surprises people. A GPU has far more memory bandwidth than a CPU, but it also has far more arithmetic capacity, so the *ratio* is not obviously better. A kernel doing one add per element loaded is bandwidth-bound on a GPU exactly as it was on a CPU — you moved the bottleneck to a different device rather than removing it. That is the same reasoning as When the Memory Bus Is the Bottleneck, applied to a different piece of hardware.

The honest test before porting is arithmetic intensity: operations performed divided by bytes moved. High intensity — dense matrix multiplication, where each loaded element participates in many multiply-adds — is the case GPUs were built for. Low intensity — summing an array, copying with a transform — will be limited by memory on either device, and the port buys you the difference in bandwidth and nothing more.

Two kernels, same element count, completely different verdicts
1// Arithmetic intensity ~ 1 op per 4 bytes loaded.
2// Bandwidth-bound on a CPU. Still bandwidth-bound on a GPU.
3// The port buys you the bandwidth ratio and nothing else.
4for i in 0..n:
5    out[i] = in[i] * 2.0
6 
7// Arithmetic intensity ~ n ops per element loaded (with tiling).
8// Each loaded value is reused many times from on-chip memory.
9// This is the shape GPUs were designed for.
10for i in 0..n:
11  for j in 0..n:
12    acc = 0
13    for k in 0..n:
14      acc += A[i][k] * B[k][j]   // every A and B tile is reused n times
15    C[i][j] = acc

Why "faster" is the wrong word

A claim like "the GPU is 40× faster" is meaningless without three qualifiers: which workload, against which CPU implementation, and counting the transfer or not. Published speedups have historically been inflated by all three — comparing a tuned kernel against an untuned single-threaded scalar baseline, on a workload chosen because it suits the device, with the host-to-device copy excluded from the timer.

The defensible comparison is against a CPU implementation that has itself been given a fair chance: multithreaded, vectorised, cache-blocked. Against that baseline, real speedups on suitable workloads remain large and worth having — but they are the honest number, and they are the one your capacity plan should use.

The comparison also has to include what happens *around* the kernel. A GPU that finishes in a tenth of the time but requires two transfers, a synchronisation and a driver launch per call may lose on wall-clock for small inputs. The Transfer You Forgot to Count works through that arithmetic; the summary is that the break-even is a real input size and you should know roughly where it sits before committing.

The comparison that produces inflated numbers
1// CPU: single-threaded, scalar, no blocking
2// GPU: tuned kernel, transfer excluded from the timer
3t_cpu = time(naive_single_threaded(data))
4t_gpu = time(kernel_only(data_already_on_device))
5speedup = t_cpu / t_gpu        // large, and not a decision-grade number
The comparison a capacity decision can rest on
1// CPU: multithreaded and vectorised — a real baseline
2// GPU: transfer in, kernel, transfer out, synchronise
3t_cpu = time(parallel_vectorised(data))
4t_gpu = time(copy_h2d + kernel + copy_d2h + sync)
5speedup = t_cpu / t_gpu        // smaller, and true end to end

Both numbers are arithmetically correct; only the second answers the question anyone actually has, which is whether the system gets faster. The first compares the best version of one thing against the worst version of another and omits a cost the user pays.

Key points

  • A CPU spends area hiding latency for one stream; a GPU spends it on lanes so a stall can be covered by other work.
  • The GPU bet needs width, regularity and arithmetic intensity at the same time — missing any one of the three sinks it.
  • A low-arithmetic-intensity kernel is bandwidth-bound on a GPU too; the port relocates the bottleneck rather than removing it.
  • Speedup claims are only decision-grade against a fairly tuned CPU baseline with the transfer included.
  • "Faster" is not a property of a device; it is a property of a device and a workload together.

Progressive depth

Overview

A CPU makes one stream of instructions fast. A GPU makes many identical streams finish quickly in aggregate. Choose by the shape of the work, not by which device sounds more powerful.

Practical

Before porting, count operations per byte moved. High intensity and thousands of independent elements means the port is likely worth it; low intensity means you will be memory-bound on the new device too. Always time the transfer, and always compare against a CPU baseline that is itself multithreaded and vectorised.

Advanced

Latency hiding is the real difference. A CPU hides latency within one thread using speculation and a reorder window; a GPU hides it across resident groups of lanes. That is why occupancy matters on a GPU and instruction-level parallelism matters on a CPU — they are the same problem solved with different resources, and it is the reason a GPU has so little per-lane cache: it does not need to avoid the stall, only to have something else ready.

Internals

The lane group is the scheduling unit, and it executes one instruction across all its lanes. Divergent branches are handled by executing both paths with lanes masked off, so control flow costs throughput rather than causing a misprediction. Memory requests from lanes in a group are coalesced into as few wide transactions as the address pattern permits, which is why a strided or scattered access pattern can cost many times what the equivalent contiguous pattern costs — the arithmetic is identical and the traffic is not.

Where the Data Is

Change an input and watch which number moves — and which one refuses to.

Where a value can be, and roughly what each costs relative to a register — 1 unit ≈ one register accessSIMPLIFIED
Register×1
L1 cache×4
L2 cache×14
L3 cache×45
DRAM×200
NVMe storage×100000
Network round trip×10000000
Ratios, not times. Absolute latencies depend on the processor, its clock, the memory it is attached to and what else is running — publishing them would be wrong everywhere except one machine. The bars are log-scaled, so each step is larger than it looks.
Register — Already in the core. Effectively free.
L3 cache — Usually shared between cores, so other work affects your hit rate.
DRAM — Two orders of magnitude past L1. This is the cliff.
NVMe storage — Another three orders of magnitude, and the OS gets involved.
Network round trip — Different universe. Included to keep the earlier rows in perspective.

The exact ratios vary by machine and the absolute times vary far more, which is why none are shown. What is stable enough to build intuition on is the shape: each level is several times the one above, and the gap between the last cache level and memory is the one that decides most program performance.

Follow the mechanism

The path through the machine, hop by hop — and the conclusions it invites that are wrong.

  1. 1
    Host → driver: the program enqueues a kernel launch, which is itself a cost paid per call rather than per element.
  2. 2
    Driver → device scheduler: the launch becomes many groups of lanes, distributed across the compute units.
  3. 3
    Compute unit → lanes: each group executes in lockstep; a group that stalls on memory is switched out for a ready group.
  4. 4
    Lanes → device memory: requests from adjacent lanes are combined into wide transactions when the addresses line up — see Lanes, Divergence and Coalescing.
  5. 5
    Device → host: results are copied back, and a synchronisation makes them visible to the program that asked.
What people conclude from this — wrongly
  • • "It is parallel hardware, so any parallel workload will benefit" — parallel is necessary but not sufficient; regularity and intensity are the other two conditions.
  • • "The kernel is 30× faster, so the program will be" — Amdahl still applies, and the transfer plus the serial remainder often dominate the end-to-end number.
  • • "CPU utilisation is 100%, so the CPU is the bottleneck and offloading will help" — a busy CPU that is mostly stalled on memory will be a GPU that is mostly stalled on memory.
  • • "More lanes means proportionally more throughput" — only until bandwidth, occupancy or divergence binds, and one of them usually binds first.

Consequences, controls and cost

What it causes
  • • Workloads with thousands of independent, similar operations can run dramatically faster than on any CPU.
  • • Workloads with long dependency chains gain nothing, because there is never enough ready work to hide a stall.
  • • Small inputs can be slower on the GPU end to end, dominated by launch and transfer rather than by computation.
  • • Data-dependent branching costs far more than the same branching would on a CPU with a good predictor.
What you can do
  • • Estimate arithmetic intensity before writing a kernel — operations per byte moved decides whether the device can help at all.
  • • Keep data resident on the device across a chain of kernels rather than round-tripping between each.
  • • Restructure the algorithm for regular access and uniform control flow, which usually matters more than tuning the kernel.
  • • Compare against a genuinely tuned CPU version; sometimes the CPU version you had was simply untuned.
  • • Batch small independent problems into one launch so the fixed launch cost is amortised.
How to see it
  • • Compute arithmetic intensity by hand from the algorithm — operations divided by unique bytes touched — before trusting any profile.
  • • Time the full path (copy in, launch, kernel, copy out, synchronise) and compare against a multithreaded vectorised CPU baseline.
  • • Use the vendor profiler to check achieved occupancy and achieved memory throughput; a kernel near peak bandwidth and far from peak arithmetic is memory-bound.
  • • Sweep input size and find the break-even point where the GPU path overtakes the CPU path end to end.
What it costs
  • • A second programming model, a second toolchain and a second set of failure modes to debug and to keep working.
  • • Device memory is a separate, smaller pool; anything that does not fit must be tiled or streamed, which complicates the algorithm.
  • • Portability narrows: kernels tuned for one vendor or generation frequently need retuning for another.
  • • Code that is restructured for regularity is often less readable than the branchy version it replaced.

Scope

§224 — what these claims are specific to.

What these claims are specific to
  • GPU-SPECIFICDiscrete GPUs with their own memory; integrated GPUs sharing the CPU memory controller avoid the transfer cost entirely and change the break-even analysis.
  • PLATFORM-SPECIFICVendor architectures differ in lane width, scheduling granularity and cache structure, and do not share terminology; the neutral terms used here map to different vendor names.

Misconceptions

Claim
“GPUs are just faster processors.”
Reality
They are throughput processors. On a long dependency chain with unpredictable branches, a CPU core will beat a GPU decisively, because the GPU has none of the machinery that makes that case survivable and the CPU has almost nothing else.
Claim
“If the work is parallel, the GPU will help.”
Reality
Parallelism is one of three conditions. Sixteen independent tasks are parallel and far too few; a parallel loop with one operation per loaded byte is parallel and bandwidth-bound. Width, regularity and arithmetic intensity all have to hold.
Claim
“GPU memory bandwidth is so high that memory stops being the bottleneck.”
Reality
Bandwidth and arithmetic capacity both rise, so the ratio does not automatically improve. Many real kernels are memory-bound on a GPU, which is why arithmetic intensity is the number that decides.

Where the rest of this lives

Concurrency & Parallelism
Amdahl's law and the serial remainder

The kernel speedup bounds nothing on its own; what bounds the program is the fraction that stayed serial, which is a concurrency argument rather than a hardware one.

What Is Actually Inside a GPU →