The Transfer You Forgot to Count
Host memory and device memory are separate pools connected by a bus that is slow relative to both. A kernel ten times faster than the CPU loses if you pay two crossings to use it — so the real question is not how fast the kernel is, but at what input size the whole path overtakes staying put.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Three costs, only one of which people time
A GPU call is three costs, not one. There is the launch, a fixed per-call overhead paid regardless of how much work the kernel does. There is the transfer, proportional to bytes moved, in each direction. And there is the kernel itself, the only part most benchmarks measure.
The relative scale below is what makes this decisive. Device memory bandwidth is high; the host bus is much slower — often by an order of magnitude or more. Moving an array across the bus therefore costs substantially more than reading the same array once inside the device, which means a kernel that reads its input a small number of times can spend more time on transfer than on work.
The rule of thumb that follows: the more times each transferred byte is *used*, the better the economics. A kernel that touches each element once is almost never worth a round trip. A kernel that reuses each element many times — or, better, a chain of kernels operating on data that stays resident — can amortise the crossing until it stops mattering.
Computing the break-even before you write the kernel
The break-even is arithmetic, and doing it in advance is much cheaper than discovering it after the port. Total GPU time is the launch overhead, plus bytes divided by bus bandwidth in each direction, plus kernel time. Total CPU time is whatever a fair CPU implementation takes. The break-even input size is where the two curves cross.
For small inputs the GPU path is dominated by the fixed launch cost and loses regardless of how good the kernel is. As input grows, the fixed cost amortises and the transfer term — which grows linearly, like the CPU term — begins to compete on slope. Whether the GPU ever wins depends on whether kernel time grows more slowly than CPU time by enough to overcome the transfer, which is exactly the arithmetic-intensity question from CPU or GPU: Two Bets About What Work Looks Like.
The practical consequence is that you should know roughly where the crossover is and route accordingly: below it, run on the CPU; above it, offload. Systems that offload unconditionally frequently spend more time in transfer than they ever save, and the profile looks confusing because the kernel really is fast.
1// All terms in consistent units. Bandwidths and overheads come2// from a one-off measurement of YOUR machine, not from a datasheet.3 4gpu_total(n) = launch_overhead5 + (bytes_in(n) / bus_bandwidth)6 + kernel_time(n)7 + (bytes_out(n) / bus_bandwidth)8 9cpu_total(n) = tuned_cpu_time(n) // multithreaded, vectorised10 11// Solve gpu_total(n) = cpu_total(n) for n. Below it, stay on the CPU.12 13// The two structural fixes, in order of effect:14// 1. Keep data resident: chain k kernels without returning to the host,15// and the transfer term is paid once rather than k times.16// 2. Raise reuse: if each byte is used r times, the transfer cost per17// useful operation falls by r.How to stop paying it
The strongest fix is structural rather than technical: stop round-tripping. A pipeline of five kernels that copies to the device and back around each one pays ten crossings; the same pipeline keeping data resident pays two. This usually requires restructuring the program so the device owns the data for a phase, which is a bigger change than optimising a kernel and generally worth far more.
After that come the technical mitigations, which are real but smaller. Overlapping transfer with computation lets one chunk compute while the next transfers, hiding much of the copy behind work — this converts a serial cost into a parallel one, but only if there is enough work to hide behind. Batching many small calls into one launch amortises the fixed overhead. And where the platform supports it, pinned or mapped host memory can raise achievable bus bandwidth.
What does not work is optimising the kernel harder. If transfer is 70% of the wall clock, making the kernel infinitely fast improves the total by 30%. That is Amdahl applied to a device boundary, and it is the reason the transfer has to be measured before any kernel tuning is worth starting.
| Fix | What it does | When it does not help |
|---|---|---|
| Keep data resident across kernels | Pays the crossing once per phase instead of once per kernel | When the host genuinely needs the intermediate results |
| Raise reuse per transferred byte | Divides the transfer cost across more useful work | When the algorithm fundamentally touches each element once |
| Overlap transfer with compute | Hides much of the copy behind kernel execution | When there is not enough compute to hide it behind |
| Batch small calls into one launch | Amortises the fixed launch overhead | When calls are genuinely serial and dependent |
| Pinned or mapped host memory | Raises achievable bus bandwidth | When the bottleneck was the fixed overhead, not bandwidth |
| Optimise the kernel further | Reduces only the part that was already smallest | Whenever transfer dominates — which is the case in question |
Key points
- A GPU call costs launch overhead plus transfer in plus kernel plus transfer out; benchmarks usually time only the kernel.
- The host bus is far slower than device memory, so a crossing can cost more than many on-device reads of the same data.
- The break-even input size is computable in advance, and below it the CPU wins no matter how good the kernel is.
- Keeping data resident across a chain of kernels is worth more than any amount of kernel tuning.
- If transfer dominates the wall clock, an infinitely fast kernel still leaves most of the time in place.
Is the Accelerator Worth It?
Change an input and watch which number moves — and which one refuses to.
The problem is big enough to pay for the round trip: the kernel's 10× more than covers the fixed transfer cost.
Where the Data Is
The exact ratios vary by machine and the absolute times vary far more, which is why none are shown. What is stable enough to build intuition on is the shape: each level is several times the one above, and the gap between the last cache level and memory is the one that decides most program performance.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Host → driver: a transfer is enqueued, paying a fixed setup cost before any byte moves.
- 2Host memory → bus: data crosses at bus bandwidth, well below either device or host memory bandwidth.
- 3Bus → device memory: the input lands in the device pool, where lanes can finally reach it.
- 4Kernel → device memory: the compute happens, at a speed the benchmark probably measured in isolation.
- 5Device memory → bus → host: results cross back, paying the same per-byte and fixed costs a second time.
- • "The kernel is 20× faster, so the feature will be" — only if the transfer is small relative to the compute, which is the thing to check first.
- • "The GPU is idle, so we need a bigger GPU" — an idle device usually means it is starved by the bus, and a bigger device would be idle too.
- • "Unified memory removes the copy" — it usually makes the copy implicit rather than absent; the cost reappears as page migration, at less predictable times.
- • "We should optimise the kernel first" — if transfer dominates, kernel work is the smallest available win.
Consequences, controls and cost
- • Small workloads run slower end to end on the GPU despite a genuinely faster kernel.
- • Pipelines that round-trip between kernels spend most of their time on a bus rather than computing.
- • Profiles look contradictory: the kernel is fast, the device is idle, and the program is slow.
- • Scaling behaves oddly — doubling input can improve the GPU's relative standing because fixed costs amortise.
- • Restructure so data stays resident on the device across a phase, rather than crossing per kernel.
- • Compute the break-even input size and route work to CPU or GPU accordingly instead of offloading unconditionally.
- • Overlap transfers with computation so copies hide behind kernels, where there is enough work to hide them.
- • Batch small independent calls into a single launch to amortise fixed overhead.
- • Use pinned or mapped host memory where the platform supports it and bandwidth is the binding term.
- • Time the four phases separately — copy in, kernel, copy out, synchronise — rather than wrapping the whole call.
- • Compare measured transfer time against bytes moved to derive achieved bus bandwidth, then compare that against the link's capability.
- • Sweep input size and plot GPU total against CPU total; the crossing point is the routing threshold.
- • Check the profiler timeline for gaps where the device is idle and the bus is active — that shape is transfer-bound.
- • Keeping data resident complicates memory management and forces the program to reason about device capacity.
- • Overlapping transfers with compute requires chunking and multiple streams, which is more code and harder debugging.
- • Pinned host memory is a limited resource and over-allocating it degrades the whole system, not just this program.
- • Routing by input size means maintaining two implementations and a threshold that needs re-measuring on new hardware.
Scope
§224 — what these claims are specific to.
- PLATFORM-SPECIFICDiscrete GPUs over a host bus. Integrated GPUs share memory and skip the crossing; unified-memory platforms convert it into implicit page migration with different timing characteristics.
- GPU-SPECIFICThe bus-to-device-memory ratio differs substantially by generation and link width; measure your own machine rather than reusing a published figure.
Misconceptions
Apply it
Where the rest of this lives
The transfer is the serial fraction of the offloaded operation, and it bounds the achievable speedup no matter how fast the parallel part becomes.