PCIe: Lanes, Generations and the Transfer Budget
The link between a CPU and an accelerator is not free capacity. It has a width, a generation and a ceiling — and for many workloads the transfer over it, not the computation at either end, is what sets the runtime.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
Lanes times generation, minus overhead
A PCIe link is built from lanes — independent serial connections used in parallel. A device negotiates a width (x1, x4, x8, x16) and a generation, and the usable bandwidth is roughly the product, less encoding and protocol overhead. Both halves matter and both are properties of the *link*, not of the device: a card capable of x16 in an x4 slot runs at x4, silently, with no error and no warning.
Each generation has approximately doubled the per-lane rate over its predecessor, so a x8 link of one generation is roughly a x16 link of the previous one. That doubling is what makes the specific numbers age badly and the *structure* durable, which is why this lesson gives the relationship rather than a table of gigabytes per second that will be wrong within a few years.
The number to actually care about is achievable bandwidth after overhead, in the direction you need, on the branch you are on — not the headline figure for the generation. Real transfers also rarely reach the theoretical ceiling: small transfers pay per-transaction overhead, and the two directions may not be symmetric in practice.
| Factor | Effect | Common surprise |
|---|---|---|
| Negotiated width (x1–x16) | Linear in lane count | A x16 card in a x4 slot runs at x4 with no error |
| Generation | Roughly doubles per-lane rate each step | A newer card in an older slot negotiates down |
| Protocol overhead | Usable is meaningfully below raw | Headline numbers are raw, not achievable |
| Transfer size | Small transfers pay fixed per-transaction cost | Many small copies reach a fraction of the ceiling |
| Branch sharing | Siblings behind a switch share the upstream | A busy NIC reduces accelerator bandwidth |
| Pinned versus pageable host memory | Pageable requires an extra staging copy | The copy nobody wrote is often the slowest part |
The transfer budget
The useful way to think about an accelerator workload is a budget. The device can perform some amount of computation per unit of data delivered. If the computation per byte is high — a large matrix multiply, where each element is used many times — the link is irrelevant and the device is the limit. If it is low — an elementwise operation over a large array, where each byte is touched once — the link is the limit and the device idles waiting.
The ratio between them is arithmetic intensity, and it decides which side of the machine you should be optimising. A kernel that is twice as fast on a transfer-bound workload produces no measurable improvement whatsoever, which is a demoralising thing to discover after the fact and trivial to predict beforehand.
The scale below expresses the ordering that matters. Moving data from host memory to device memory is dramatically more expensive per byte than the device reading its own memory — which is why the first rule of accelerator programming is to move data once and keep it there, and the second is to overlap transfer with computation so the link and the device are both busy.
When the link is the program
The diagnostic signature of a transfer-bound workload is straightforward once you know to look for it: the accelerator's utilisation is low, its own memory bandwidth is far from saturated, and total runtime tracks data volume rather than problem complexity. Doubling the compute changes nothing; halving the data halves the runtime.
The fixes follow directly. Move less — keep intermediate results on the device instead of round-tripping them, and send lower-precision data where accuracy permits, which is one of the underrated benefits of quantization discussed in Model Memory, and Why the Naive Number Is Always Too Low. Move it once — hoist transfers out of loops. Overlap — issue transfers asynchronously so the link works while the device computes, which turns two serial costs into one.
And use pinned host memory for anything on a hot path. Transfers from ordinary pageable memory require the runtime to stage through a pinned buffer it allocates itself, so an "obvious" copy is silently two copies. This is a small change with an unusually large effect, and it is invisible in the source.
1for (batch in batches) {2 copy_to_device(batch) // pageable: staged through a3 // runtime-allocated pinned buffer,4 // so this is really two copies5 run_kernel() // device works; link idle6 copy_from_device(result) // link works; device idle7}8 9// link and device alternate. each waits for the other.10// total = sum of all transfers + sum of all compute1pin(host_buffers) // one-time cost, removes the2 // hidden staging copy entirely3 4for (batch in batches) {5 copy_to_device_async(batch, stream_a)6 run_kernel_async(stream_b) // computes batch N-1 while7 // batch N is still arriving8 copy_from_device_async(result, stream_c)9}10 11// link and device both busy.12// total ~= max(sum of transfers, sum of compute)Neither version changes the kernel or the amount of data. The first serialises transfer and compute so the total is their sum; the second overlaps them so the total is the larger of the two, and removes a staging copy nobody wrote. On a transfer-bound workload this is the entire optimisation.
Key points
- Link bandwidth is lanes × per-lane rate minus overhead — a property of the link, not of the card plugged into it.
- A wide card in a narrow slot negotiates down silently, with no error and no warning.
- Whether the link or the device binds is decided by arithmetic intensity: computation performed per byte delivered.
- On a transfer-bound workload, optimising the kernel produces no measurable improvement at all.
- Pinned memory and asynchronous overlapped transfers are the two highest-leverage changes, and both are invisible in the source.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Host memory → link: the transfer is DMAed across the negotiated lanes at the negotiated generation rate.
- 2Pageable source → staging buffer: if host memory is not pinned, the runtime first copies into a pinned buffer it owns, doubling the work.
- 3Link → branch arbitration: the transfer shares the upstream link with every sibling device behind the same switch.
- 4Link → device memory: bytes land in on-device memory, which the accelerator can read one to two orders of magnitude more cheaply.
- 5Device → kernel: computation proceeds, and if it finishes faster than the next transfer arrives, the device idles on the link.
- • "The GPU is only 20% utilised, so we need a bigger GPU." Low utilisation on a transfer-bound workload means the link cannot feed the device you already have.
- • "The transfers are small compared to the compute, so they do not matter." What matters is bytes per unit of computation, not absolute transfer time.
- • "We are using the stated bandwidth of the generation." That is the raw figure, before overhead, before sharing and before transfer-size effects.
- • "The copy is one line, so it is one copy." From pageable memory it is two, and the second one is not in your source.
Consequences, controls and cost
- • Accelerator utilisation is low and runtime tracks data volume rather than problem size — the signature of a transfer-bound workload.
- • Kernel optimisations produce no measurable improvement, which is usually discovered only after they are complete.
- • A card in the wrong slot delivers a fraction of expected throughput with nothing reporting an error.
- • Copies from pageable memory cost roughly twice what the source suggests, because of a staging copy the runtime performs.
- • Compute arithmetic intensity before optimising anything — it tells you which side of the link to work on.
- • Keep intermediate results on the device rather than round-tripping them through host memory.
- • Use pinned host memory for hot-path transfers to eliminate the hidden staging copy.
- • Overlap transfer with computation using asynchronous copies and multiple streams, so total time approaches the larger of the two rather than their sum.
- • Verify the negotiated link width and generation rather than assuming the card got what it asked for.
- • Achieved transfer bandwidth against the link's negotiated capability, in each direction separately.
- • Accelerator utilisation during the run — sustained low utilisation with high data volume indicates transfer-bound.
- • Negotiated link width and generation, checked rather than assumed.
- • Runtime as a function of data size at fixed problem complexity; linear scaling in bytes points at the link.
- • Pinned memory cannot be paged out, so large pinned regions reserve physical memory the rest of the system cannot use.
- • Overlapping transfers with compute requires restructuring into asynchronous stages, which is more complex and harder to debug.
- • Keeping data resident on the device saves transfers and consumes device memory, which is usually the scarcer resource.
Scope
§224 — what these claims are specific to.
- PLATFORM-SPECIFICLane counts, generations and topology are per machine. Per-generation rates change with each revision, which is why this lesson gives the relationship rather than a table of figures.
- GPU-SPECIFICThe transfer-budget framing applies to discrete accelerators across a link. Integrated devices sharing host memory have entirely different economics and may have no transfer at all.
- SIMPLIFIEDOmits peer-to-peer device transfers, vendor-specific high-bandwidth interconnects that bypass PCIe between devices, and per-direction asymmetry in real links.