Accelerators: The Specialization Spectrum
GPUs, AI accelerators, NPUs and FPGAs are not four unrelated products. They are points on one axis running from fully general to fully fixed, and each step along it trades away the ability to run arbitrary work in exchange for doing one kind of work with less silicon and less power.
Software view, hardware view
The gap between what you wrote and what the machine does is where this whole domain lives.
One axis, several points on it
Every step from general towards specialized removes decisions from runtime and bakes them into the hardware. A CPU fetches, decodes, predicts, reorders and schedules — all machinery that exists to handle work it was not told about in advance. Remove the requirement to handle arbitrary work and all of that machinery becomes unnecessary, and the area and power it consumed can go to arithmetic instead.
That is where the efficiency comes from, and it is why the gains are not a marketing artefact: an accelerator genuinely does more useful arithmetic per watt, because a much larger fraction of it is arithmetic. The price is exactly proportional — it can only do the thing it was built for, and work that does not match runs badly or not at all.
FPGAs occupy an unusual position on this axis. They are reconfigurable, so the specialization happens after manufacture and can be changed. That buys the ability to build a fixed-function pipeline for a workload without committing to silicon, at the cost of lower clock speeds and much lower density than a fixed implementation of the same function.
| Device | What is fixed | Efficiency on target work | Cost of being wrong |
|---|---|---|---|
| CPU | Almost nothing; everything decided at runtime | Lowest — most area is control, not arithmetic | None; it runs whatever you have |
| GPU | Execution model: wide, uniform, lockstep lanes | High on wide regular work | Divergent or narrow work runs poorly |
| AI accelerator / NPU | Operation mix: matrix multiply and a fixed set of others | Very high on that operation mix | Anything outside the supported set falls back or fails |
| FPGA | Chosen after manufacture, changeable | Moderate-to-high; beats a CPU, rarely beats fixed silicon | Long development cycles; reconfiguration is not cheap |
| Fixed-function block | Everything, at design time | Highest possible for that one function | Useless for anything else; obsolete when the workload shifts |
What each one actually removes
It is more illuminating to ask what a device *lacks* than what it has. A GPU lacks the per-lane branch prediction and out-of-order machinery a CPU has, which is precisely why it can afford so many lanes — and precisely why divergent control flow hurts. An NPU typically lacks general programmability altogether: it implements a set of operations, and the compiler's job is to express your model in terms of that set.
This framing predicts behaviour on unsupported work. Ask what happens when a model contains an operation the accelerator does not implement: usually it falls back to a host CPU, which means a round trip across a bus in the middle of the graph. One unsupported operation in the wrong place can cost more than everything the accelerator saved, and the failure is invisible in a per-operation benchmark — it appears only in the end-to-end number.
The same reasoning applies to precision. Accelerators frequently achieve their headline throughput only at reduced precision, and a model requiring higher precision for a particular layer may fall back or run at a fraction of peak. Knowing which operations and precisions are native is more predictive than any published throughput figure.
1// Model graph, as written:2// conv -> relu -> custom_op -> conv -> softmax3//4// Accelerator supports conv, relu, softmax. Not custom_op.5//6// What actually executes:7// [accel] conv, relu8// -> copy activations to host <-- bus crossing9// [host] custom_op10// -> copy activations back <-- bus crossing11// [accel] conv, softmax12//13// Every supported op benchmarked beautifully. The graph is slow,14// and the profile shows an idle accelerator waiting on a bus.Choosing a point on the axis
The decision turns on how stable and how large the workload is. A stable, large, well-understood workload justifies moving right along the axis: the efficiency gain is real and it compounds across every unit deployed. An evolving or uncertain workload argues for staying left, because flexibility is what lets you absorb a change of shape without changing hardware.
Volume matters independently, because the cost of specialization is largely fixed — design effort, toolchain investment, a codebase that only runs on one device. Amortised across a large deployment those are small; across a small one they dominate. This is why the same technical decision comes out differently for a hyperscaler and for a single team.
The honest default for most software is to stay left. GPUs are already specialized enough to deliver large gains on suitable work, and they retain enough generality to survive a change in what the workload looks like. Moving further right is a bet that the workload will not change materially before the hardware is retired — and The Specialization Trade-off is about what happens when that bet is wrong.
- Stable and large — specialization pays, and the efficiency compounds across the deployment.
- Evolving or uncertain — stay general; flexibility is the thing you are actually buying.
- Low volume — fixed costs of specialization dominate regardless of the per-unit efficiency.
- Latency-critical and small — a CPU often wins outright; there is nothing to amortise the fixed costs against.
- Check the operation coverage first — one unsupported operation mid-graph can erase the entire advantage.
Key points
- Accelerators sit on one axis from general to fixed; each step removes runtime decisions and the silicon that supported them.
- The efficiency gain is real because a larger fraction of the device is arithmetic rather than control.
- Ask what a device lacks, not what it has — that predicts how it behaves on work it was not designed for.
- One unsupported operation mid-graph can cost more than the accelerator saves, and per-operation benchmarks hide it.
- Specialization is a bet that the workload will not change before the hardware is retired.
Follow the mechanism
The path through the machine, hop by hop — and the conclusions it invites that are wrong.
- 1Design time → fixed decisions: control, scheduling and operation mix are decided in silicon rather than at runtime.
- 2Removed control → freed area: the transistors that would have decoded, predicted and reordered become arithmetic instead.
- 3Compiler → supported operations: the toolchain maps the program onto the fixed operation set the device implements.
- 4Unsupported operation → host fallback: execution leaves the device mid-graph and crosses a bus in both directions.
- 5Workload drift → stranded hardware: when the shape of the work changes, fixed decisions cannot be revisited.
- • "It is faster on every published benchmark, so it will be faster on ours" — published benchmarks use graphs chosen to be fully supported.
- • "An FPGA gives ASIC efficiency with software flexibility" — it gives some of each and neither in full, at much lower clocks and density.
- • "Specialized means it cannot run our code at all" — usually it runs it, but partly on the host, which is worse than either extreme.
- • "We should buy the most specialized option available" — only if the workload is stable and large enough to amortise it.
Consequences, controls and cost
- • Large efficiency gains per watt on the target workload, which is why specialized hardware dominates at scale.
- • Sharp performance cliffs when a workload steps outside what the device implements natively.
- • Toolchain lock-in: code written for one accelerator rarely moves to another without substantial rework.
- • Hardware can be stranded by a change in workload shape long before it wears out.
- • Check native operation and precision coverage against your actual graph before selecting a device.
- • Prefer devices one step less specialized than the minimum that meets the requirement, to retain room for change.
- • Keep a working general implementation so a fallback path exists when coverage gaps appear.
- • Measure end to end rather than per operation, since fallbacks are invisible in per-operation numbers.
- • Compile your real graph for the target device and read the report of which operations mapped natively.
- • Profile end to end and look for host fallbacks — an accelerator idle while a bus is busy is the signature.
- • Compare performance per watt on your workload, not on the vendor's, since that ratio is the entire argument for specializing.
- • Track how often the workload has changed shape historically; that rate is what the specialization bet is against.
- • Efficiency is bought with flexibility, and the exchange rate is unfavourable when the workload moves.
- • Each device brings a toolchain, a debugging story and a hiring requirement of its own.
- • Hardware lead times are long relative to how fast workloads change, especially in machine learning.
- • Fallback paths must be kept working, which means maintaining two implementations rather than one.
Scope
§224 — what these claims are specific to.
- PLATFORM-SPECIFICOperation coverage, precision support and fallback behaviour differ entirely by vendor and toolchain, and change between software releases for the same hardware.
- GPU-SPECIFICThe GPU row describes discrete general-purpose GPUs; parts with dedicated matrix units sit further right on this axis than the row suggests.