The question this answers
Is this work uniform enough, arithmetic-dense enough and large enough to be worth shipping to a separate device and back?
A batch of 50,000 embedding vectors that must each be compared against a 4-million-row index — and, as the counter-example, one 300-element array that needs normalizing inside a request handler.
Nothing is shared between host and device in the ordinary sense: they hold separate memories, and the "sharing" is an explicit copy in each direction. Within a kernel, threads in a block share a small fast scratch memory and must coordinate through barriers; threads in different blocks share only global device memory.
Every element of the output buffer is written by exactly one device thread before the host reads it back, and the host does not read the result buffer until the kernel that wrote it has completed.
Synchronization exists to preserve this sentence. If a schedule can make it false, the code is wrong no matter which primitive it uses.
Four steps, and only one of them is your computation
Using a GPU is not "calling a faster function". It is a four-step protocol: allocate device memory, copy input across the bus, launch a kernel that runs the same small program across thousands of threads, copy results back. Two of those four steps are data movement over an interconnect that is far narrower than either the host's or the device's own memory bandwidth, and they are paid in full regardless of how clever the kernel is.
That is the whole economics of offload. If the kernel takes 2ms and the transfers take 18ms, you built a 20ms operation to replace an 8ms CPU loop. The break-even is a ratio — arithmetic operations per byte transferred — and it is why GPUs dominate matrix multiplication and neural network inference (huge arithmetic intensity, data reused many times once resident) and lose badly on "apply one cheap function to a modest array once".
The launch itself is asynchronous and that is a correctness point as much as a performance one: the launch call returns before the kernel finishes. Reading the output buffer without synchronizing is not "a bit early", it is reading undefined data. Every GPU API therefore has an explicit synchronize or an event to wait on, and it is exactly Fork/Join with a very wide fork.
- The launch returns before the kernel completes. Reading results without synchronizing is a correctness bug, not a timing one.
- Transfers are paid per offload, so batching many small offloads into one large one is usually the single biggest win available.
- Keeping data resident on the device across many kernels removes the transfers from the inner loop entirely — the shape every high-performance GPU pipeline converges on.
Where the time goes for a small offload
The timeline below is the argument. Two workloads, the same device: one small array normalized once, and one large batch of vector comparisons. The kernel lane is the only lane doing your computation; every other lane is logistics. For the small job the logistics dominate by an order of magnitude, and the host thread is blocked in synchronize for most of it — so you did not even get concurrency out of the deal.
The fix for the small case is not a better kernel. It is to not offload, or to batch until the ratio inverts. That is why inference servers accumulate requests into batches before dispatching, accepting queueing latency to buy arithmetic intensity — a direct trade of p50 for throughput, and the same reasoning as Bounded vs Unbounded Queues applied to a device instead of a worker pool.
The second structural fix is overlap: while kernel N runs, copy the input for batch N+1 and copy back the output of batch N-1. With separate copy engines and multiple streams the three lanes genuinely run at once, which turns the protocol into Pipeline Parallelism: Different Items, Different Stages with the device as the middle stage. It does not make the transfers cheaper; it hides them behind the kernel.
Uniform work only — and what "uniform" costs you
GPU threads are scheduled in groups that execute in lockstep. When threads within a group take different branches, the hardware runs both paths and masks off the inactive threads, so a kernel whose branch outcome varies per element pays for every path taken by any thread in the group. This is the same predication idea as SIMD: One Instruction, Many Elements, scaled up and made much more punishing, and it is why "highly parallel" is a necessary but insufficient condition — the work must also be *homogeneous*.
Memory access patterns matter for the same structural reason. Adjacent threads reading adjacent addresses have their loads combined into a small number of wide memory transactions; adjacent threads reading scattered addresses do not, and the effective bandwidth collapses. A hash lookup per element, a linked structure, or an indirection table can make a nominally parallel kernel slower than the CPU loop it replaced.
The reasoning summary for a programmer: a GPU is not a general parallel machine you offload arbitrary work to. It is an extremely wide data-parallel machine that rewards uniform arithmetic over contiguous data and punishes branching, indirection and small batches. Everything else — occupancy, shared memory, warp scheduling — is tuning within that envelope, and belongs to Computer Architecture rather than here.
| Workload trait | GPU fit | Why | If it does not fit |
|---|---|---|---|
| Dense matrix / tensor math | Excellent | Enormous arithmetic intensity; data reused many times once resident | — |
| Elementwise math over millions of items | Good | Uniform, contiguous, coalesced access | Vectorize + thread on CPU |
| Per-element branching on data values | Poor | Divergent groups execute both paths under a mask | Sort or bucket by branch outcome first |
| Pointer chasing / hash probes | Poor | Uncoalesced access destroys effective bandwidth | Keep on CPU; fix the data layout |
| Small arrays inside a request | Bad | Transfer and launch overhead dwarf the kernel | Do not offload; or batch across requests |
| Long dependency chains | Bad | Nothing to run in parallel; thousands of lanes sit idle | CPU, and look for a different algorithm |
| I/O-bound work | Irrelevant | The bottleneck is waiting, not arithmetic | Async Is Not Parallelism |
Key points
- Offload is a four-step protocol and two of the steps are data movement over a comparatively narrow bus; the kernel is often the smallest part.
- The break-even metric is arithmetic operations per byte transferred, not "is the work parallel".
- Kernel launches are asynchronous — reading the output buffer without synchronizing reads undefined data, which is a correctness bug.
- Threads execute in lockstep groups, so per-element branching makes the group pay for every path taken; uniformity matters as much as parallelism.
- Batching and keeping data resident across kernels are the two moves that turn a losing offload into a winning one.
The loop, answered
Every field is required, which is why no lesson here can recommend concurrency without naming the interleaving that breaks it, the complexity it adds, and the simpler thing to consider first.
- • Allocate buffers in device memory; the host cannot dereference these pointers and the device cannot dereference host pointers.
- • Copy the input across the interconnect — a cost proportional to bytes, with a meaningful fixed component per transfer.
- • Launch a kernel as a grid of blocks of threads; the call returns immediately and the work is queued on a stream.
- • Each thread computes its own index from its block and thread identifiers and writes its own output element — data parallelism with a very large W.
- • Synchronize on the stream or an event, then copy results back; only after that completes may the host read the results.
- • Host and device run concurrently after a launch: host issues launch; host reads result buffer; device kernel writes result buffer — the host read observed uninitialized memory and the program printed zeros with no error.
- • The correct schedule inserts the wait: host issues launch; device runs kernel to completion; host waits on the event; host copies device->host; host reads. The wait is what establishes the ordering.
- • Two kernels on the same stream are ordered by the stream; two kernels on *different* streams are not, so a second kernel reading the first's output without an event dependency can read partial results.
- • Within a block, thread 0 writes shared scratch; thread 31 reads it before thread 0's write — unless a block-level barrier separates them. This is the on-device equivalent of Barriers.
- • Across blocks there is no ordering at all during a kernel. Code that assumes block 0 finishes before block 7 starts is relying on a schedule the hardware never promised.
- • A stream guarantees that operations enqueued on it execute in the order enqueued — a FIFO ordering guarantee, and the main tool for expressing dependencies.
- • A completed synchronize guarantees the kernel's writes to device memory are visible to a subsequent copy on that stream.
- • A block-level barrier guarantees that all threads in the block reached it and that their prior shared-memory writes are visible to the others.
- • There is NO guarantee of ordering between blocks, or between operations on different streams without an explicit event dependency.
- • There is NO guarantee that a kernel launch has started, let alone finished, when the launch call returns. Error reporting is often deferred to the next synchronization point, which is why GPU errors surface far from their cause.
- • The host-device interconnect is a shared, comparatively narrow resource; multiple processes or multiple streams contend for it and each other's transfers become queueing delay.
- • Device memory bandwidth is contended by all resident blocks — the on-device version of Memory Bandwidth: More Cores, Same Bus, with the same tell: adding parallelism stops helping while the device looks busy.
- • Multiple processes sharing one physical device time-slice or partition it, so a "dedicated" GPU in a shared host is often neither dedicated nor predictable.
- • The host thread blocked in a synchronous wait is a thread doing nothing; in a server that thread was supposed to be serving requests. Use asynchronous completion, or a dedicated offload worker.
- • Reading an output buffer before synchronizing: undefined data, frequently zeros, frequently correct on a fast machine and wrong on a loaded one — a genuine Heisenbugs: The Bug That Leaves When You Look at It case.
- • Out-of-memory on the device when batch size scales with load; the failure is abrupt and often kills the whole process rather than degrading.
- • Deferred error reporting: a fault inside kernel N is reported at the synchronize after kernel N+3, sending you to the wrong code.
- • Divergence collapse: a kernel that was fast in testing becomes slow in production because real data branches unevenly.
- • A losing offload that nobody notices, because the GPU version is "obviously faster" and nobody compared it against the vectorized CPU loop.
- • Dense linear algebra, model training and inference, image and video processing, large-scale similarity search — arithmetic-dense, uniform, and enormous.
- • Throughput-oriented batch pipelines where queueing a few milliseconds to build a bigger batch is acceptable.
- • Work that can stay resident on the device across many kernels, so the transfers are paid once rather than per operation.
- • Small per-request work: the transfer and launch overhead exceed the entire CPU implementation.
- • Branchy, data-dependent control flow, where lockstep execution makes you pay for every path.
- • Irregular memory access — graphs, hash tables, linked structures — where coalescing fails and effective bandwidth collapses.
- • Latency-critical single operations, where a device queue shared with other work adds unpredictable delay.
- • Any budget conversation where the accelerator is idle 95% of the time; see GPU and Accelerator Infrastructure in Cloud for what that costs.
- • Split elapsed time into transfer, launch and kernel. If the kernel is under half, the answer is batching or residency, not kernel tuning.
- • Compute arithmetic intensity — operations per byte moved to the device. A low number predicts a losing offload before you write the kernel.
- • Always benchmark against a *vectorized, multithreaded* CPU baseline, not a naive single-threaded loop. The honest comparison changes many decisions.
- • Track device memory high-water mark against batch size; the ceiling is a hard failure, not a slowdown.
- • Track device utilization over the whole deploy, not the kernel. Low utilization plus a large bill is the most common accelerator finding.
- • Two memory spaces, explicit lifetimes and explicit transfers — an entire class of resource-management bugs the CPU code did not have.
- • A second implementation to maintain, because you still need a CPU path for small inputs, unsupported hardware and local development.
- • Asynchronous, deferred error reporting makes debugging structurally harder: the stack you get is not the stack where the fault happened.
- • Batching for efficiency adds a queue, a timeout, a maximum batch size and a partial-failure story — real distributed-systems surface inside one process.
- • Deployment and cost complexity: driver and toolkit versions, scheduling on shared devices, and a resource that is expensive whether or not it is busy.
- • Vectorize and thread on the CPU first (SIMD: One Instruction, Many Elements, Fork/Join). For many workloads this closes most of the gap with none of the transfer cost or operational surface.
- • A better algorithm: an approximate nearest-neighbour index beats brute-force similarity search by more than a GPU beats a CPU, and runs anywhere.
- • A managed inference or batch service, when the GPU work is one well-known operation and you do not want the operational surface at all.
- • Do it offline. If results can be precomputed, the latency argument for an accelerator in the request path disappears entirely.
CPU parallelism simulator
Amdahl’s term, a synchronisation term, an oversubscription term and a bandwidth ceiling, each one a knob you can switch off. Real curves have more causes than four and are rarely this smooth. There is no ideal core count to read off this chart.
What people believe, and what is true
GPUs are just faster processors, so parallel work belongs there.
They are a different device with separate memory. The work has to cross a bus twice, and for anything small that cost exceeds the entire computation.
If the kernel launched without an error, it ran correctly.
The launch is asynchronous and errors are usually reported at the next synchronization point. A clean launch tells you nothing about execution.
More parallelism always helps on a GPU.
Only uniform parallelism. Divergent branches make the group execute every path, and scattered memory access collapses effective bandwidth.
The GPU version is faster because it beat our Python loop.
That comparison is meaningless. Benchmark against a vectorized, multithreaded native baseline before concluding anything.