The question this answers
When does a workload need an accelerator, and what changes about scheduling, memory and cost when it does?
The team wants to serve its own fine-tuned model rather than call a hosted API — because the data may not leave the boundary, and the request volume is high enough that per-token pricing has stopped being the cheaper option.
Hardware that executes the large parallel matrix operations inference is made of, at a throughput a general-purpose CPU cannot reach — on the condition that the model fits in accelerator memory and the device is kept busy.
Not every workload wants one, and most do not
An accelerator is good at one thing: applying the same operation to enormous arrays in parallel. Model inference and training are made almost entirely of that operation, which is why they run one to two orders of magnitude faster there. Almost everything else your platform does — parsing JSON, serving HTTP, running a query, orchestrating an agent loop — is branchy, latency-bound serial work that a GPU is actively worse at. The tokenizer, the retrieval step and the business logic all belong on CPU.
That split is a scheduling problem, not a philosophy. In a mixed cluster the accelerator is a scarce, explicitly requested resource: a workload declares it needs one device, the scheduler places it only on a node that has a free one, and everything else stays off those nodes. Get this wrong — no request declared, or ordinary pods allowed to land on the expensive nodes — and you end up paying accelerator prices to run a log shipper. The Kubernetes framing of this is a device request plus a taint on the accelerator node pool so that only workloads that tolerate it are scheduled there; see Scheduling: How a Pod Chooses a Node and Requests vs Limits: Two Numbers That Do Different Jobs.
The other structural difference is that accelerators are not fractional by default. A CPU request of 500m genuinely gets you half a core's worth of time slices. A device request is usually whole-device: one pod, one accelerator, exclusive. Sharing exists — time-slicing and partitioning schemes differ by vendor and generation — but it is opt-in, comes with caveats about isolation and memory, and is not something to assume.
| Workload | Where it belongs | Why | What happens if you get it wrong |
|---|---|---|---|
| Model inference (large model) | Accelerator | Dominated by large parallel matrix operations | CPU inference is slow enough that the feature is unusable |
| Embedding generation, bulk | Accelerator, batched | Same operation, highly batchable, latency-tolerant | Serialized on CPU it becomes an overnight job |
| Small classifier or reranker | Usually CPU | Model is small; per-request overhead dominates | You rent a device that sits at 4% utilization |
| Tokenization, pre/post-processing | CPU | Branchy serial string work | The accelerator idles while the CPU stage is the bottleneck |
| Retrieval, API calls, agent loop | CPU | Network-bound and latency-bound | Expensive nodes spend their life blocked on I/O |
| Training or fine-tuning | Accelerator, scheduled | Throughput job with no latency SLO | Run it on the serving pool and inference latency collapses |
Memory is the constraint that actually bites
Engineers arrive expecting the limiting factor to be compute throughput. In practice the first wall is memory: the model weights, plus the activations for whatever batch you are running, plus the cache of attention state for every in-flight request, must all be resident on the device. If they do not fit, the workload does not run slowly — it fails outright, at load time or at the first oversized batch, with an out-of-memory error. There is no gradual degradation and no swapping to host memory that you would be willing to accept.
This has three practical consequences. First, model selection is a capacity-planning decision: a larger model may simply not be deployable on the devices you can actually obtain. Second, batch size is bounded by memory before it is bounded by speed — you increase batching until the device runs out of room for concurrent request state, and that ceiling moves as request lengths change. Third, a workload that ran fine for months can OOM the day someone sends a much longer input, because the per-request memory footprint grows with input length.
The failure looks like OOM Kills and CPU Throttling from the orchestrator's point of view — a container killed, restarted, killed again — but the cause is on the device, not in the host memory cgroup, and host memory graphs will look completely healthy while it happens. That mismatch is why device-level memory metrics have to be collected separately and explicitly; nothing you already monitor reports them.
node device mem-used / mem-total util% resident model in-flight ------------ ------ -------------------- ----- -------------- --------- gpu-pool-01 0 62% of capacity 11% serving-7b 1 gpu-pool-01 1 61% of capacity 9% serving-7b 1 gpu-pool-02 0 63% of capacity 14% serving-7b 2 gpu-pool-03 0 0% of capacity 0% (none) 0 <- rented, empty, billing reading: memory is committed by the resident weights, so it looks busy. util% is what you are actually paying for, and it is ~11%. batch size 1 with one request per device is the whole explanation.
Utilization and batching are the entire cost story
A GPU instance bills for every hour it exists. It does not bill less when it is idle, it does not scale to zero on its own, and in most regions it is one of the most expensive things a provider will rent you. So the only question that matters financially is: what fraction of the hours you paid for did useful work? A fleet at 12% device utilization is a fleet where roughly seven of every eight euros bought nothing.
Batching is the lever. Because the device does the same operation across an array, processing eight requests together costs far less than eight times one request — the weights are already resident, and the parallel units were mostly idle at batch size one. Continuous or dynamic batching, where the server keeps admitting new requests into an in-flight batch rather than waiting for a fixed window to fill, is what turns a 12% fleet into a 60% fleet without buying anything. The cost is tail latency: a request that arrives just after a batch is admitted waits, and the p99 gets worse even as throughput and cost-per-request get dramatically better. That is a real trade-off to make deliberately, not a free win.
The second lever is separating the pools. Interactive inference has a latency SLO and needs headroom; batch embedding and fine-tuning have no SLO and should soak up capacity opportunistically, ideally on interruptible/spot capacity with checkpointing. Running both on one pool means either the batch job ruins interactive latency or the interactive headroom sits idle. And the third lever is the least glamorous: scale the pool down. Accelerator capacity left running over a weekend because scaling it down felt risky is the single most common line item in an "our AI costs exploded" review. See Idle Capacity: Headroom or Waste? and Right-Sizing Without Causing an Outage.
Bars are relative weights, not currency. Real rates depend on provider, region, commitment and volume.
Key points
- Accelerators are for large parallel array operations; tokenization, retrieval, orchestration and business logic all belong on CPU.
- The model, its activations and per-request state must fit in device memory — this is a hard failure boundary, not a slowdown.
- Device requests are usually whole-device and exclusive, so scheduling means a dedicated node pool plus explicit requests, not fractional shares.
- A GPU instance bills the same idle as saturated; utilization percentage is the cost metric that matters.
- Continuous batching is the main lever for utilization, and it trades tail latency for throughput and cost per request.
- Separate interactive serving from batch and training pools, or one of them will always be paying for the other's headroom.
The loop, answered
Every field is required, which is why no lesson here can recommend something without saying what it costs and what simpler thing to consider first.
- • A workload declares a device requirement; the scheduler places it only on nodes advertising a free device, typically an isolated node pool.
- • On startup the runtime loads model weights from storage into device memory, where they stay resident for the life of the process.
- • Incoming requests are queued by the inference server and admitted into a batch; the batch executes as one pass over the resident weights.
- • Per-request state — attention cache and activations — is allocated on the device and released when the request completes, so concurrency is memory-bounded.
- • Results are returned to CPU for post-processing and serialization; the device moves immediately to the next batch.
- • On scale-out, a new node must be provisioned, pull the weights and load them before it serves anything — a multi-minute cold start.
- • Keep accelerator nodes in their own pool with placement restrictions, so nothing without a device requirement can land there.
- • Collect device utilization and device memory as first-class metrics — host CPU and host memory tell you nothing about either.
- • Own driver and runtime versions: the driver, the container toolkit and the framework must agree, and a mismatch presents as "no device found" rather than as a version error.
- • Tune batch size and admission policy against a real request-length distribution, and re-tune when the distribution shifts.
- • Checkpoint any training or long batch job, because interruptible capacity is where the savings are and it will be reclaimed.
- • Decide and automate the scale-down policy before launch; nobody ever does it later.
- • Device out-of-memory on a longer-than-usual input: the process dies, restarts, loads weights for minutes, and dies again on the retry of the same request.
- • Driver/runtime version mismatch after a node image update: the pod starts, finds no device, and either crashes or silently falls back to CPU at 1/50th the speed.
- • A pod without a device request scheduled onto the accelerator pool, occupying an expensive node to do nothing.
- • Batch job admitted to the interactive pool: p99 latency triples and the cause is invisible in application traces.
- • Scale-out lag — the autoscaler adds a node, but weights take minutes to load, so the queue drains long after the traffic spike ended. See Startup Time & Cold Start.
- • Capacity simply unavailable: accelerator instance types are frequently constrained by region, and "we will just scale out" assumes an inventory that may not exist.
- • Device memory runs out before device throughput does; concurrency is bounded by per-request state, not by clock speed.
- • Batching raises throughput sub-linearly in cost and linearly in tail latency — there is an optimum, and it is workload-specific.
- • Scale-out is minutes, not seconds, because weights must be pulled and loaded, so accelerator pools need more headroom than stateless web tiers.
- • Scaling to zero is possible for batch workloads and usually unacceptable for interactive ones, purely because of that cold start.
- • Beyond one node, distributed training introduces interconnect bandwidth as a new bottleneck; single-node inference never meets it.
- • Accelerator nodes usually run vendor drivers and privileged device plugins — a larger and more privileged host surface than an ordinary worker node.
- • Device memory is not reliably zeroed between tenants unless the platform guarantees it; treat multi-tenant device sharing as a real isolation question, not a scheduling detail.
- • Model weights are valuable intellectual property and often trained on regulated data: their storage bucket deserves the same classification and access control as a database.
- • Self-hosting is frequently chosen *for* a security requirement — the data never leaves the boundary — so undermining it with a public inference endpoint and no authentication defeats the point.
- • The inference endpoint itself should sit behind the platform's normal identity and rate-limiting layers; it is an expensive resource and an unauthenticated one is a denial-of-wallet target.
- • Hourly device rental dominates, and it is charged identically whether the device is saturated or idle.
- • Effective cost per request equals device-hour price divided by requests actually served in that hour — batching is the only meaningful lever on the denominator.
- • Interruptible/spot capacity is materially cheaper and appropriate for checkpointed batch and training work, not for interactive serving.
- • Committed-use or reserved pricing rewards steady utilization and punishes a pool you meant to turn off.
- • Weight storage and repeated cold-start pulls are small but real, and they scale with node count rather than with traffic.
- • Device utilization percentage, per device — the single number that says whether the money is buying anything.
- • Device memory used against capacity, with headroom tracked against the longest inputs you actually receive.
- • Average batch size and queue wait time at the inference server, which together explain both cost and tail latency.
- • Requests served per device-hour, as the cost-efficiency metric leadership will actually ask about.
- • The signal that lies: host CPU and host memory on an accelerator node, which look calm and idle while the device is either saturated or, worse, empty.
- • A hosted model API. For most teams, most of the time, this is the correct answer: no capacity risk, no drivers, no idle billing, and you pay only for what you use. Consider self-hosting only when data residency, volume economics or a genuinely custom model forces it. See Hosted APIs, Managed Inference or Your Own Cluster.
- • A smaller model on CPU. Rerankers, classifiers and small embedding models often run acceptably on ordinary instances, and that removes an entire class of operations.
- • Managed inference endpoints, where the provider owns the devices, the drivers and the scaling and you deploy a model artifact — most of the control, far less of the operational burden.
- • Batching offline instead of serving online: if the answer is not needed within a second, a scheduled batch job on interruptible capacity is a fraction of the cost of a warm pool.
- • Caching. A large share of production inference requests are near-duplicates, and a cache hit costs nothing at all.
- • Buys throughput no CPU fleet can match; costs a scarce, expensive, always-billing resource with a multi-minute cold start.
- • Batching buys utilization and cost efficiency; costs tail latency, and the trade is not adjustable after the fact without re-tuning.
- • Self-hosting buys data control and predictable unit economics at volume; costs driver management, capacity risk and a permanent operational commitment.
- • Interruptible capacity buys a large discount; costs you the obligation to checkpoint and to tolerate reclamation.
Batching on a GPU: throughput bought with latency
effective batch = min(max batch 8, VRAM ceiling 61, arrivals within 40 ms 2) = 2 throughput = batch ÷ batch time = 2 ÷ 0.025 s = 80 rps · latency = fill wait + batch time
Where the bill actually comes from
fixed weight is committed at provision time; usage weight follows the workload. idle = 100% − 35% used → headroom 25% (chosen) + waste 40% (not chosen)
What people believe, and what is true
A GPU makes the application faster.
It makes large parallel array operations faster. Serving HTTP, parsing, querying and orchestrating are unaffected, and a GPU-attached instance running those is a very expensive ordinary server.
The limit is how many FLOPs the device can do.
The limit you meet first is memory. Weights plus per-request state must be resident; when they are not, the workload fails rather than slows.
We will autoscale the GPU pool like the web tier.
Weight loading makes scale-out a multi-minute operation, so the pool needs standing headroom. Aggressive scale-to-zero and an interactive latency SLO are not compatible.