Throughput vs Latency
Throughput is predictions per unit time; latency is how long one waits. Batching raises the first by spending the second, and queue depth — not CPU — is the signal that says you are running out of both.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
We need more predictions per second without blowing the per-request latency budget. Which of batching, concurrency and autoscaling buys what, and what does each cost in latency?
A content platform scores every uploaded image for policy violations with a convolutional network. Uploads have tripled; the queue of unscored images grows during the evening peak and the moderation team sees violations hours late. The per-image model call is fast, but the fleet cannot keep up.
Add more GPU workers. Throughput is workers times per-worker rate, so double the fleet and the backlog clears. Keep each worker scoring one image at a time so latency stays minimal.
Doubling the fleet doubles a fleet that is mostly idle. The bottleneck is not GPU compute; it is a worker loop that runs one image through a device built for thousands at once (GPU Fundamentals).
- Doubling the fleet doubles a fleet that is mostly idle. The bottleneck is not GPU compute; it is a worker loop that runs one image through a device built for thousands at once (GPU Fundamentals).
- Cost doubles, throughput improves by less than that, and the evening backlog still grows because arrival rate at peak exceeds the new capacity by a margin the mean-based capacity plan never saw.
- Mean latency is low because it is measured per model call, not from upload to score. The moderator's experience — queue age — is hours.
- Autoscaling is on GPU utilisation, which stays low because the workers are the bottleneck, so the autoscaler never scales.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model classifies an image as violating or not; the serving system's target is to score every upload within a bounded delay at peak volume, at a cost the platform can sustain.
- The decision is asynchronous — content is published and reviewed shortly after — so a modest per-image latency is acceptable and a growing backlog is not.
- Uploads arrive on a queue at a rate that varies by a factor of ten across the day. Each upload is one inference; the network runs on a GPU fleet behind a worker pool that pulls one image at a time.
- The GPU is busy for a few milliseconds per image and idle while the worker decodes the next one, so utilisation is low even during the backlog.
- Nobody measures queue age; the dashboard shows GPU utilisation and mean latency, both of which look healthy.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Throughput is completed predictions per second; latency is the time from a request's arrival to its result. For a single server they are linked by concurrency: with
Lrequests in the system on average, arriving at rateλ, each spends on averageW = L / λ— Little's law. Raise concurrency to raise throughput and the average wait rises with it once the server is busy. - Batching changes the per-prediction cost. A GPU processes a batch of sixty-four images in barely more time than one, so throughput per device rises almost linearly with batch size — but each request waits for the batch to fill. Batching converts latency into throughput at an exchange rate set by the batch size and the wait cap (Inference Batching).
- Below saturation, latency is roughly the service time and throughput equals the arrival rate. Approaching saturation, queueing time grows sharply — the latency curve bends upward long before utilisation reaches its ceiling. That knee is why queue depth is the honest signal and utilisation is a late one.
- Autoscaling on queue depth or queue age acts before the knee; autoscaling on CPU or GPU utilisation acts after it, and often not at all when the bottleneck is elsewhere in the worker.
Little's law at the model server
The relationship is L = λ · W: average requests in the system equal arrival rate times average time in the system. It holds for any stable queue. Read it as a constraint: to serve a higher λ without raising W, the system must hold more requests in flight, and it can only do that if the server can genuinely work on them concurrently rather than lining them up.
A GPU worker that scores one image at a time has concurrency one at the device. Batching raises the device's effective concurrency to the batch size at almost no extra service time, which is why it is the lever — and why the added wait for the batch is the price.
1async def batch_loop(queue, model, max_batch=64, max_wait_ms=5):2 while True:3 first = await queue.get() # block until there is work4 batch = [first]5 deadline = now_ms() + max_wait_ms6 while len(batch) < max_batch and now_ms() < deadline:7 try:8 batch.append(queue.get_nowait())9 except QueueEmpty:10 await sleep_ms(0.5) # give arrivals a chance11 inputs = collate([b.input for b in batch])12 outputs = model(inputs) # one device call for the whole batch13 for req, out in zip(batch, outputs):14 req.future.set_result(out)The wait cap is the latency you are willing to spend. At peak the batch fills long before the deadline and the cap costs nothing; at night every request pays the full cap. A cap that adapts to arrival rate fixes the overnight case.
What each lever buys and costs
Batching, concurrency and autoscaling are not alternatives; they operate at different layers. Batching raises per-device throughput. Concurrency — more workers per device, pipelining decode and compute — raises device utilisation. Autoscaling raises the number of devices. Each has a latency price and a cost price.
The mistake in the problem statement was reaching for the third lever first. It is the most expensive and it does nothing while the first two are unused.
| Option | Latency | Cost | Operational | Note |
|---|---|---|---|---|
| Batching at the model boundary | Large throughput gain per device on an accelerator; adds up to the wait cap per request; needs a batching layer in the server. | |||
| Pipelined workers (decode overlaps compute) | Raises utilisation without adding latency; some engineering in the worker; gains bounded by the slower of the two stages. | |||
| Autoscale on queue depth | Adds capacity at peak; cold starts load the model and take minutes; oscillation and cost if the signal is noisy. | |||
| Add fixed capacity for peak | Simple and predictable; idle most of the day; still starved if the worker loop is the bottleneck. |
caveat The scores assume an accelerator-backed neural network with an asynchronous path; on a CPU tree ensemble the batching row drops to almost no gain, and for a synchronous path with a tight budget the latency column dominates every other consideration.
Queue depth is the signal
The autoscaler was watching GPU utilisation and never fired, because the device was starved rather than busy. Queue depth would have fired within a minute of the evening peak starting. The assumption a throughput design makes is that its scaling signal leads the failure rather than lagging it.
This is where the perf domain's queueing material earns its cross-link: the knee in the latency curve, saturation and queue age are its vocabulary, and the model server is one more queue.
Queue depth or queue age triggers new capacity early enough that queue age never exceeds the review SLO at peak.
holds when The signal is queue-based, capacity is planned against the peak arrival rate with headroom, and warm capacity or fast model loading covers the scale-up delay.
breaks when The signal is utilisation on a device that is starved rather than busy; a model change lowers per-device throughput; the peak grows faster than the fleet's scale-up time.
respond Fix the worker loop and batching before adding devices; then re-plan capacity against the observed peak and re-benchmark after every model change.
How to build it
Most important first.
- Batch at the model boundary with a wait cap: collect requests for up to a few milliseconds or until the batch is full, whichever first. The cap bounds added latency; the batch raises device throughput several-fold.
- Pipeline the worker so decoding and preprocessing overlap with model compute; the GPU should never wait for a CPU to decode the next image.
- Autoscale on queue depth or queue age, with capacity planned against the peak arrival rate, not the mean. A queue that grows at peak is under-provisioned at peak regardless of what the daily average says.
- Separate the latency-sensitive and the throughput-sensitive paths if both exist: a synchronous "is this upload obviously illegal" check with tiny batches and a strict cap, and the asynchronous full scoring with large batches.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Queue age — the age of the oldest unscored item — and queue depth over time. These map directly to the moderator's complaint and to the autoscaling decision.
- Predictions per second per device against device utilisation; low utilisation with a growing queue says the worker, not the device, is the bottleneck.
- Per-call model latency is the number the team has and the one that says least about the backlog.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Peak arrival rate stays below the fleet's batched throughput with headroom, and the autoscaler's signal — queue depth — reacts before queue age exceeds the review SLO.
- The batch wait cap keeps added latency inside the budget for the latency-sensitive path, and low-traffic periods do not turn the cap into the latency floor.
- Model changes that alter per-batch compute — a bigger network, a larger input size — are re-benchmarked for throughput before deployment, because the capacity plan depends on that number.
- Offline: load-test the worker with recorded upload traffic at peak rate and above, and record throughput and latency at each batch size; find the knee.
- Online: alert on queue age against the review SLO and on device utilisation falling while queue depth rises.
- Over time: re-run the load test after every model change and every fleet change; the knee moves with both.
What can go wrong
- Dynamic batching with a generous wait cap serves the low-traffic overnight hours with every request waiting the full cap for a batch that never fills; latency rises when the system is emptiest.
- Queue-depth autoscaling adds workers that all cold-start together, each loading the model, and the queue grows during the minutes of warm-up — then they all scale down together and the cycle repeats.
- A batch containing one oversized image runs at the pace of that image, and the sixty-three others wait; tail latency now depends on the worst input in the batch.
- Batching adds latency — the wait for the batch — and complexity in the request path; it is the right trade for the asynchronous path and the wrong one for a synchronous check with a tight budget.
- Autoscaling on queue depth is responsive and can oscillate; a floor of warm capacity costs money at night to avoid cold-starting the fleet at the evening peak.
- Capacity planned for peak is idle most of the day. Spot or preemptible capacity is cheaper and can vanish at peak (GPU and Accelerator Infrastructure and The Instance Lifecycle in the cloud domain).
- "GPU utilisation is low, so we have plenty of capacity." Low utilisation with a growing queue means the device is starved by the worker. Capacity is being wasted, not held in reserve.
- "Add servers to fix latency." Servers add throughput. If the latency is queueing, throughput helps; if it is service time — a slow model, a slow fetch — it does not, and the breakdown in Latency Breakdown says which.
- "Bigger batches are always better." Up to the device's saturation point, yes for throughput; beyond it, no gain and a longer wait for every request. And any batch is a latency cost the synchronous path may not afford.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALLittle's law and the queueing knee apply to any server, not only model servers; the ML-specific part is that batching on an accelerator changes the per-prediction cost so sharply that the trade is worth making explicitly.
- MODEL-SPECIFICBatching buys a large multiple on GPUs running neural networks, where the device is underused by single requests; for a tree ensemble on CPU the per-prediction cost barely changes with batch size and the latency spent on batching buys almost nothing.
- SIMPLIFIEDLittle's law is stated for averages and a stable system; real systems have bursty arrivals and the tail behaves worse than the average suggests, which the performance domain treats properly and this lesson only names.
Where the depth lives
This domain teaches the model and hands the rest off by name.