Inference Modes
Batch, online and streaming inference, how to choose between them from freshness, latency and volume, the serving architecture, batching, and CPU against GPU.
Dataset → Model → Predictions, on a schedule. When the prediction can be precomputed, batch is the cheapest and most debuggable mode — and the staleness window is a property to design, not a defect.
Request → Features → Model → Prediction → Response, inside a latency budget. The feature fetch is usually the latency, and the timeout and fallback are part of the model's quality, not an infrastructure detail.
Continuous events drive predictions: the model sits inside a stream processor, features are state kept per key, and ordering, late events and where the model sits in the topology decide correctness more than the weights do.
How fresh must the prediction be, can it be precomputed, does it need live features, what is the latency budget, what is the volume — those five questions decide batch, online, streaming or hybrid. Online is often unnecessary, and the churn case shows why.
Client → Backend → Model Service → Artifact → Prediction. Where preprocessing runs, model-in-process against model-as-service, versioned endpoints, warm-up and health checks — and the line where the Backend domain takes over.
Individual requests are grouped into a batch before the accelerator sees them. Throughput rises because the hardware runs one large matrix multiply instead of many small ones; latency rises because every request waits for the batch. Dynamic batching with a maximum wait is the knob.
A workload decision: model size, available batch size, latency budget, cost per prediction and utilisation. A small tree model on CPU beats a GPU round-trip; a large transformer at volume does not fit on CPU. "GPU makes inference faster" is false as stated.