Inference Cost
Cost per prediction is hardware cost per hour divided by predictions per hour, plus feature fetch and storage. Utilisation is the lever, and the first question is whether the prediction needs this model at all.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does one prediction actually cost, which term dominates, and which of the cheaper options — simpler model, batching, caching, quantization, fewer retrains — would move it?
A product recommendation service costs more per month than the revenue it can be shown to influence. Finance wants the number per prediction and the engineering lead wants to know whether to quantize, batch, move to CPU, retrain less often, or replace the model with something simpler — and in what order.
Cost is the GPU bill. Reduce it by moving to cheaper GPUs or by quantizing so each GPU does more; the model and the retraining cadence are the ML team's business and stay as they are.
The GPU bill is a device price per hour divided over a trickle of predictions because the fleet is sized for peak and idles at night; quantization makes each idle device idle faster. Utilisation, not device price, is the term.
- The GPU bill is a device price per hour divided over a trickle of predictions because the fleet is sized for peak and idles at night; quantization makes each idle device idle faster. Utilisation, not device price, is the term.
- The feature fetch and prediction logging are a meaningful share of the per-prediction cost and are untouched by anything done to the model.
- Nightly retraining on a cluster costs more than serving and buys nothing measurable on most nights; the retraining cadence was a habit, not a decision (Retraining as a Decision).
- Most of the predictions are repeated for the same user and page within minutes; a short-lived cache would have served them at nearly no cost, with no quality change the user could see.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model ranks products for a user; the cost question's target is the marginal cost of one served prediction, decomposed into terms that can be reduced, against the marginal value that prediction produces.
- The decision is a portfolio of interventions ranked by cost saved per quality lost — and possibly the decision that some predictions should not be made at all.
- The service runs a mid-sized neural ranker on GPU instances that are provisioned for peak and idle most of the day. Each request fetches a few dozen features from an online store and logs the prediction.
- The model is retrained nightly on a GPU cluster whether or not the data changed materially; the training bill is a large share of the total.
- A large fraction of requests are for the same few thousand users on the same product pages, minutes apart, and are scored fresh every time.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Cost per prediction decomposes:
(hardware cost per hour ÷ predictions per hour) + feature fetch per prediction + logging and storage per prediction + amortised training cost per prediction. The first term is where utilisation lives: a device at low occupancy has the same hourly price and a fraction of the predictions per hour, so the per-prediction cost is inversely proportional to how full the device is. - Utilisation is raised by batching (Inference Batching), by consolidating models onto fewer devices, by autoscaling the fleet down when traffic falls (Throughput vs Latency), and by moving low-traffic models to hardware whose price scales in smaller steps (CPU or GPU for Inference). Quantization raises predictions per hour per device only when the device was the bound; on an idle device it raises nothing.
- The feature fetch and logging terms scale with request count, not model size, and are reduced by caching predictions for repeated inputs, by precomputing for users who can be scored in batch (Batch Inference), and by logging less or cheaper. Training cost is amortised over the predictions served between retrains, so retraining less often — when the data justifies it — lowers the per-prediction cost directly.
- The zeroth term is whether the model is needed for this prediction. A simpler model that captures most of the value at a fraction of the compute, a rule for the easy cases with the model reserved for the rest, or no prediction for requests where the outcome does not change the decision, all reduce cost without any of the above (When Not to Use ML).
The decomposition
Write the cost per prediction as a sum and fill in each term from billing and logs. The exercise usually surprises: the arithmetic the model performs is a small share, the idle time of the devices that perform it is a large one, and the training cluster that runs every night is larger still.
The table is for a peak-provisioned fleet serving a trickle at night, with a nightly retrain. The numbers are for the shape; the shape is common.
| Term | How it is computed | Illustrative share | Lever |
|---|---|---|---|
| Serving hardware | device cost per hour ÷ predictions per hour, at observed occupancy | large | utilisation: batch, consolidate, autoscale down, CPU for trickles |
| Feature fetch | online store cost ÷ requests, plus the cross-zone egress | moderate | cache repeated inputs; precompute for batchable users |
| Logging and storage | per-prediction log write and retention | small but request-scaled | sample, compress, shorten retention where safe |
| Training, amortised | cluster cost per retrain ÷ predictions served until the next retrain | large, and invisible in the serving bill | retrain when the data justifies it, not nightly |
| Model arithmetic | the share of device time spent in kernels | small at low occupancy | quantize — only once the device is the bound |
The cost questions, in order
The interventions have an order because each one changes what the next is worth. Whether the prediction is needed comes first, because it removes every downstream term. Retraining cadence comes next because it is often the largest term and has no serving-side dependency. Batching, precomputing and caching raise utilisation and cut request-scaled terms. Quantization is last because it only pays once the device is the bound.
Every step has a quality cost measured per segment. The ordering is by cost saved per quality lost, and that ratio has to be measured for this system, not assumed from a blog post.
Which term dominates the decomposition, and what does the product tolerate?
when A rule handles the easy cases; a segment's predictions earn less than they cost.
cost Route those requests to a rule, a cached default or no prediction; add a routing decision to monitor.
when Nightly retrains on a cluster with no measurable quality change most nights.
cost Retrain on a trigger — drift, decay, data volume — instead of a schedule; measure decay so the trigger is honest.
when Low occupancy, peak-provisioned fleet, trickle traffic off-peak.
cost Batch, consolidate models per device, autoscale on queue depth, move trickles to CPU; accept less surge headroom.
when High volume, many repeated inputs, expensive feature fetch.
cost Cache predictions with a TTL chosen as a freshness decision; precompute for users who can be batched; accept staleness for the cached share.
when High occupancy, the model's arithmetic is the largest term.
cost Quantize, then distil; spend a per-slice quality budget and keep the original as fallback.
Utilisation is the assumption
Every cost figure in the decomposition is a function of how full the devices are, and that number is set by traffic, batching and fleet size, none of which the model controls. A cost model computed at one utilisation is wrong at another; a fleet consolidated to raise occupancy is a fleet with less headroom for the surge that arrives with the next product launch.
So the per-prediction cost is an assumption with a monitor: occupancy stays where the model assumed, or the cost per prediction — and the case for every intervention ranked on it — has to be recomputed.
Device occupancy and predictions per device-hour stay within the range the cost decomposition was computed at, so the per-prediction hardware term is what the dashboard says.
holds when Batching fills at production arrival rates; autoscaling tracks traffic with adequate warm capacity; model consolidation holds; traffic shape has not changed.
breaks when Traffic falls or fragments across more models; a wait cap is tightened for latency; a consolidation is rolled back after an incident; a surge forces over-provisioning that is never scaled back.
respond Re-rank the interventions against the new dominant term; do not quantize an idle fleet or cache a fetch that is no longer the cost.
How to build it
Most important first.
- Measure the decomposition before touching anything: hardware cost per prediction at observed utilisation, feature fetch cost, logging cost, training cost per prediction served. The largest term is the first target and it is rarely the model's arithmetic.
- Ask the §101 questions in order: can a simpler model do it; can it be trained less often; can predictions be batched or precomputed; can repeated inputs be cached; can the model be quantized to raise predictions per device. Each is a cost lever with a quality cost that must be measured on slices.
- Raise utilisation before buying anything: batch, consolidate, autoscale down off-peak, and move trickle-traffic models to CPU. A cheaper device at the same occupancy is a small saving; the same device at higher occupancy is a large one.
- Put the cost per prediction on the same dashboard as the value per prediction, by segment; a segment whose predictions cost more than they earn is a segment to serve with a cheaper rung or not at all.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Cost per thousand predictions, decomposed by term, over time — the number that ranks the interventions and the one finance asked for.
- Device occupancy and predictions per device-hour, since the hardware term moves with these and not with the device's list price.
- Total monthly GPU spend is the number everyone has; it hides utilisation, training and the request-scaled terms, and cannot say what one prediction costs.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The cost decomposition is measured at production utilisation and kept current, so the largest term is known and interventions are ranked against it rather than against the device list price.
- Every cost intervention — caching, reduced retraining, quantization, a simpler model — has its quality effect measured per segment against the value that segment produces, and the net is positive.
- Utilisation stays in the range the cost model assumed; a traffic drop or a model consolidation that fails leaves devices idle and the per-prediction cost quietly multiplies.
- Offline: compute the decomposition from billing, request logs and training job records; replay a day of traffic through candidate interventions and cost each.
- Online: track cost per thousand predictions by term and device occupancy; canary each intervention with quality by segment alongside cost (Canary Rollout).
- Over time: re-run the decomposition quarterly and after every model, device or traffic change; the dominant term moves.
What can go wrong
- Caching predictions cuts cost and serves stale rankings after a user's behaviour changes; the cache TTL is a freshness decision that was made as a cost decision (Feature Freshness).
- Retraining is reduced to weekly to save cost and the model decays on a segment that moves daily; the saving is smaller than the lost value and nobody measured the second number (Performance Decay).
- The fleet is autoscaled aggressively off-peak, and the morning ramp arrives before new devices have loaded the model; the latency budget is blown daily at nine.
- Higher utilisation means less headroom: a fleet that is full at peak has nothing left for a surge, and the cost of the surge is a latency incident.
- Caching, precomputing and retraining less often each trade freshness for money, and freshness has a value that is harder to measure than the bill.
- A simpler model or a rule for easy cases lowers cost and adds a routing decision — which requests go where — that becomes its own thing to monitor.
- "A GPU automatically makes inference cheaper because it is faster." A GPU makes batched arithmetic cheaper per operation. At low occupancy it is the most expensive way to serve a prediction, and the occupancy is the number to look at.
- "Quantize first; it is the standard cost lever." It raises predictions per hour per device only when the device was the bound. On an idle fleet the lever is utilisation, and on a fetch-dominated cost the lever is caching; quantization spends quality for nothing there.
- "Retraining is an ML quality decision, not a cost one." It is both. Nightly retraining that does not move the quality metric is the single largest term in many budgets, and cutting it is a cost decision that needs a quality measurement to be made honestly.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe decomposition — hardware over predictions, plus request-scaled terms, plus amortised training — holds for every served model; the dominant term differs, and that is the point of computing it.
- SCALE-SPECIFICAt a trickle of requests the hardware term is idle-device cost and the answer is CPU or consolidation; at very high volume the request-scaled terms — fetch and logging — dominate and the answer is caching and precomputation; training amortises to nothing at high volume and dominates at low.
- SIMULATEDThe worked decomposition uses illustrative figures chosen to show the shape of a peak-provisioned GPU fleet with a nightly retrain; they were not measured on any real system, and your terms will rank differently.
Where the depth lives
This domain teaches the model and hands the rest off by name.