ML Cost Optimisation
Before buying cheaper GPUs, ask whether a simpler model works, whether training can happen less often, whether inference can be batched, quantized, cached or precomputed. Utilisation is the lever.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The ML bill tripled and the first proposal is a cheaper GPU tier. Which questions come before that, and which number actually decides the bill?
Our ML spend went from a rounding error to the second-largest line in the infrastructure budget in a year. Finance wants a plan. The team's plan is to negotiate a GPU discount and move training to spot instances. I suspect we are optimising the price of things we should not be doing at all, and I want to know how to tell.
The bill is the price times the quantity. Reduce the price: cheaper instance tier, spot for training, a reserved-capacity discount. Everything else is the model team's business.
The discount cuts the unit price and the quantity keeps growing, because the quantity was never questioned. Daily retraining of a model whose data changes weekly is a sevenfold overspend that no discount touches (Retraining Strategies).
- The discount cuts the unit price and the quantity keeps growing, because the quantity was never questioned. Daily retraining of a model whose data changes weekly is a sevenfold overspend that no discount touches (Retraining Strategies).
- Spot instances cut the training price and the training job has no checkpointing, so a preemption at hour five restarts from zero. The effective price per completed run rises (Checkpointing).
- The GPU serving endpoint runs at a fixed replica count sized for peak, so its utilisation off-peak is a fraction of that, and the bill is for the idle hours. A cheaper tier makes the idle hours cheaper; it does not make them fewer (Inference Cost).
- Nobody asked whether the transformer's click-through gain over matrix factorisation was worth its cost. It may be; the decision was never framed that way.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The models predict product recommendations and a fraud score. The cost target is total spend for the same product outcome — the same recall at the operating threshold, the same recommendation click-through — and the decision is where in the system a unit of spend produces the least outcome.
- A cost breakdown by job: recommendation model retrained daily on GPUs, fraud model retrained weekly, a GPU serving endpoint for recommendations at a fixed replica count, a CPU endpoint for fraud. GPU utilisation during training is unmeasured; serving GPU utilisation is measured and low outside peak hours.
- The recommendation model was moved from a matrix factorisation model to a transformer-based model last year. The click-through improvement was measured in an A/B test; the cost difference was not part of the decision.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- ML cost is a product of four things: how often you train, how much each training run costs, how many predictions you serve, and how much each prediction costs. Each is a lever, and the price of compute is a multiplier on all of them — the last thing to optimise, not the first.
- Utilisation decides the real cost. A GPU billed per hour that is busy a third of the time costs three times its price per useful hour. Training utilisation is lost to data loading and small batches; serving utilisation is lost to fixed replicas at variable load and to batch sizes of one (Inference Batching, Throughput vs Latency).
- The questions come in order of leverage. Can a simpler model produce the same outcome (Baselines Are Mandatory, Which Model Should We Use?)? Can training happen less often, triggered by data change rather than by the calendar? Can inference be batched or precomputed instead of served online (Choosing the Inference Mode, Batch Inference)? Can the model be quantized or distilled to a cheaper footprint (Quantization, Pruning & Distillation)? Can results be cached for repeated inputs? Only then: can the compute be cheaper?
The questions before the price
The levers are ordered by how much they can move the bill and by how little they touch the outcome. A simpler model that matches the outcome removes a whole cost centre. Training less often divides a cost by a small integer. Batching and precomputation multiply serving utilisation. Quantization and distillation shrink the footprint. Caching removes repeated work. The compute price comes last because it multiplies whatever is left.
The decision device asks them in that order. A team that starts at the bottom — spot instances and discounts — saves a fraction of a quantity it never questioned.
Which lever to pull first on a model whose cost has become a problem
when A baseline or the previous model family gets close to the same outcome per the evaluation and slices, and the expensive model's gain was never priced
cost An experiment to rerun the comparison on current data; possibly a measured loss of outcome that must be weighed against the saving
when Retraining is on a calendar and the data or the model's decay does not move at that cadence
cost A retraining policy with evidence triggers, and a monitor for decay so that "less often" is not "too late"
when Predictions are consumed later than they are produced, or the input set is enumerable in advance
cost Freshness; a batch pipeline; storage for precomputed results; a fallback for inputs the batch missed
when Serving cost dominates, the model is large, and quality on critical slices can be measured before and after
cost A quality regression that may be uneven across slices; a second artifact to maintain
when A meaningful share of requests repeat within the window where the prediction is still valid
cost Staleness; cache invalidation tied to feature updates; memory
when GPU busy time is a small fraction of billed time in training or serving
cost Autoscaling on the right signal, dynamic batching, data-loader work — engineering rather than a purchase
when The quantities above are already justified, and checkpointing exists for preemptible training
cost Preemption handling; capacity risk near deadlines; a discount that commits to a volume
Utilisation is the number the bill hides
A GPU is billed by the hour and useful by the operation. The gap between the two is utilisation, and it is where most ML spend goes. In training, the GPU waits on data loading and on small batches. In serving, a fixed replica count sized for peak idles off-peak, and a request at a time keeps a device built for thousands of parallel operations mostly dark.
The offline/online device here is not a model metric. It is the load test against the production bill, and the gap has the same shape as every other in the domain: the staging number was true about a system that does not exist at production load.
Cost model built from the load test: per-request GPU time times expected request volume, assuming the device is busy whenever a request is in flight.
The monthly bill several times the estimate; utilisation dashboards showing the GPU mostly idle outside two peak hours, and batch size one during them.
- 1Replicas were fixed at the peak count, so off-peak hours were billed at full price for a nearly idle device.
- 2Requests were served one at a time; the load test's per-request GPU time assumed the device was saturated, and it never was.
- 3The precomputable share of recommendations — users whose candidates change daily, not per request — was served online anyway.
Spot training that actually saves money
Preemptible compute is the one price lever that is usually worth pulling, and the one that goes wrong most predictably. A training job that restarts from zero on preemption pays for every interrupted hour twice. The prerequisite is checkpointing: model, optimizer state and step, written often enough that an interruption loses minutes, and a resume path exercised in the pipeline rather than at three in the morning.
The assumption device names what has to stay true. It is a good example of the domain's general rule that a cost optimisation is a set of assumptions about the workload, and the workload moves.
A training run on preemptible compute resumes from its last checkpoint after interruption and finishes within the deploy window at lower total cost than on-demand.
holds when Checkpoints include model, optimizer and data-loader position; the interval is short relative to the run; the resume path is tested per release; preemption rates in the region are moderate.
breaks when A dependency upgrade changes the optimizer state format; the checkpoint interval is lengthened to save storage; a capacity crunch preempts every attempt before the next checkpoint; the run must finish by a deadline that a single wave of preemptions blows.
respond Fall back to on-demand for that run rather than retrying spot indefinitely, and fix the checkpoint path before the next scheduled run.
1def cost_per_outcome(billing, runs, outcomes, model):2 # billing: rows of (job_id, hours, rate); runs: job_id -> model, stage; outcomes: model -> count3 spend = {"training": 0.0, "serving": 0.0, "features": 0.0}4 for job_id, hours, rate in billing:5 run = runs.get(job_id)6 if run and run.model == model:7 spend[run.stage] += hours * rate8 total = sum(spend.values())9 n = outcomes[model] # e.g. fraud cases caught at the operating threshold10 return {11 "total": total,12 "by_stage": spend,13 "per_outcome": total / n if n else float("inf"),14 # the price is a multiplier; utilisation is the lever15 "training_utilisation": runs.gpu_busy_hours(model, "training") / runs.billed_hours(model, "training"),16 }The join from billing to run records is the hard part and the reason this is rarely done. Without it the bill is one number and every decision about it is a guess.
How to build it
Most important first.
- Attribute cost per model per stage — training, serving, feature computation — and per outcome. A cost per thousand predictions, and a cost per retrain, are the numbers the decisions need (Training Cost).
- Ask the simpler-model question with an experiment, not an opinion: rerun the matrix factorisation baseline against the transformer on the current data and compare outcome per unit cost. Keep the expensive model only where it wins by enough.
- Move training triggers from schedule to evidence — data volume, drift reviewed by a person, a measured decay — and add checkpointing before adopting spot (Retraining as a Decision).
- For serving, raise utilisation before lowering price: batch the recommendation inference and precompute for the users who will see it, autoscale the endpoint on the right signal, quantize where the quality cost is measured and acceptable. The decision device below is the order.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Cost per unit of outcome: per fraud case caught at the operating threshold, per incremental click. This is the number that decides whether a model earns its bill.
- GPU utilisation during training and serving, as a fraction of billed hours. This is the lever; the price is a multiplier.
- Total ML spend is the number finance sees and the wrong one to optimise directly: it can fall while cost per outcome rises, by cutting a model that was earning its keep.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Cost is attributed per model and per stage, and the attribution is checked against the bill, so the levers are chosen on measured numbers.
- Every cost reduction has a measured quality effect on the outcome metric and on the critical slices, and the reduction is reverted if the effect exceeds the agreed budget.
- Training on preemptible compute resumes from checkpoints, and the resume path is exercised in the pipeline rather than assumed.
- Offline: for each proposed lever, run the outcome comparison — simpler model versus current, quantized versus full precision — on the current evaluation set and slices, and record cost per outcome for both.
- Online: after a serving change, compare utilisation and tail latency before and after at peak, and the outcome metric in a holdout.
- Over time: report cost per outcome monthly per model. A model whose cost per outcome rises while the bill is flat is being starved; one whose bill rises while cost per outcome is flat is growing and fine.
What can go wrong
- Quantization is applied without a quality check on the slices that matter, and the fraud model's recall on a small merchant category drops while the aggregate holds (Evaluation Slices).
- Precomputed recommendations go stale for the users whose behaviour changed since the batch, and the click-through gain the transformer earned is lost to freshness rather than to the model (Feature Freshness).
- Spot training with checkpointing works until a region-wide preemption wave leaves the retrain unfinished before the deploy window, and the fallback is the old model for another week.
- Cost attribution per model is engineering work — tagging jobs, joining billing exports to run records — before any saving appears.
- Simpler models and less frequent training can cost outcome; the whole point of measuring cost per outcome is that sometimes the expensive thing wins.
- Batching and precomputation trade freshness and latency for utilisation, which is the right trade for some products and the wrong one for others.
- "Get a GPU discount." The price is a multiplier on quantities nobody has questioned. Daily retraining of a weekly-changing model is the same overspend at any discount.
- "GPU automatically makes inference faster, so move everything to GPU." A GPU at batch size one and low utilisation is slower per dollar than a CPU for many models; utilisation is the lever (CPU or GPU for Inference).
- "Cut the most expensive model." The most expensive model may have the lowest cost per outcome. Cut the one that earns the least per unit spent, which requires attributing outcome as well as cost.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat cost is frequency times unit cost across training and serving, and that utilisation multiplies the price, holds for every model and provider; the ordering of the questions is general even though the answers are not.
- SCALE-SPECIFICAt low volume the dominant cost is engineer time and any optimisation beyond "train less often" is premature; at high volume the serving levers — batching, precomputation, quantization — dominate and pay for the engineering that measures them.
- CONTESTEDA serious position holds that a bigger model is often the cheapest path to an outcome once engineering time is counted: a transformer that lifts click-through beats a quarter of feature engineering on a simpler model, and compute is cheaper than people. That is frequently right at the model-choice stage; the reply is that the comparison must include serving cost at production volume, which the A/B test that chose the model did not.
- SIMULATEDThe relative costs and utilisation fractions in the illustrations are for the shape of the argument; a real cost decision needs the team's own billing export joined to its run records.
Where the depth lives
This domain teaches the model and hands the rest off by name.