Training Cost
A training run costs GPU-hours, CPU-hours, storage, network and — the multiplier that dominates — the number of times you run it. A hyperparameter search turns one run's cost into a bill, and most of the questions that reduce it are not about the hardware.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does a training run actually cost, what multiplies that cost, and which questions should be asked before buying more GPUs?
A finance director has a cloud bill line called "ML training" that tripled in a quarter and asks the team what it buys. The team knows the per-hour GPU price and the length of the main run. Neither number explains the bill.
Cost is the GPU price per hour times the hours of the production training run. The rest is rounding.
The production run is one run. The tracker shows a hundred and forty runs in the quarter — a grid search over six hyperparameters, repeated when the feature set changed, repeated again for three seeds each. The production run is under two percent of the GPU-hours.
- The production run is one run. The tracker shows a hundred and forty runs in the quarter — a grid search over six hyperparameters, repeated when the feature set changed, repeated again for three seeds each. The production run is under two percent of the GPU-hours.
- CPU preprocessing of the raw logs into training tables runs before every one of those runs, from scratch, because the intermediate tables were never cached. It is the second largest line.
- Each run writes hourly checkpoints to object storage and nobody deletes them. Storage for dead runs' checkpoints now exceeds the storage for the datasets.
- The training cluster sits in a different region from the data. Every run pulls the dataset across regions; egress is a line the team did not know existed.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding system is a set of models for a lending product (Designing a Churn Prediction System and its neighbours). This lesson's target is the total cost of producing and maintaining those models, attributed to the decisions that drive it.
- The decision it informs is whether the next increment of cost — a larger model, a wider search, a more frequent retrain, more GPUs — buys a product improvement worth its price.
- The cloud bill by resource: GPU instance hours, CPU instance hours for preprocessing and evaluation, object storage for datasets, checkpoints and artifacts, and network egress between regions and to storage.
- The experiment tracker: every run with its configuration, duration, hardware and outcome (Experiment Tracking). It is the only record of how many runs there were and why.
- The retraining schedule: which models retrain how often, on how much data, with what search around them.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Training cost has five components. GPU-hours: the accelerator time, priced per hour and by type, and multiplied by the worker count for a distributed run. CPU-hours: preprocessing, feature computation, evaluation, data loading. Storage: datasets, intermediate tables, checkpoints, artifacts, tracked over their retention. Network: egress across regions, to and from storage, and the interconnect a distributed run leans on. Experiment count: the number of times the previous four are paid.
- The experiment count is the multiplier. A hyperparameter search over a grid of g configurations at s seeds runs the pipeline g·s times; a search repeated for every feature change or data refresh multiplies again. The per-run cost is what people quote; the search is what they pay (The Tuning Budget).
- Distribution multiplies GPU-hours by the worker count and divides wall-clock by the scaling efficiency, so a run that is twice as fast on four workers costs twice as much per epoch. It is a purchase of calendar time, not a saving (Distributed Training).
- Most of the cost is decided before the hardware is chosen: by whether a simpler model would do, how often retraining is actually needed, whether the search is bounded, whether intermediate data is cached and whether the training data is smaller than the raw data. The GPU price is the last factor and the one with the least leverage.
Five lines, one multiplier
The bill has five components and one of them multiplies the other four. GPU-hours, CPU-hours, storage and network are per-run costs. The experiment count is how many times they are paid, and a hyperparameter search is the ordinary way that count reaches the hundreds: a grid of configurations, each at several seeds, re-run when the features or the data change.
The illustrative quarter below is the shape of most bills once anyone breaks them down: the production run is a small slice; the search is most of the GPU line; preprocessing that could have been cached is a large CPU line; checkpoints from dead runs are a storage line that only grows.
| Line | Production runs (illustrative) | Search & experiments (illustrative) | Lever |
|---|---|---|---|
| GPU-hours | 1 run × 48 h × 4 GPUs = 192 | 140 runs × 12 h × 4 GPUs = 6,720 | bound the search; early stopping; simpler model |
| CPU-hours (preprocessing) | 1 × 20 h | 140 × 20 h = 2,800 | cache versioned intermediate tables |
| Storage (checkpoints) | last 3 kept | 140 runs × 12 hourly checkpoints, never deleted | retention policy by run status |
| Network (egress) | 1 dataset pull across regions | 140 dataset pulls across regions | co-locate compute with data |
The questions that come before the hardware
Buying cheaper or faster GPUs changes the smallest factor. The factors that move the bill are decisions about the model and the process: is there a simpler model that meets the metric; does the model need retraining weekly, or would monthly with a drift trigger do (Retraining Strategies); can the training data be a sample; can the serving cost be batched, quantized or cached so the model itself can be smaller.
These are also the questions a cost review should ask first, because each has a large effect and none of them requires infrastructure. The hardware question — which card, preemptible or on-demand, how many — is worth asking after them, with checkpointing in place so preemptible capacity is actually cheaper (Checkpointing).
What would reduce this bill the most?
when The baseline or a smaller model is within the product's tolerance of the large one on the metric that matters (Beating the Baseline).
cost A possibly smaller ceiling on quality; the comparison itself is a run.
when Retraining is on a calendar rather than triggered by measured decay; performance between retrains is flat.
cost A drift monitor and a trigger policy that someone must own (Performance Decay).
when Searches are grid-shaped, re-run per feature change, or unbounded.
cost A chance of missing a configuration a wider search would have found; early stopping needs a defensible stopping rule.
when Preprocessing re-runs from raw per experiment; data and compute are in different regions.
cost Cache invalidation discipline and a versioned intermediate; a one-time migration.
when The training cost is driven by a large model chosen for serving quality that a compressed or batched deployment could match (Quantization, Batch Inference).
cost A quality check for the compressed model and a serving change.
when All of the above are done and the run is compute-bound.
cost Preemptible capacity needs checkpointing; a different card needs re-profiling.
What must stay true for the bill to stay explained
A cost that can be attributed can be reduced; one that cannot is renegotiated with finance every quarter. Attribution depends on the tracker recording every run with its hardware and purpose, on the caches and retention rules being enforced, and on the search and retrain decisions having owners.
Each of those degrades quietly. Untracked runs appear when someone launches from a notebook; the cache goes stale when a transformation changes; the retention job is disabled during an incident and not re-enabled. The bill is the monitor of last resort, and it arrives a month late.
Every GPU-hour on the bill maps to a recorded run with a purpose, and the search size, retrain cadence and retention policy are deliberate and owned.
holds when Launches go through a tracked path that records hardware and duration; monthly reconciliation of tracker hours against billed hours is clean; retention and cache invalidation run by rule.
breaks when Ad-hoc launches bypass the tracker; the reconciliation is skipped; the retention job stops; a search is widened "just this once" and the new width becomes the default.
respond Close the untracked path first. Then review the search and cadence decisions with their owners against cost per unit of improvement.
1def search_cost(per_run_gpu_hours, gpus, price_per_gpu_hour,2 configs, seeds, reruns_per_quarter,3 cpu_hours_per_run=0.0, cpu_price=0.0):4 runs = configs * seeds * reruns_per_quarter5 gpu = runs * per_run_gpu_hours * gpus * price_per_gpu_hour6 cpu = runs * cpu_hours_per_run * cpu_price7 return runs, gpu + cpu8 9# The production run is (1, per_run_gpu_hours * gpus * price).10# Everything that reduces 'runs' — random search with a budget,11# early stopping, not re-running per feature change — moves the bill12# more than any change to price_per_gpu_hour.The function is trivial and the point is that nobody writes it. The runs term is the one a team controls entirely and the one that finance never sees, because it lives in the experiment tracker rather than the bill.
How to build it
Most important first.
- Attribute cost per run and per model in the experiment tracker — hardware, hours, storage written — so the bill can be read as a list of decisions rather than a total (Cost Attribution on the data side is the same discipline).
- Bound the search. Random search with a fixed budget, early stopping of unpromising configurations, and a written rule for when a search is re-run — not every time a feature changes (Grid Search and Random Search).
- Cache the preprocessed training tables and version them; recompute only when the transformation or the raw data changes (Dataset Versioning).
- Set a retention policy for checkpoints — keep the last few of a live run, none of a dead one — and for old artifacts (Artifact Retention on the DevOps side).
- Put the training compute in the data's region, and use cheaper preemptible capacity with Checkpointing for runs that can tolerate interruption. Ask the §101 questions before the hardware question: simpler model, train less often, batch instead of online, quantize, cache.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Cost per model per month, broken into GPU, CPU, storage, network and attributed to production runs versus search. The split says what to change; the total does not.
- Cost per unit of product improvement — GPU-hours spent per point of the business metric moved — across the last several searches. When it climbs, the search has saturated and the next increment is expensive.
- Checkpoint and artifact storage over time by run status. Growth in dead runs' storage is pure waste.
- GPU price per hour is not the number to optimise. A cheaper card that halves the utilisation costs the same and takes longer.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every run is recorded with its hardware, duration and purpose, so the experiment count and its cost are known rather than reconstructed from the bill.
- The retrain cadence and the search size are decisions with owners and written justification, and they are revisited when the cost per unit of improvement rises.
- Cached intermediates and retained checkpoints are invalidated and expired by rule, so a saving in recompute is not silently a staleness bug or a storage leak.
- Offline: reconcile the tracker's recorded runs against the bill's GPU-hours each month; unexplained hours are untracked runs.
- Before a search: estimate its cost from the per-run cost and the configuration count, and require the estimate to be approved against the expected improvement.
- Over time: trend cost per model and cost per unit of improvement, and review the retrain cadence and search size whenever the trend breaks.
What can go wrong
- Cost attribution is added and the search budget is cut, so the team stops exploring and the models stagnate; the saving is visible and the foregone improvement is not.
- Preemptible capacity is adopted without working checkpointing, and the restart-from-scratch runs cost more than on-demand would have.
- The cache of preprocessed tables goes stale relative to a changed transformation, and every run trains on old features while the bill goes down and the model quietly degrades.
- A cost dashboard shows GPU-hours only, and the team optimises GPU-hours while CPU preprocessing and cross-region egress grow unwatched.
- Bounding the search caps the cost and caps the chance of finding a better configuration; the bound has to be set knowing what a missed improvement is worth.
- Preemptible capacity is cheaper per hour and demands checkpointing, resumption logic and a tolerance for unpredictable wall-clock.
- Cost attribution is instrumentation and discipline that a small team may reasonably decide is not worth it — until the bill line triples.
- "Training cost is the GPU price times the production run's hours." The production run is one of many. The search, the preprocessing, the storage and the network are the bill.
- "Distributed training will reduce cost." It reduces wall-clock at a cost multiplied by the worker count and divided by the scaling efficiency. It buys time, and it is only cheaper if the calendar was the expensive thing.
- "Cheaper GPUs will fix it." The hardware price is the last factor. A simpler model, a bounded search, a cached preprocessing step and a sane retrain cadence each move the bill by more.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- SIMULATEDThe cost figures in the matrix below are illustrative values chosen for the shape of the argument — the search dominating the production run — and are not measurements from any real bill, cloud provider or model; the ratios, not the numbers, are the lesson.
- SCALE-SPECIFICFor a team training small models on CPU the whole bill is engineer time and this lesson is moot; it applies once GPU-hours become a visible line, and the balance between search cost and production cost shifts again at foundation-model scale where a single run dominates everything.
- CONTESTEDA serious position holds that tight cost attribution and search budgets are a false economy for a research-stage team: the most valuable model improvements came from unbudgeted exploration, and the instrumentation and approval process slows the loop that finds them. The counter is that unbounded search mostly re-finds what a random search with early stopping would have found at a fraction of the cost, and that a team which cannot say what a run costs cannot say whether it was worth running.
Where the depth lives
This domain teaches the model and hands the rest off by name.