Training Smoke Tests
On a tiny dataset, in CI, in minutes: the pipeline runs end to end, the loss goes down, an artifact appears, and the model can memorise a handful of examples. A pipeline that cannot overfit ten rows is broken, whatever the full run reports.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
How do you test a training pipeline on every commit without running the training — and what does "the model can overfit ten examples" prove?
A vision team's weekly training job produced a model with the same validation metric for three weeks running, to four decimal places. It turned out a refactor two weeks earlier had frozen the backbone by mistake and the head was learning nothing; the metric was the pretrained model's. No test had failed, because the training pipeline had no tests — it took nine hours to run.
Training is too slow to test in CI. Test the components — the loader, the loss function, the export — with unit tests, and rely on the weekly run's validation metric to catch anything the units miss.
The frozen backbone was not in any component. Each component worked; the composition — an optimizer constructed before the model's parameters were unfrozen — was the bug. Only running training end to end would have shown a loss that did not move.
- The frozen backbone was not in any component. Each component worked; the composition — an optimizer constructed before the model's parameters were unfrozen — was the bug. Only running training end to end would have shown a loss that did not move.
- The validation metric did catch it, in the sense that it was suspiciously constant — but a constant metric looks like a stable model, and nobody reads four decimal places every week. The signal was there and it was not a test.
- A second bug found later: the augmentation pipeline was applied to validation images too, and the validation metric had been slightly wrong for months. A smoke test that asserts validation runs without augmentation is one line; no unit test of the augmentation function could have expressed it.
- A third: the artifact export dropped the class-index mapping, so the served model returned the right index and the wrong label. Nine hours of training to discover it each time; a two-minute smoke test would have exported and reloaded the artifact on every commit.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model classifies product images into a taxonomy (Classification). The target of this lesson is the pipeline's ability to *learn* — a property of the code, checkable on every commit, independent of whether the full run produces a good model.
- "Can learn" is defined operationally: given a small set of examples, the loss decreases and the model reaches near-zero training error on them. A model that cannot memorise ten examples has a bug between the data and the gradient.
- The full training set is millions of images on object storage. The smoke test uses a fixed, versioned set of a few dozen images across every class, checked into the repository or a small fixture bucket.
- The smoke set is chosen for coverage of the code paths — every class, an image with an unusual aspect ratio, a grayscale one — not for representativeness of the distribution.
- The pipeline is the same code the full job runs: the loader, the augmentation, the model, the loss, the optimizer, the checkpoint writer and the artifact export.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A training smoke test runs the real pipeline on a tiny dataset with a tiny model configuration and asserts a small number of things that any working pipeline must satisfy: it runs to completion; the training loss at the end is lower than at the start; an artifact is written, and reloading it and predicting on the smoke set reproduces the in-memory model's outputs; and, with enough steps, the model reaches near-perfect accuracy on the smoke set — it can overfit.
- The overfitting test is the sharp one. A neural network of any reasonable capacity can memorise a handful of examples, and gradient descent will get it there if the gradient reaches the parameters. If it does not — the parameters are frozen, the learning rate is zero, the labels are shuffled relative to the inputs, the loss ignores its input, the gradient is clipped to nothing, the data loader yields the same batch every step — the model cannot memorise, and the test fails. It is a test of the whole path from data to update (Backpropagation).
- Loss decreasing is weaker than overfitting but catches a different class: a pipeline that learns slowly, or whose learning rate schedule is wrong at the start, may not reach zero error in the budget but still show movement. Both assertions belong in the test.
- The test runs in minutes because the data is tiny and the model configuration is small — fewer layers, lower resolution, a handful of steps. It runs the same code as production with different configuration, which is the point: the configuration is a parameter, and the code path is identical (CI, CT and CD for ML distinguishes this from continuous training, which runs the real job on the real data).
Four assertions in two minutes
The smoke test asks the pipeline the questions a working pipeline cannot fail. Does it run? Does the loss fall? Does the artifact it writes, reloaded, predict what the in-memory model predicts? Can it memorise the handful of examples it was given? Each is a property of the code path, not of the data, and each is checkable in the time it takes CI to build the container.
The last one is the test with teeth. Memorising ten examples is trivially within the capacity of any network, and gradient descent will do it — if the gradient reaches the parameters, if the labels line up with the inputs, if the learning rate is nonzero, if the loader is yielding different batches. Every one of those is a bug that has shipped, and the overfit test fails on all of them.
1def test_pipeline_can_learn(tmp_path):2 cfg = smoke_config(data="fixtures/smoke", width=8, depth=2, steps=200, res=32)3 run = train(cfg, out_dir=tmp_path) # the production entry point4 5 assert run.completed6 assert run.loss[-1] < 0.5 * run.loss[0] # learning is happening7 assert run.trainable_params == expected_trainable(cfg) # nothing frozen by accident8 9 acc = accuracy(run.model, smoke_set())10 assert acc > 0.95 # it can memorise 40 examples11 12 art = load_artifact(tmp_path / "model") # export round-trips13 assert allclose(art.predict(smoke_inputs()), run.model.predict(smoke_inputs()))14 assert art.class_mapping == smoke_class_mapping() # the incident from last quarter15 16 val = evaluate(run.model, smoke_set(), augment=False)17 assert run.validation_augmented is False # the other incidentTwo of the assertions are incidents encoded as tests. That is the pattern: every training bug that reached production becomes a one-line assertion in the smoke test, and the test grows into the team's memory of what has gone wrong.
What overfitting ten examples proves, and what it does not
Reaching near-perfect accuracy on the smoke set proves the entire path from data to update is connected: the loader delivers inputs and matching labels, the forward pass produces a loss that depends on them, the backward pass produces gradients that reach every trainable parameter, and the optimizer applies them. Break any link and the model cannot memorise.
It proves nothing about the model that the full run will produce. The smoke set is not a sample of the distribution; the smoke configuration is not the production model; the accuracy is training accuracy on memorised examples. A team that reports it as a metric has misread a plumbing test as a model evaluation (Overfitting is the concept, deliberately induced here).
Each component passes; the composition — optimizer built before unfreezing, augmentation on the validation path, export without the class mapping — is untested until a nine-hour run produces a number somebody may or may not read.
The composition runs end to end in minutes; a loss that does not fall, a parameter count that is wrong or an export that does not round-trip fails the build before merge.
Training bugs live in the composition, and the composition is only tested by running it. Scale is a configuration parameter, so the test can run the real code on tiny data and still cover the path.
Keeping the smoke test honest
The test degrades in two directions. The budget gets trimmed to keep CI fast until the overfit assertion is flaky, then the threshold is loosened until it cannot fail. Or the smoke set drifts — a class dropped, an imbalance introduced — until a majority-class predictor passes the threshold. Both leave a green check that tests nothing.
The defence is the same as for any test: encode the assumptions and check them. The smoke set is versioned and balanced by assertion; the threshold has margin measured on a working pipeline; and the test is periodically broken on purpose to confirm it still fails.
The smoke test exercises the production code path on a balanced, versioned smoke set with a step budget and threshold that a working pipeline passes deterministically and a broken one cannot.
holds when The test calls the production entry point with a smoke config; the smoke set asserts its own class balance; the threshold was set with margin against several runs; a deliberate-break drill confirms failure for each classic bug.
breaks when A simplified test script diverges from production; the budget is trimmed below the margin; the smoke set is edited; the threshold is loosened after a flaky failure rather than the flakiness fixed.
respond Restore the margin — more steps, a smaller model, a fixed seed — rather than lowering the bar; and add the incident that revealed the gap as a new assertion.
How to build it
Most important first.
- Make the pipeline configurable down to a smoke size: dataset path, model width and depth, image resolution, step count. The smoke test is the production entry point with a smoke config, never a separate script.
- Fix and version the smoke set for code-path coverage — every class, every edge case the loader handles — and keep it out of the training data.
- Assert in order: runs without error; final loss below initial loss; artifact exported and reloaded produces the same predictions as the in-memory model; training accuracy on the smoke set above a near-perfect threshold after a fixed step budget.
- Add the assertions that encode past incidents: validation path has no augmentation; the artifact carries the class mapping; the number of trainable parameters equals the expected count.
- Run it on every commit and on every dependency bump, and fail the build on it. A smoke test that is advisory is documentation.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Pass or fail, on every commit, in under a few minutes. The time bound is part of the design; a smoke test that grows to twenty minutes stops being run.
- Final training accuracy on the smoke set after the fixed budget — the number that says the gradient reaches the parameters.
- Loss ratio, final over initial, as the weaker signal that learning is happening at all.
- The smoke run's validation metric means nothing and should not be reported. The smoke set is not representative and is not meant to be.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The smoke test runs the same code path as the production training job, differing only in configuration, so a pass says something about the job that will run on the full data.
- The smoke set stays small, balanced, versioned and covers every class and loader edge case, so the overfit assertion cannot be satisfied by a degenerate predictor.
- The step budget and threshold have enough margin that a working pipeline passes deterministically, so a failure is a bug rather than noise.
- Offline: break the pipeline deliberately in each of the classic ways — freeze the parameters, zero the learning rate, shuffle labels against inputs, drop the class mapping from the export — and confirm the smoke test fails for each.
- On every commit: the test runs in CI as a required check; the build cannot merge on a failure.
- Over time: track the test's wall-clock and its final accuracy across commits; a creeping duration or a falling accuracy at the same budget is a pipeline change worth a look.
What can go wrong
- The smoke test uses a separate, simplified script rather than the production entry point, and the production bug lives in the code the smoke script skips.
- The overfit threshold is set at a level a broken pipeline can reach by predicting the majority class of an unbalanced smoke set; the smoke set must be balanced or the threshold set against a per-class metric.
- The step budget is trimmed to keep CI fast and the test becomes flaky at the threshold, then is loosened until it cannot fail.
- The smoke set ends up in the training data through a fixture directory that a glob picks up, so the model has seen it — harmless for the smoke test itself, and a leak for any evaluation that reuses the set.
- The pipeline must be parameterised down to smoke scale, which is a refactor for a job written as a monolithic script with hard-coded paths and sizes.
- A few minutes on every commit is a real CI cost across a team, and the temptation to trim the budget until the test is meaningless is constant.
- The smoke test proves the pipeline can learn; it says nothing about whether the full model is good, and a team that reads a green smoke test as a model test has replaced one gap with another.
- "Training takes hours; it cannot be tested in CI." The full run cannot. The pipeline can, at smoke scale, in minutes, and the bugs that cost the most — frozen parameters, wrong labels, a broken export — show at any scale.
- "The model reached perfect accuracy on the smoke set, so it works." It proves the gradient reaches the parameters and the export round-trips. It proves nothing about generalisation; the smoke set is ten examples the model memorised on purpose.
- "We unit test the loss and the loader, so training is tested." The frozen backbone was in neither. The composition is where training bugs live, and only running the composition tests it.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe overfit assertion holds for any model trained by gradient descent with enough capacity to memorise a handful of examples; it applies unchanged to tabular networks, vision and language models, and in a weaker form — loss decreases, artifact round-trips — to tree ensembles and linear models.
- MODEL-SPECIFICFor models with a hard capacity limit — a heavily regularised linear model, a depth-limited tree — near-perfect training accuracy on the smoke set is not guaranteed and the assertion should be loss-decreases plus a training accuracy well above the majority-class rate rather than near one.
- FRAMEWORK-SPECIFICThe specific failure of an optimizer constructed before parameters are unfrozen is a property of frameworks where the optimizer captures a parameter list at construction; other frameworks fail differently, but every framework has a composition bug that only an end-to-end run exposes.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — a smoke test is an integration test with a deliberately small fixture, and the discipline of keeping its budget, threshold and fixture honest is ordinary test hygiene this lesson relies on.