CI, CT and CD for ML
Three different loops with three different triggers. CI proves the code and data are sound; CT proves a new model can be trained; CD proves it is safe to serve. A green one proves only its own claim.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The CI pipeline is green, the model retrained overnight, and the deployment succeeded. Which of those three facts says the model in production is good — and which questions does each one actually answer?
We have a "CI/CD for ML" pipeline that a consultant set up. Every push runs tests, every night a retrain runs, and every retrain that finishes gets deployed. Last week a nightly retrain deployed a model that predicts the same score for everyone. All three stages were green. I need to explain to my director what "green" was actually checking.
Software has CI/CD; ML is software; so run the tests on push, train on a schedule, deploy what trains. If all three pass, the model is fine — that is what pipelines are for.
CI tested the encoder in isolation and passed. It did not test that the encoded feature still has more than one value on real data, because that is a data test, not a unit test (Data & Feature Tests).
- CI tested the encoder in isolation and passed. It did not test that the encoded feature still has more than one value on real data, because that is a data test, not a unit test (Data & Feature Tests).
- CT trained a model on the degenerate features and finished. It finished because training does not fail when the inputs carry no signal; it converges to the base rate and exits zero. There was no smoke test that the model's predictions vary (Training Smoke Tests).
- CD deployed the artifact because the artifact existed. There was no evaluation gate against the incumbent, no shadow, no canary — "deploy what trains" treated the existence of a file as evidence of quality.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The model scores leads for the sales team; the label is whether a lead converted within 30 days. The surrounding system's target is that each of the three loops answers its own question truthfully: CI that the code and data are sound, CT that training produced a plausible model, CD that the model is safe on live traffic.
- A repository with training code, feature definitions and serving code. A nightly extract of the last 90 days of leads with 30-day-old labels. The CI runs unit tests on the feature code; the nightly job trains on the extract and pushes the artifact; the deploy job swaps it in.
- The constant-score model came from a feature-definition change merged the day before: a categorical feature's encoding now emits a single value, so every lead looks identical. The unit tests for the encoder passed because they tested the function, not its output distribution.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- CI is triggered by a code change. It proves the code is sound: unit tests on feature logic, a data test that runs the feature pipeline on a fixed sample and checks the output distribution, a training smoke test on a tiny dataset that loss falls and predictions vary. It proves nothing about the current data or the current model.
- CT — continuous or triggered training — is triggered by a schedule, a data-volume threshold, a drift signal or a manual decision (Retraining Strategies). It proves that a new model can be trained from current data and that it beats the incumbent on the agreed evaluation, with its own gates: validation before training, evaluation after. It produces a registry candidate, not a deployment.
- CD is triggered by a candidate reaching the registry with a passing evaluation. It proves the candidate is safe to serve: a serving contract test, a shadow comparison on live traffic (Shadow Deployment), a canary with a rollback (Canary Rollout, Rollback & Fallback). Each loop has its own trigger, its own gates and its own claim.
Three loops, three triggers, three claims
The word "pipeline" hides the fact that three different things happen for three different reasons. Code changes when an engineer pushes; a model changes when data accumulates or drifts; a deployment changes when a candidate is judged safe. Collapsing them into one linear flow makes each step's success look like evidence for the next, which it is not.
The matrix states what each loop is triggered by, what it gates on, what it produces and — the column people skip — what a green result does not prove.
| Loop | Trigger | Gates | Produces | Green does not prove |
|---|---|---|---|---|
| CI | A code or feature-definition change | Unit tests; data and feature tests on a fixed sample; smoke training on tiny data; serving contract test | A mergeable commit | That current data is sound, or that any model is good |
| CT | Schedule, data volume, drift signal, or a manual decision | Data validation; training smoke test; evaluation vs incumbent on fresh time-based split and slices | A registry candidate with an evaluation record | That the candidate is safe on live traffic, or that it behaves under serving load |
| CD | A candidate with a passing evaluation record | Serving contract; shadow comparison; canary with rollback criteria | A serving deployment | That the model will stay good as data moves — that is monitoring |
The constant-score model passed all three
Walk the incident through the loops. The encoder change passed CI because CI had unit tests and no data test. CT trained on one-valued features and finished because training on no signal converges rather than crashes. CD deployed because the artifact existed. Every stage did exactly what it was built to do; none was built to ask the question that mattered.
The failure simulator at /ml/failures includes this shape — a collapsed prediction distribution — among its injected failures. The signals it produces are distinct from a drifted model's, which is why the signals are worth learning to read.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Encoder emits a single value for every row | Unit tests green; feature cardinality is one on real data | CI has no data test on feature output | Add a feature-distribution test on a maintained sample to CI; block the PR |
| Training on features with no signal | Loss barely moves from the base rate; predictions nearly identical | CT has no smoke test asserting prediction variance and loss improvement | Smoke test after training: prediction spread above a floor, loss below the constant predictor's |
| Candidate promoted without evaluation | Artifact deployed with no comparison to the incumbent | CD is triggered by artifact existence, not by a passing evaluation record | CD reads the registry; a candidate without an evaluation record is not deployable |
| Full traffic swap on deploy | A day of identical lead scores before anyone notices | No shadow or canary; monitoring watches latency only | Shadow compare on live traffic; canary with a prediction-distribution check; alert on collapse |
The smoke test that CT was missing
A training run fails loudly when it crashes and silently when it learns nothing. The smoke test asks the questions a crash would not: did the loss improve over the constant predictor, do the predictions vary, do a few known-ordered examples come out in order. It runs in CI on a tiny dataset to catch code regressions and in CT on the real run to catch data regressions.
It is deliberately weak — it will not catch a model that is subtly worse. That is the evaluation gate's job. The smoke test catches the model that is obviously broken, which is the one that gets deployed at three in the morning.
A candidate can reach serving only through CI-passing code, a CT run with a passing evaluation record, and CD gates that ran on it.
holds when CD reads the registry and refuses candidates without evaluation records; the registry stage change is the only trigger; manual overrides are logged and reviewed.
breaks when An engineer copies an artifact to the serving bucket under deadline; the CT job writes the registry record before evaluation "for tracking"; the evaluation threshold is lowered to unblock a release.
respond Roll back the bypassed model to the last gated one, then fix the bypass path — usually by removing write access to the serving bucket from humans.
1import numpy as np2 3def smoke_test(model, X_val, y_val, baseline_loss, ordered_pairs):4 p = model.predict_proba(X_val)[:, 1]5 problems = []6 # 1. predictions vary: a collapsed model scores everyone the same7 if np.std(p) < 0.02:8 problems.append(f"prediction spread {np.std(p):.3f} — model is near-constant")9 # 2. loss beat the constant predictor (the base rate)10 eps = 1e-911 loss = -np.mean(y_val * np.log(p + eps) + (1 - y_val) * np.log(1 - p + eps))12 if loss >= baseline_loss:13 problems.append(f"log loss {loss:.3f} not better than base-rate {baseline_loss:.3f}")14 # 3. known-ordered examples: an engaged lead should outscore a bounced one15 for easy, hard in ordered_pairs:16 if model.predict_proba(easy)[0, 1] <= model.predict_proba(hard)[0, 1]:17 problems.append("known-easy example did not outscore known-hard example")18 return problems # non-empty: do not register the candidateThe thresholds are loose on purpose. This test is not evaluation; it is the check that training did anything at all, and it should almost never fire — which is why it is the first thing removed and the one that would have caught this incident.
How to build it
Most important first.
- Separate the three loops with explicit triggers and explicit outputs: CI produces a passing commit, CT produces a registry candidate, CD produces a serving deployment. Never let one loop's success trigger the next without the next loop's own gates.
- Put the data test in CI: run the feature pipeline on a fixed sample and assert cardinality, null rate and range per feature. This is the test that would have caught the encoder change at the pull request.
- Put the smoke test in both CI (tiny data, seconds) and CT (real data, after training): loss fell, predictions vary, a known-easy example scores higher than a known-hard one (Model Invariant Tests).
- Gate CD on an evaluation record in the registry, then shadow, then canary. A candidate with no evaluation record is not deployable, however green CT was (Promotion Is a Checklist, Not a Score).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per loop, the fraction of runs blocked by its own gates. A CT loop that has never blocked a candidate has no gates.
- For CD, the canary's comparison metric against the incumbent — prediction distribution where labels are slow, conversion where they are fast — and the time to rollback when it fails.
- A green build badge is a fact about the last commit. It does not map to the model in production, which was trained by a different loop from different data.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Each loop's gates run on every execution and cannot be skipped by the previous loop's success; the bypass path is logged.
- The CI data sample and the CT evaluation set are refreshed on a schedule and their refresh is itself validated, so the tests measure current data.
- The evaluation in CT compares candidate and incumbent on the same fresh held-out data, not the candidate today against the incumbent's historical number.
- Offline: merge a deliberate encoder regression on a branch and confirm CI blocks it at the data test. Train on a shuffled-label dataset and confirm the CT smoke test blocks the candidate.
- Online: promote a candidate with a slightly degraded evaluation and confirm CD's shadow comparison flags it before the canary. If it reaches the canary, the shadow gate is decorative.
- Over time: audit which loop caught each real regression. Regressions caught only in production mean an earlier loop's gate is missing or stale.
What can go wrong
- CT is triggered by a drift alert, so every noisy drift alert triggers a retrain, and the retrain on drifted-but-unlabelled data is worse than the incumbent (Drift Is Not Failure).
- CD's evaluation gate compares against the incumbent's recorded metric from months ago rather than re-evaluating the incumbent on the same fresh data, so the comparison is between two different test sets.
- CI's data test uses a fixed sample that is refreshed by hand, and after a year it no longer resembles production; the test is green because the sample is old.
- Three loops with three sets of gates are three things to maintain, and each has expectations that go stale at different rates.
- A CT loop with a real evaluation gate produces candidates that do not deploy, and someone has to look at why; the "train and ship" simplicity is gone.
- Shadow and canary in CD need serving infrastructure that can run two models and route traffic, which is real work before the first safe deployment.
- "CI is green, so the model is fine." CI tested the code on a sample. The model in production came from CT on last night's data and was deployed by CD. Three different loops, three different claims.
- "We retrain nightly, so we have continuous training." A scheduled job that trains and deploys without gates is continuous deployment of untested models. CT is the gates, not the schedule.
- "Just deploy it and monitor." Monitoring is the last line. A constant-score model shows up in monitoring as a prediction-distribution collapse — after it has scored a day of leads.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe three loops and their distinct claims follow from what code, training and deployment each change, so the separation applies to any model family and any serving mode.
- SCALE-SPECIFICA team retraining quarterly can run CT by hand with a checklist and CD as a reviewed promotion; a team retraining daily needs all three automated, and the gates become the only thing standing between a bad extract and production.
- CONTESTEDA serious position holds that CT with automatic promotion is right for high-volume, fast-label systems such as ads ranking: labels arrive in hours, the canary is the evaluation, and a human gate adds latency without adding information. That is correct where labels are fast and the canary is well instrumented; where labels take weeks, automatic promotion deploys models nobody has evaluated on outcomes.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — the discipline of test pyramids, flaky-test hygiene and what a test is allowed to depend on decides whether the data test in CI stays green for the right reason.