The ML Pipeline
Raw data → dataset → features → split → model → training → evaluation → artifact → deployment → inference → feedback. Eleven stages, each a place an assumption enters, and the loop back is what makes it a system rather than a script.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What are the stages between an event happening in the world and a model influencing the next such event, and what does each stage decide that the model can no longer change?
A new engineer asks how the fraud model "works" and gets a description of gradient boosting. Two weeks later they change a feature pipeline and the model degrades, and nobody can explain the path from their change to the review queue.
The pipeline is: load the data, train the model, save it, serve it. Everything else is detail that a framework handles.
The stages the naive version skips — dataset construction, feature computation, the split — are exactly where the label leaks, the sampling biases and the future creeps into the past. The offline number looks excellent because those stages were done carelessly, not despite it.
- The stages the naive version skips — dataset construction, feature computation, the split — are exactly where the label leaks, the sampling biases and the future creeps into the past. The offline number looks excellent because those stages were done carelessly, not despite it.
- Without an artifact stage, "the model" is whatever is on the serving box; without a feedback stage, next quarter's training set is silently shaped by this quarter's decisions.
- Because the stages are not named, a change in one is not recognised as a change to the model. The engineer who altered the feature pipeline did not know they had retrained the fraud model without retraining it.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The running example predicts whether a card transaction will be charged back within ninety days. The label arrives up to three months late, which is why the feedback stage matters.
- The pipeline's target is a model that can be rebuilt, redeployed and explained at every stage — not a single trained artifact.
- Raw data is the transaction event stream plus cardholder and merchant tables, owned by Data Engineering. One example is one transaction at the moment it was authorised, with features computed from what was known at that moment.
- Labels join in ninety days later from the chargeback system. The dataset is therefore always at least three months behind the present.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Each stage consumes the previous stage's output and fixes something the later stages must live with. Dataset construction fixes what one example is and which examples exist; features fix the representation; the split fixes what the evaluation can honestly claim; training fixes the weights; evaluation fixes the number that will be repeated; the artifact fixes what ships; deployment and inference fix what the user sees; feedback fixes what the next dataset contains.
- Information can only flow forward in the honest pipeline. Every classic failure is information flowing backward: the label into the features (leakage), the test set into training (evaluation leakage), the model's own predictions into its labels (feedback loops).
- The loop closes. The outcome of a prediction becomes raw data for the next dataset, so the pipeline is not a line but a cycle, and the model shapes the world it will next be trained on.
Eleven stages, one direction
The pipeline below is the reference shape for the whole domain. Read it as a chain of decisions that are frozen in order: once the dataset stage has decided what one example is, no later stage can un-decide it. That is why the early stages, which look like data plumbing, decide more about the final model than the training stage does.
The failsBy column previews the next lesson: each stage has a characteristic way of going wrong that shows up later, usually as a good offline number and a bad product.
- 1Raw data
Events and records as the source systems produced them, owned by Data Engineering.
fails by Missing, duplicated or late events that nothing downstream can detect.
- 2Dataset
Decides what one example is, which examples exist, and joins the label to each.
fails by Sampling bias, bad labels, label leakage.
- 3Features
Transforms each example into the representation the model will learn from.
fails by A transformation that cannot be reproduced at serving time.
- 4Train / Validation / Test
Partitions examples so the evaluation says something about unseen data.
fails by A random split on temporal or grouped data; tuning on the test set.
- 5Model
Chooses a function family and its capacity.
fails by Capacity mismatched to data: overfitting or underfitting.
- 6Training
Fits the weights by optimising a loss on the training fold.
fails by Non-reproducible runs; an objective that is not the business objective.
- 7Evaluation
Produces the number that will be repeated.
fails by The wrong metric; a number without its split, date range and baseline.
- 8Artifact
Packages weights, preprocessing and metadata as one versioned thing.
fails by Weights without the feature code they need; no lineage.
- 9Deployment
Puts the artifact where requests can reach it.
fails by Version mismatch between artifact, features and serving code.
- 10Inference
Computes a prediction for a live input under a latency budget.
fails by Train/serve skew; latency; a fallback that silently takes over.
- 11Feedback
Records outcomes and actions as tomorrow's raw data.
fails by Feedback loops; delayed or missing ground truth; staleness.
The loop from Feedback back to Raw data is not drawn as a step because it is not a step — it is the reason the pipeline runs more than once.
What each stage freezes
A useful way to hold the pipeline in your head is to ask, for each stage, "what can no later stage fix?" The dataset stage freezes the population and the meaning of the label. The feature stage freezes the representation. The split freezes what the evaluation can claim. Training freezes the weights. The artifact freezes what ships.
The consequence is that the cost of a mistake rises the earlier it was made and the later it is found. A label rule that is wrong is discovered, if at all, after training, evaluation, deployment and the arrival of production outcomes — months, in the fraud example — and every stage in between was correct work on a wrong foundation.
One notebook: load, clean, featurise, split, fit, score, pickle. Every stage is a cell; the boundaries are blank lines.
Each stage is a named unit with a versioned input, a versioned output, an owner and a check at the boundary. A change to any of them is a change to the model.
The failures of this domain enter at stage boundaries and are invisible to the final metric. Only a pipeline with named boundaries can locate a failure at the stage where it entered rather than at the number where it surfaced.
The loop closes
The stage the naive version most reliably forgets is the last one. Predictions cause actions, actions have outcomes, outcomes are recorded, and the record is the raw data of the next training run. A fraud model that blocks a transaction never learns whether it would have been charged back; a churn model whose flagged accounts are contacted learns that flagged accounts do not churn.
So the pipeline's most important assumption is about its own output: that the feedback it collects still reflects the world rather than the model. This is checkable, but only if the actions taken on predictions are logged alongside the outcomes — a design decision at the feedback stage that no later stage can retrofit.
The labels in the next training set record what happened, and where the model's own action changed what happened, that action is recorded and can be accounted for.
holds when Every prediction is logged with the action taken; a held-out control group is not acted on; blocked or intervened cases are marked as such rather than treated as negatives.
breaks when Actions are taken on predictions and not logged; the model's threshold changes and the label rate moves with it; blocked transactions are recorded as "not fraud" because no chargeback ever arrived.
respond Do not retrain on the contaminated data. Reconstruct labels using the control group and the action log, and fix the logging before the next run.
How to build it
Most important first.
- Name every stage explicitly in the codebase, with an owner and an input/output contract, so a change to any of them is recognised as a change to the model (The MLOps Pipeline).
- Make each stage rebuildable from versioned inputs: the dataset from a versioned query and label rule, the features from versioned code, the artifact from a recorded run (Dataset Versioning, Model Lineage).
- Put the split before any transformation that learns from data — normalisers, encoders, imputers — so nothing fitted on validation rows reaches the training fold (Preprocessing Leakage).
- Treat the feedback stage as a design decision, not an accident: decide what is logged, whether a control group exists, and how the model's own actions are marked in the next dataset (Feedback Loops).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Per stage: row counts, null rates and distribution summaries at each boundary, so a change is located at the stage where it entered rather than at the metric where it surfaced.
- The evaluation number is a measurement of the split stage as much as of the model. Report it with the split strategy, the date range and the class balance, or it is a number without a meaning.
- Do not measure the pipeline by "did the training job succeed". A training job that succeeds on a broken dataset produces a working artifact of the wrong model.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Each stage still consumes what the previous one produces: the schema at every boundary is unchanged, and a change upstream triggers a re-evaluation downstream.
- Information still flows only forward: nothing computed from the label, the future or the validation rows reaches the features or the weights.
- The feedback that becomes the next dataset reflects outcomes, not the model's own decisions, or the model's influence on it is recorded and can be corrected for.
- Offline: rebuild the artifact from the recorded stage versions and confirm the evaluation reproduces; diff the feature distributions at each stage boundary between the rebuild and the original.
- Online: at deployment, confirm the serving path's feature vectors match what the dataset stage would have produced for the same events (Serving Contract Tests).
- Over time: sample the feedback data and check that outcomes for accounts the model acted on are marked, and that the next training set's label rate has not moved in step with the model's threshold.
What can go wrong
- The stages are named but the artifact does not record which version of each produced it, so a rollback restores the weights and not the feature code they were trained against (Preprocessing Lives in the Artifact).
- The feedback stage is instrumented but the actions taken on predictions are not logged, so the next dataset cannot distinguish "did not churn" from "was contacted and saved".
- The pipeline is made rebuildable and nobody ever rebuilds it, so the first rebuild — during an incident — is the first time it is discovered not to work.
- Explicit stages with contracts are slower to change than a script; a data scientist iterating on features pays for boundaries that exist to protect production.
- Rebuildability requires versioning datasets, which is storage and a retention policy, and versioning label logic, which is a second thing to test.
- A control group in the feedback stage means deliberately not acting on some predictions the model is confident about — a measurable business cost paid for measurability.
- "The pipeline is the orchestration DAG." The DAG schedules stages; the pipeline is the set of contracts between them. A perfectly scheduled DAG can carry a leaked label from end to end (ML Orchestration).
- "Training is the expensive stage, so that is where the engineering goes." Training is the stage that is easiest to rerun. Dataset construction and feedback are where the unrecoverable mistakes are made.
- "If the training job is green, the model is fine." The job is green when the code ran. Whether the data it ran on still describes the world is not something a job status can report.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe stage sequence and the forward-only rule hold for every supervised system; unsupervised and self-supervised pipelines drop the label join but keep every other stage and every other failure.
- SIMPLIFIEDReal pipelines branch: several feature groups, several candidate models, a shadow deployment beside the live one. The single line here is a teaching shape; the contracts between stages are what carry over.
Where the depth lives
This domain teaches the model and hands the rest off by name.