The ML Pipeline
Raw data to feedback in eleven stages. Each one names what it produces and what it decides for everything downstream. Then press the button and read where each of the sixteen failure classes enters the system — which is almost never where it is noticed.
A tutorial draws this pipeline as a happy path, and read left to right that is what it is. Read on the reveal it is a map of where things are born: leakage enters at the dataset, is invisible at evaluation and is discovered in production; a feedback loop is closed at the last stage and shows up as a suspiciously good validation number two quarters later. Most of the sixteen are invisible offline — that is not an accident of this list, it is the defining property of the discipline. A stage that decides nothing is a box on a slide, so every stage here says what it fixes for the stages after it.
- 1Raw Data
produces Events, records, logs and documents as the product happened to record them — produced for the product's purposes, not for learning, with whatever gaps, duplicates and late arrivals that implies.
What this stage decides- — What the system can ever know. A signal that was never logged cannot be a feature, and a population that never reached the product cannot be in the training set.
- — The grain of time: whether an event carries the moment it happened or the moment it was written, which decides whether point-in-time correctness is even possible later.
- — Which humans and processes touched the data before you did — and therefore whose incentives are encoded in it.
- 2Dataset
produces A table of examples with a label: one row per unit of prediction, built by filtering, joining, snapshotting and labelling the raw data as of some point in time.
What this stage decides- — What one example *is* — a user, a user-week, a session, an order — which decides the number of rows, the correlation between them and the unit of every metric afterwards.
- — The label definition and its horizon, which fixes the delay before ground truth exists for any future prediction.
- — The point in time each row is frozen at, which decides whether every feature computed later is honest or a glimpse of the future.
- 3Features
produces Numeric representations of each example — aggregates, encodings, normalised values, embeddings — computed by code that will have to run again, identically, on the serving path.
What this stage decides- — Which regularities the model can express: a tree cannot learn a ratio it was not given, and a linear model cannot learn an interaction that was not built.
- — The contract with serving: every transformation here is a promise that the same function, with the same reference data, will run at prediction time.
- — Which statistics are fitted on which data — a normaliser's mean, an encoder's categories — and therefore where preprocessing leakage can enter.
- 4Train / Validation / Test
produces Three disjoint sets with three jobs: fit the parameters, choose among models and settings, and estimate — once — how the chosen model will do on data it has never influenced.
What this stage decides- — Whether the validation metric means anything: a split that lets the same user, the same week or the same document appear on both sides measures memorisation, not generalisation.
- — Whether the estimate resembles deployment: training on the past and validating on the future is the only split that rehearses what production will do.
- — How much data is left to learn from, and how noisy the estimate on what remains will be.
- 5Model
produces A function family with tunable parameters — a hypothesis about the shape of the relationship between features and target — plus the hyperparameters that fix its capacity.
What this stage decides- — The capacity and inductive bias: what kinds of pattern are easy for it to represent and what kinds it will approximate badly.
- — The serving cost profile: memory, latency per prediction and whether a GPU is even relevant.
- — How explainable the prediction will be to the person who has to act on it.
- 6Training
produces Fitted parameters: the weights that minimise the loss on the training set under whatever optimiser, schedule, seed and stopping rule were used — and a record of all of those, or not.
What this stage decides- — Which minimum of the loss the model lands in, which depends on initialisation, learning rate, batch size and seed — so "the model" is one draw from a distribution of models.
- — Whether the run is reproducible: with the data version, code version, environment and seeds recorded, or with none of them.
- — How long the fit was allowed to continue, which — with early stopping — is where the validation set quietly becomes part of training.
- 7Evaluation
produces Numbers: a validation metric, a test metric used once, a confusion matrix at a threshold, slices, calibration — and a decision about whether this candidate beats the baseline and the champion.
What this stage decides- — Whether the model ships, on the basis of a number that is a statement about one dataset, one split and one point in time.
- — The threshold, if it is chosen here, which turns a score into a decision and fixes how many positives the downstream action will receive.
- — What "better" means for the next iteration, because whatever is reported here is what will be optimised.
- 8Artifact
produces A versioned bundle: weights, preprocessing code and fitted constants, feature schema, metadata about the training data and metrics, a signature — registered so that a serving system can load it by name and a human can trace it back.
What this stage decides- — What serving is allowed to assume: the input schema, the output format and the preprocessing that must run — or, if the artifact is bare weights, nothing at all.
- — Whether a prediction can ever be traced to a training run, a dataset version and a code commit.
- — What "rolling back" means: a previous artifact that can be loaded unchanged, or a rebuild from memory.
- 9Deployment
produces The artifact loaded into a serving context — a batch job, a model server, an in-process library — with a feature pipeline feeding it, a rollout strategy and a fallback.
What this stage decides- — The inference mode: batch, online or streaming, and therefore the freshness of the features the model sees and the latency budget it has to meet.
- — How the features are computed here, by whose code, from which store — the other half of the train/serve contract.
- — What fraction of traffic sees the new model, and what happens when it fails.
- 10Inference
produces A prediction per request or per row — a score, a class, a ranking — plus, if anyone thought to log it, the features it was made from, the model version and a request id.
What this stage decides- — The threshold or decision rule actually applied, which is often set in a config file far from the evaluation that chose it.
- — What is logged, and therefore what can be debugged or joined to an outcome later.
- — The fallback behaviour when a feature is missing, a timeout fires or the model server is down.
- 11Feedback
produces Outcomes joined back to predictions: what actually happened to the user this model scored, when it became known, and what action was taken in between — which becomes the next training set.
What this stage decides- — Whether ground truth exists at all for this prediction, and how late it arrives.
- — Whether the next model learns the world or learns this model's decisions: a prediction that triggered an intervention has changed the outcome it will be evaluated against.
- — When retraining is warranted — a decision this stage should inform, not a schedule.
Take it further
The same pipeline from the other end: one request, fourteen stops, and what each stop assumes about the ones either side of it.
ML Failure Simulator →Inject one of these failures and read the monitoring signals — then find out whether the signals alone could have told you which one.