ML System Design Architecture
Sources → data platform → features → training → registry → serving → application → monitoring. Eight boxes, five owning teams, and four interfaces that decide whether the system can be reasoned about at all.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What are the boxes in an ML system, who owns each one, and what has to cross the boundaries between them for the whole thing to stay debuggable?
We have a model in a notebook that the product team wants "in production by next quarter". Three teams are involved and nobody can say what the data platform team delivers to the ML team, what the ML team hands to backend, or who gets paged when the predictions go wrong.
Draw the model in the middle, the database on the left, the API on the right. Data Engineering gives us a table, we train on it, backend wraps the pickle in an endpoint. Three boxes, three teams, a sprint each.
The table Data Engineering delivered is a current-state snapshot, so the training rows contain the future; the offline metric is excellent and the deployed model has nothing to work with (Temporal Leakage). Nobody owned "as of what date is each row true".
- The table Data Engineering delivered is a current-state snapshot, so the training rows contain the future; the offline metric is excellent and the deployed model has nothing to work with (Temporal Leakage). Nobody owned "as of what date is each row true".
- Backend wrapped the pickle and re-implemented the feature computation from a document, so the serving vector differs from the training vector for the same entity. No exception, no schema error, and a quality complaint three weeks later.
- The model was retrained and redeployed with the same file name. When quality drops, nobody can say which artifact produced which prediction, because the prediction log — if there is one — carries no version.
- Monitoring shows p95 latency and error rate, both green. Precision has halved because the label delay is thirty days and no one wired the outcome back to the prediction.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The target of the surrounding system is whatever decision the application makes from a prediction — here, treat it as a generic scored decision so that the architecture can be drawn without it. The four worked designs later in this module (Designing a Recommendation System, Designing a Fraud Detection System, Designing a Churn Prediction System, Designing Search Ranking) each bend this architecture around a specific target.
- What the architecture itself optimises is traceability: for any prediction the application acted on, the system can name the artifact, the feature definitions, the dataset version and the training run behind it.
- Sources are operational systems that were never designed for modelling: an OLTP database that shows current state rather than history, an event stream with late and duplicated events, third-party feeds with their own clocks. Data Engineering lands them in a platform (The Data Lake, The Data Warehouse) with lineage and quality checks.
- One training example is a row the feature pipeline produced for one entity at one point in time, joined to a label the label pipeline produced later. Both pipelines have versions, and a dataset is the pair of versions plus a snapshot range (Dataset Versioning).
- Serving-time data is a different animal: a request carrying an entity id and whatever the application knows at that moment, resolved into a feature vector by a lookup or a computation that must reproduce training (Train / Serve Skew).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An ML system is a pipeline of eight stages with four artifacts crossing the boundaries. Sources feed a data platform (owned by Data Engineering) that produces versioned datasets. A feature pipeline turns them into feature definitions that are executed twice — in batch for training, at request time for serving. Training produces an artifact; the registry stores it with its lineage; serving loads it and answers requests; the application turns predictions into decisions; monitoring watches features, predictions and, eventually, outcomes.
- The interfaces are what make this a system rather than a chain of hand-offs: a *dataset version* (which rows, which label definition, which snapshot date), a *feature definition* (name, source, window, null policy, and a version), an *artifact* (weights plus preprocessing plus the feature schema it expects, What a Model Artifact Contains), and a *prediction log* (entity, features, artifact version, score, decision, and later the outcome, Prediction Logging).
- Each interface is checkable. A dataset version can be re-materialised; a feature definition can be executed on both paths and diffed; an artifact can be loaded and scored against recorded rows; a prediction log can be joined to outcomes. A boundary without a checkable interface is where the invisible failures live.
The eight boxes and who owns them
The picture below is the reference architecture with ownership attached. Read the arrows as hand-offs of a specific artifact, not as "data flows": the data platform hands the feature pipeline a *dataset version*; the feature pipeline hands training and serving a *feature definition*; training hands the registry an *artifact*; serving hands monitoring a *prediction log*.
The boundary with neighbouring domains runs through these boxes. Sources and the data platform are Data Engineering (Feature Pipelines, Data Lineage). The application and the request path are Backend (The Request Lifecycle). Compute for training and serving is Cloud (GPU and Accelerator Infrastructure). The signal pipeline behind monitoring is Observability (The Four Golden Signals). This domain owns what is in between and the model-specific signals monitoring needs.
The four interfaces, as things a test can check
Each seam in the diagram is safe only if the thing crossing it can be verified by a machine. A dataset version that is "the table as of last Tuesday" cannot; one that names a label query, a feature-pipeline version and a snapshot range can be re-materialised and diffed. The matrix names each interface, who produces it, who consumes it, and the check that proves it is intact.
The cross-domain map is worth stating explicitly, because each interface hands off to a neighbouring domain: the feature pipeline to Data Engineering, the model API to Backend, the GPU to Computer Architecture (What Is Actually Inside a GPU), distributed training to Distributed Systems (Splitting a Computation Across Machines), the registry and canary to DevOps (Artifact Registries, Canary Deployments), prediction drift to Observability (From Symptom to Root Cause), poisoning to Security (Data Poisoning here, Artifact Integrity there), transformers and embeddings in use to Agentic (RAG Overview).
| Interface | Produced by | Consumed by | Check that proves it |
|---|---|---|---|
| Dataset version | Data platform + label pipeline | Training | Re-materialise from the version and compare row counts and label prevalence |
| Feature definition | Feature pipeline | Training and serving | Replay production requests through the batch path; diff vectors |
| Artifact | Training | Registry, serving | Load, score fixed rows, compare to values recorded at training |
| Prediction log | Serving | Monitoring, next dataset | Every row carries artifact version and feature vector; joins to outcomes |
What must stay true after the first deploy
The architecture does not fail loudly. A seam degrades — a feature definition changes on one side, a log field is dropped to save storage — and the system keeps answering requests. The assumption the whole design rests on is that every decision remains traceable, and that has to be checked, not believed.
The cheapest check is a random trace: once a week, take one logged prediction and follow it back to a registry entry and a dataset version. The day it cannot be followed is the day the architecture stopped existing, whatever the diagram on the wiki says.
For any prediction the application acted on, the artifact version, the feature vector it was scored on, and the dataset version it was trained from can be recovered.
holds when Serving logs the vector and version; the registry entry names the dataset version; the data platform can re-materialise that version because it keeps history rather than current state.
breaks when A log field is trimmed for cost; the registry is repointed by hand; a source table is overwritten in place; a hot-fix redeploys an artifact under an existing version name.
respond Restore the broken link before the next promotion, and treat the artifact deployed during the gap as untraceable — do not use its prediction log as training data.
1{2 "request_id": "6f1c…",3 "entity_id": "acct_48213",4 "scored_at": "2026-08-24T09:12:03Z",5 "artifact": "churn-v14",6 "feature_defs": "features@a91e0c2",7 "features": { "active_days_30": 4, "seats_used": 0.6, "days_to_renewal": 19 },8 "score": 0.71,9 "decision": "call_list",10 "outcome": null,11 "outcome_available_after": "2026-09-23"12}The two fields that are most often missing are feature_defs and outcome_available_after. Without the first, drift cannot be attributed to a definition change; without the second, someone computes precision on a week-old row and reports a number that means nothing.
How to build it
Most important first.
- Assign ownership per box before drawing arrows: Data Engineering owns sources and the data platform; ML Engineering owns features, training, evaluation, the registry and the model half of serving; Backend owns the request path and the application; Cloud provides compute and storage; Observability owns the signal pipeline that monitoring feeds. The seams are where the four interfaces live.
- Make every interface a versioned, testable artifact rather than a conversation: feature definitions in code that both paths execute (Feature Stores are one way, not the only way); preprocessing inside the model artifact (Preprocessing Lives in the Artifact); a registry entry that names dataset, code and environment (Model Lineage).
- Log at the serving boundary — the feature vector, the artifact version and the score — because that log is the only place production truth exists, and it is the future training set once outcomes join it (Tracing a Prediction).
- Design monitoring in three layers by how soon each can fire: feature distributions today, prediction distributions today, outcome metrics after the label delay (Model Monitoring, Ground-Truth Delay). Green infrastructure dashboards say nothing about a model.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- For the architecture, the number that matters is time-to-trace: given a bad decision the application made, how long until someone can name the artifact, the feature values and the training dataset behind it. Minutes means the interfaces exist; days means they do not.
- Per interface: the feature equivalence mismatch rate between the two execution paths; the fraction of predictions logged with an artifact version; the fraction of registry entries whose dataset can be re-materialised.
- Infrastructure metrics — latency, error rate, GPU utilisation — measure the service, not the model. They are necessary and they are the ones most often mistaken for sufficient.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every prediction the application acts on is logged with the artifact version and the feature vector it was scored on, so any decision can be traced back through the four interfaces.
- The feature definitions executed at serving time are the ones the deployed artifact was trained against, and a change to either is a new version of both.
- The dataset version behind the current champion can be re-materialised from the data platform, which requires the platform to keep history rather than current state.
- Each of the five owning teams has a pager for its box and an agreed protocol for incidents that cross a seam.
- Offline: pick a recorded prediction at random and trace it — artifact version, registry entry, dataset version, feature definitions — without asking anyone. If the chain breaks, that break is the design flaw.
- Online: replay a sample of production requests through the training feature path and diff the vectors (Serving Contract Tests); assert that every logged prediction carries a version.
- Over time: when outcomes arrive, join them to the prediction log and check that the outcome metric can be computed per artifact version and per feature slice. If it can only be computed in aggregate, the log is too thin.
What can go wrong
- The feature pipeline is versioned but its upstream table is not, so a Data Engineering change to a source column silently changes every downstream feature while the feature version stays the same (Data Contracts exist for this).
- Prediction logs exist but store the entity id and score only, so when outcomes arrive nobody can compute per-feature drift for the population that was scored wrongly.
- The registry is a bucket of files with dates. Rollback works — to a file — but the feature definitions the previous artifact expected have moved on, and the rolled-back model receives a schema it never saw.
- Ownership is drawn per box and the seams are owned by nobody; the first incident that crosses a seam is triaged by three teams each proving it is not theirs.
- Versioned interfaces are slower to change than a shared document and a Slack thread. A feature change now touches a definition, a version, a registry entry and a contract test.
- Logging the feature vector at serving time is storage, a privacy review and a retention policy — and it is the only path to training on what production saw.
- Five owning teams means five sets of priorities; a system with clear seams still needs someone who owns the whole and can say the system is broken when every box is green.
- "We need a feature store, a registry and a serving platform before the first model." The interfaces matter; the platforms are one way to get them. A single repository with versioned feature code, a bucket with hashed artifacts and a logging table give most of the traceability at a fraction of the cost.
- "Backend owns serving, so serving quality is backend's problem." Backend owns the request path. What the artifact does with the features is ML Engineering's, and a seam nobody owns is where skew lives.
- "MLOps means Kubernetes." MLOps means the four interfaces are versioned and testable. Kubernetes is one substrate for running the jobs and services; a scheduled batch job on a VM can be a fully operated ML system.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe eight stages and four interfaces apply to any predictive system, batch or online, tabular or deep; what changes per system is which boxes are heavy and which interfaces are latency-critical.
- SCALE-SPECIFICA two-person team can hold every interface in one repository and one scheduled job; the ownership map in this lesson describes an organisation with separate data, ML, backend and platform teams, where the seams are between people and not only between modules.
- CONTESTEDA serious position holds that drawing the full architecture up front is premature: ship the batch-scored table first, discover which interfaces actually hurt, and build only those. That is right for a first model; the counter-argument is that the prediction log and the dataset version are nearly free on day one and impossible to reconstruct on day ninety.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — each of the four interfaces is a contract between teams, and the discipline of keeping contract tests green across independently deployed components is a testing question this domain borrows.