What ML Engineering Is
Not "which algorithm". Turning data into a system that learns useful patterns, generalises, serves predictions and stays measurable after it ships — and knowing which neighbouring domain owns each thing it depends on.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A team has a notebook with a good model in it. What is the distance between that notebook and a system the business can rely on, and which of it is ML engineering's job?
A data scientist built a churn model in a notebook that beats the current heuristic on a held-out set. Product wants it "in the app by next quarter". Nobody can say who owns the feature pipeline, the API, the GPU bill, the retraining, or the call at 3 a.m. when it starts flagging every user.
ML engineering is the notebook plus deployment: wrap the model in an endpoint, schedule a job, done. The hard part — the model — is already finished.
The notebook's features were computed over a static export that included data from after each snapshot date. The daily job cannot see the future, so the production features differ from the training features and the model is confidently wrong (Label Leakage).
- The notebook's features were computed over a static export that included data from after each snapshot date. The daily job cannot see the future, so the production features differ from the training features and the model is confidently wrong (Label Leakage).
- The endpoint works. The feature tables it reads are refreshed at 06:00 by a pipeline that sometimes fails silently, and on those days the model scores every account on yesterday's numbers and nobody notices for a week.
- The retention team acts on the list, so accounts the model flags are contacted and some are saved — and next quarter's training data records them as "did not churn". The model is trained on the effect of its own predictions (Feedback Loops).
- The held-out number that justified the project cannot be reproduced three months later because nobody recorded which rows, which split, which code version or which random seed produced it.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The notebook predicts whether a subscriber will cancel within thirty days, from account and usage history. The label is a cancellation event observed thirty days after the snapshot.
- The target of the *system* is different: a daily list of accounts for the retention team to contact, that stays useful as the product, the users and the data pipelines change underneath it.
- One example is one subscriber on one snapshot date, joined to thirty days of usage before the snapshot and one bit of outcome thirty days after. The notebook built this from a warehouse export that a data engineer produced once, by hand.
- In production the same features must be computed every day, from tables owned by three other teams, at the moment the prediction is needed — none of which the notebook expresses.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An ML system is a pipeline: raw data → dataset → features → split → model → training → evaluation → artifact → deployment → inference → feedback (The ML Pipeline). The learned function is one stage. Every other stage is a place where an assumption can silently stop being true.
- The learned function differs from ordinary code in one way that decides the whole discipline: its behaviour is fixed by the data it was trained on, not by rules someone wrote (Learning vs Programming). So its correctness is a property of the data's relationship to the world, and that relationship changes without anyone editing anything.
- ML engineering is the work of making that pipeline reproducible, testable, observable and operable: knowing what the model learned from, what it optimised, what must stay true for it to keep working, and how you would notice when it does not (Don't Delegate Understanding).
The model is the smallest part
Lay out what the churn notebook depends on and the model is one box among a dozen. Upstream: event collection, the warehouse tables, the join that builds one example per subscriber-day, the label rule that decides what "cancelled" means. Downstream: an artifact, a serving path, features computed at request time, a threshold, an action, an outcome, and the log that turns the outcome into next quarter's training data.
Each box is somebody's domain. The point is not to learn all of them but to know, for every box, what the model assumes about it — and which neighbour to link to for the depth. The pipeline diagram is the map of where those assumptions live.
Where the neighbours end and this domain begins
The fastest way to ruin an ML system is to re-teach its neighbours inside it. Each row below names what the neighbour owns and what stays here — the model-level view of the same thing. When a lesson in this domain seems to be explaining stream processing or HTTP, it should be linking instead.
The two neighbours most often confused with this domain are Data Engineering and Agentic Engineering. Data Engineering ends where a validated, served table begins; the meaning of that table to a model is ours. Agentic Engineering starts where a model is *used* — prompts, tools, retrieval, memory, agent evals; the model itself, its training, serving and drift, is ours.
| Neighbour | Owns | This domain owns |
|---|---|---|
| Data Engineering | Collect, move, transform, validate, serve data | What the model needs from it: point-in-time features, label logic, dataset construction |
| Backend Engineering | Expose the model through a production API | What is being served and why identical weights can be wrong |
| Cloud & Infrastructure | Compute, accelerators, storage, orchestration | What to ask for — GPU hours, VRAM, batch jobs — and what it hides |
| Computer Architecture | How a GPU executes | Why batching, quantization and bandwidth decide inference cost |
| Distributed Systems | How nodes agree and fail | Gradient synchronisation as one instance |
| DevOps | Delivery, rollout, rollback | Registry, shadow, canary, champion/challenger for a model |
| Observability | Measure a running system | Feature, prediction and outcome distributions |
| Security | The attacker model | Poisoning, artifact integrity, adversarial inputs, inference abuse |
| Agentic Engineering | LLM and agent systems: prompting, RAG, tools, memory, evals | The model underneath: training, fine-tuning, embeddings, inference, drift |
What the notebook cannot say
A notebook records a result. It does not record what must remain true for the result to keep being true — that the features will be computed the same way tomorrow, that the pipeline ran, that the world is still the one the data described. Those are the assumptions the model is made of, and the domain's job is to make them explicit and checkable.
This is the habit the rest of the domain builds: for every model, write down the assumptions, decide how each would be detected if it broke, and put that detection in production before the model. A model deployed without it is correct until it silently is not.
Every prediction made today is computed from features that were refreshed today, from a pipeline that completed without partial loads.
holds when The feature job is monitored for completion and row counts, publishes atomically, and the serving path can tell a stale table from a fresh one.
breaks when The upstream job fails after writing half its partitions; a source table is late; a schema change makes a join silently produce nulls that the model reads as a legitimate value.
respond Fall back to the previous day's scores or the rule-based list, page the pipeline owner, and do not retrain — the model is fine, the input is not.
How to build it
Most important first.
- Start from the decision the prediction serves and work backwards to the target, the data and the point in time the prediction must exist (Problem Formulation). Most notebook models fail here, before any code.
- Draw the boundary with each neighbouring domain and name an owner on each side. Data Engineering owns collecting, moving, validating and serving the tables; you own what the model does with them and what it needs from them (The ML Pipeline).
- Treat the artifact as code plus data plus configuration, versioned together, with the evaluation that approved it recorded alongside (Reproducibility, What a Model Artifact Contains).
- Instrument the system for the three distributions ordinary services never log — features, predictions and outcomes — because those are the signals that tell you an assumption broke (Model Monitoring).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The business number the prediction was meant to move: retained revenue per contact for the retention list, measured against a control group that was not contacted. This is the only number that says the system works.
- Production quality at the operating threshold once labels arrive, per slice, compared with the offline number that approved the model. A persistent gap is the domain's signature and means an assumption is wrong somewhere in the pipeline.
- Do not measure "the notebook's held-out AUC" and treat it as a property of the system. It is a statement about one dataset, one split and one date.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The relationship between the features and the outcome that held in the training data still holds in the population being scored today — the world has not moved in a way the model cannot see.
- The features computed at prediction time are the same quantities, computed the same way from the same sources, as the features the model was trained on.
- The pipeline that feeds the model ran, ran to completion, and ran on today's data — not yesterday's, and not a partial load.
- The outcome the model is trained on is still the outcome the business cares about, and the action taken on the prediction has not changed what gets recorded as the outcome.
- Offline: rebuild the training set from the versioned pipeline, not the export, and confirm the evaluation reproduces to within noise. If it cannot be rebuilt, the number is not evidence.
- Online: compare the distribution of features at the serving boundary against the training set on day one, before any labels exist; log every prediction with its feature vector and model version so the gap can later be explained (Prediction Logging).
- Over time: when labels arrive, compute production quality against the offline number and against a held-out control group, and treat divergence as a broken assumption to diagnose rather than a reason to retrain.
What can go wrong
- The boundaries are drawn but nobody owns the seam: the feature pipeline is "data engineering's" and the model is "ML's", so a schema change that breaks the model is nobody's incident (Data & Feature Tests).
- The team builds the platform first — registry, feature store, orchestration — for one model that could have been a cron job and a CSV, and the model never ships.
- The neighbouring domain is re-implemented badly inside the ML system: a hand-rolled stream processor for features, a bespoke HTTP server for inference, a private copy of the deployment pipeline.
- Reproducibility, monitoring and ownership are engineering cost that shows no improvement in any offline metric; a team judged on validation numbers has no incentive to pay it until the first invisible failure.
- Drawing hard boundaries with neighbouring domains means depending on other teams' roadmaps for tables, endpoints and clusters, which is slower than doing it badly yourself.
- The discipline of asking "on what data, how split, against which baseline" before believing a number slows every conversation that starts with a number.
- "ML engineering is MLOps, and MLOps means Kubernetes." The practices — reproducibility, testing, monitoring, rollout — are the discipline; the platform is one way to host them, and a single model rarely needs it (What MLOps Is).
- "The model is done, the rest is plumbing." The plumbing is where the model's assumptions live. A model whose features are computed differently in production is a different model, with the same weights.
- "Agentic engineering has replaced this." Agentic systems sit on top of models; someone still trains, fine-tunes, serves, monitors and retrains the model underneath, and the boundary is deliberate (The Boundary With Agentic Engineering).
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat the learned function is one stage in a pipeline whose other stages carry assumptions is true of every ML system, from a linear model on a cron job to a fine-tuned foundation model behind an agent.
- SCALE-SPECIFICThe boundaries drawn here become separate teams at scale; on a three-person team one engineer plays every role, which does not remove the seams — it removes the ownership conversation that would have made them visible.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — the ML test stack (data, feature, training, model, serving, drift) is a testing discipline this domain assumes; its general theory of confidence and resilience lives in a domain that does not exist yet.
- — Programming Languages & Runtime Internals — a model artifact is executed by a runtime with its own memory model, numeric semantics and versioning; why a pickled object from one interpreter fails in another is a runtime question.