What MLOps Is
Engineering practices and platform capabilities that make ML systems reproducible, testable, deployable, observable and maintainable. Not a product, and not a cluster.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A team has three models in production that nobody can retrain without the original author. What is actually missing — and why is the answer not "adopt a platform"?
Our churn model was trained by someone who left. The notebook is in a shared drive, the training data was a CSV on her laptop, and the artifact in production has no record of which features it expects. Leadership has asked us to "do MLOps". We have been offered a Kubernetes migration and a vendor platform, and we do not know which one is the answer.
MLOps is a tooling problem: adopt a platform, move training onto a cluster, and the reproducibility and deployment concerns take care of themselves. The vendors say so, and the platform demo shows a model going from notebook to endpoint in ten minutes.
The platform ships and the notebook runs on it unchanged. The training data is still whatever table existed on the day; the artifact still carries no feature contract; the promotion decision is still one person looking at one number. The infrastructure moved and the practices did not.
- The platform ships and the notebook runs on it unchanged. The training data is still whatever table existed on the day; the artifact still carries no feature contract; the promotion decision is still one person looking at one number. The infrastructure moved and the practices did not.
- Six months later a model degrades. Nobody can reproduce the run that produced the serving artifact, because the platform tracked the run but not the dataset snapshot (Dataset Versioning), so the "fix" is a retrain on different data that may or may not be better (Reproducibility).
- The Kubernetes migration consumed the quarter. Nothing about the model — its tests, its monitors, its rollback path — changed, because those were never Kubernetes problems.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The models themselves predict churn, lead score and delivery time. This lesson is about the surrounding system: the target is that any of those models can be rebuilt, tested, deployed and rolled back by someone other than its author, from what is recorded.
- The outcome to optimise is the time from "the model is wrong" to "a corrected model is safely serving", and the confidence that the corrected model is actually better.
- Three training scripts of three styles, two of which read from a warehouse table that has since been renamed. One artifact per model in an object store bucket, named by date. No record of the training data snapshot, the code commit or the evaluation that justified promotion.
- Serving code in a separate repository that hard-codes feature names. Logs of predictions exist for one model only, because that team happened to need them for billing.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An ML system has more moving parts than a service: code, data, features, hyperparameters, the trained artifact, the serving path and the outcome feedback. A change to any of them changes behaviour, and most of them are not in version control by default. MLOps is the set of practices that puts each of them under control: versioned, tested, promoted through gates, observed after deployment.
- The capabilities are separable. Reproducibility comes from recording code, data and config per run (Experiment Tracking). Testability comes from a test stack that covers data, training and serving (The ML Testing Stack). Deployability comes from a registry with a promotion policy and a rollout path (The Model Registry, Canary Rollout). Observability comes from logging predictions and joining them to outcomes (Prediction Logging). None of these requires a particular scheduler.
- Infrastructure is where those practices run, not what they are. A cron job, a versioned bucket and a test script implement most of them at small scale. A cluster and a workflow engine implement them at large scale. The practices are the same; the implementation is chosen by volume and team size.
A model is more than code, so more things need versioning
A service is a function of its code and its config. A model is a function of its code, its config, its training data, its feature definitions and the random state of a training run. Change any one of them and the behaviour changes, and only the first two are in version control by habit.
MLOps is the discipline of bringing the rest under the same control — not because a tool requires it, but because otherwise the artifact in production cannot be explained, tested or rebuilt. The practices are separable and each one can be implemented with very little infrastructure.
| Capability | What it needs | Minimal implementation | What it is not |
|---|---|---|---|
| Reproducibility | Code commit, dataset snapshot, config, seed per run | A run manifest written next to the artifact | An experiment-tracker UI |
| Testability | Data tests, a training smoke test, a serving contract test | A test script the CI runs | High unit-test coverage of the training code |
| Deployability | A registry with stages, a promotion gate, a rollout and a rollback | A versioned bucket, an evaluation script, a canary flag | An endpoint that accepts a new file |
| Observability | Logged predictions joined to outcomes, feature and prediction distributions | A prediction log table and a weekly report | CPU and latency dashboards |
| Maintainability | Someone other than the author can do all of the above | A runbook that has been followed once by a stranger | A wiki page |
Where "MLOps means Kubernetes" comes from, and why it is wrong
The slogan comes from a real correlation: teams with mature MLOps often run on orchestrated containers, because at their scale that is the sensible way to schedule training jobs and serve models. The direction of causation is the other way. They needed the scheduler because they had many pipelines; the pipelines were reproducible and gated because of the practices, which predated the cluster.
The test is to ask what a Kubernetes migration changes about a single model. It changes where the training job runs and how the serving container is placed. It changes nothing about whether the training data is versioned, whether promotion is gated or whether the rollback works. Those are the failures the team actually had.
A quarter spent containerising training scripts and standing up a cluster and a tracking server. The notebook now runs on the cluster, reads the same live table, and produces an artifact with the same missing feature contract.
A week: training reads a named snapshot, writes a manifest with commit, data hash and config, and the artifact bundles its feature list. A CI job rebuilds and compares metrics. The cron job stays a cron job until there are enough pipelines to need more.
The failure was "cannot rebuild", which is a property of what is recorded per run, not of where the run executes. The practice fix addresses it directly; the platform fix addresses something else and hopes the practice follows.
What must stay true after the practices are in place
Practices decay. The manifest keeps being written until someone adds a feature computed from a table the manifest does not name; the gate keeps being enforced until a deadline; the rollback keeps working until the serving image changes its loading code. The assumption that any of these still holds is checked by exercising them, not by their existence.
This is the same discipline the domain applies to models: a capability is a set of assumptions, and the operational question is how you would know one of them stopped holding.
The code commit, dataset snapshot and config recorded for the production artifact are sufficient to retrain a model with matching evaluation metrics.
holds when Training reads only from immutable, named inputs; every input is in the manifest; the rebuild is exercised on a schedule and the metrics compared.
breaks when A feature is added that reads a live table; a preprocessing step lives outside the versioned code; the snapshot storage expires old versions; a dependency is unpinned and its behaviour changes.
respond Treat a failed rebuild as an incident against the practice, not the model: find the unrecorded input and put it in the manifest before anything else changes.
1run_id: churn-2026-08-20-172code:3 repo: models/churn4 commit: 3f9a1c25data:6 dataset: churn_training_v14 # an immutable, named snapshot7 sha256: 9b1f... # of the snapshot manifest, not the live table8 label_window: "2025-11-01/2026-05-01"9features:10 definition_version: features-churn@711config:12 model: gradient_boosting13 params: { max_depth: 6, n_estimators: 400, learning_rate: 0.05 }14 seed: 2026082015evaluation:16 split: time-based17 report: s3://models/churn/runs/2026-08-20-17/eval.json18artifact:19 path: s3://models/churn/runs/2026-08-20-17/model.bin20 feature_contract: [tenure_days, plan_tier, tickets_30d, logins_7d, ...]The line that matters most is the data hash. A manifest that names a table but not a snapshot records an input that will not exist next week.
How to build it
Most important first.
- Start from the failure you had: the model could not be rebuilt. Fix that first — training reads from a named dataset snapshot, the run records its commit and config, the artifact carries its feature contract (What a Model Artifact Contains).
- Add gates in the order the failures cost you: a data test before training, an evaluation gate before registry promotion, a shadow or canary before full traffic, a rollback that has been exercised (Rollback & Fallback).
- Only then ask what infrastructure the gates need. If one training job a week fits on a VM, use the VM. Adopt orchestration when the number of pipelines makes hand-scheduling the bottleneck (ML Orchestration).
- Treat "MLOps means Kubernetes" as a claim to refute, not a plan: Kubernetes is one way to schedule containers, which is one part of one capability. A team can have excellent MLOps on a single machine and no MLOps at all on a large cluster.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Time to rebuild any production model from records alone, by someone who did not train it. This is the number that says whether the practices exist.
- Fraction of production models with a data test, an evaluation gate, a monitor and an exercised rollback. Count capabilities per model, not tools per team.
- Cluster utilisation and platform adoption are not MLOps metrics. A fully adopted platform hosting unreproducible models has solved nothing.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every production artifact can be traced to a code commit, a dataset snapshot and a config, and that trace is checked by rebuilding, not by the presence of a tracking record.
- The gates in the pipeline are enforced — a model cannot reach the registry's production stage without passing them — and the bypass path is audited.
- The people operating the model are not the people who trained it, and the recorded information is sufficient for them.
- Offline: pick a production model at random, delete the local copy, and rebuild it from the registry record. Compare the evaluation metrics to the recorded ones. A mismatch is a reproducibility gap.
- Online: trigger the rollback path for a model on a quiet day and time it. A rollback that has never been exercised is a plan, not a capability.
- Over time: re-run the rebuild exercise quarterly, because the data sources and the code move underneath the records.
What can go wrong
- The platform becomes the practice: because the platform "tracks experiments", nobody checks that the tracked run actually records the dataset snapshot, and reproducibility is assumed rather than tested.
- Gates are added but not owned. The evaluation gate compares against a threshold nobody updates, and the promotion policy is bypassed the first time it blocks a deadline.
- The team builds general MLOps infrastructure for three models and spends more on the platform than the models earn (ML Platform Engineering).
- Recording everything per run is storage and discipline; the team pays it on every run to benefit on the rare bad one.
- Gates slow the happy path. A model that would have shipped in an afternoon now waits a day for a shadow comparison.
- Deferring infrastructure means the first real scaling problem arrives without a platform, and the migration then happens under pressure.
- "MLOps means Kubernetes." It means practices. Kubernetes is one scheduler, useful when the training and serving workload justifies containers at scale, and irrelevant to whether the model is reproducible.
- "We have an experiment tracker, so we have reproducibility." The tracker records what it was told. If the dataset was a query against a live table, the record names a dataset that no longer exists.
- "MLOps is the platform team's job." The platform provides the capability; the model team owns the tests, the gates and the monitors for their model. A capability nobody uses is a cost.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat an ML system has more versioned inputs than a service — data, features, artifact, config — holds for every model family and every deployment target, so the practices apply everywhere; only the implementation scales.
- SCALE-SPECIFICFor one or two models the whole set of practices fits in a repository, a cron job and a versioned bucket; for dozens of models across teams the practices need a shared platform, and the platform question becomes real.
- CONTESTEDA serious position holds that practices without a platform do not survive: each team re-implements tracking, gates and monitors slightly differently, the implementations rot, and only a shared platform with enforced defaults keeps the practices alive across team turnover. That is a strong argument once there are many models; the reply is that a platform adopted before the practices exist enshrines whatever the first team did.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — the question of which gates are worth their slowdown, and how a test that blocks a deadline gets bypassed, is a testing-culture question this domain assumes rather than answers.