ML Platform Engineering
Shared capabilities — dataset access, feature pipelines, training jobs, tracking, registry, serving, monitoring, GPU scheduling — built once for many model teams. Premature before the third model.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Four teams each built their own training and serving path. When does a shared ML platform pay for itself, what does it own, and where does its responsibility stop and the model team's begin?
We have four product teams each running their own models. Each has its own way of getting training data, its own serving container and its own idea of monitoring. Two have been burned by the same skew bug. Leadership wants a central ML platform team. The product teams are worried they will lose the ability to ship, and the platform candidates are worried they will become a ticket queue.
Build a platform that does everything: one training service, one feature store, one serving stack, one monitoring dashboard. Migrate everyone. Standardisation is efficiency.
The platform is built to the first team's needs — the fraud team, who needed real-time features — and the forecasting team, whose models are batch jobs over a warehouse, gets a feature store it cannot use and a serving stack it does not need. They keep their own path and the platform covers three of four teams.
- The platform is built to the first team's needs — the fraud team, who needed real-time features — and the forecasting team, whose models are batch jobs over a warehouse, gets a feature store it cannot use and a serving stack it does not need. They keep their own path and the platform covers three of four teams.
- The platform team owns everything, so every model change goes through them. The queue grows, the product teams route around it, and within a year there are five stacks instead of four (Feature Stores for how a shared feature layer specifically fails this way).
- The migration costs two quarters across four teams during which no model improves. The platform is judged on adoption, and adoption is declared before the capabilities that would have prevented the skew bug exist.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The models are unchanged — fraud, search ranking, recommendations, forecasting. The platform's target is the shared set of capabilities every one of them needs, provided once so that a model team spends its time on the model rather than on rebuilding a registry.
- The outcome to optimise is time from a model team's idea to a safely deployed model, across all teams, without the platform becoming the bottleneck.
- Four training paths: two nightly cron jobs, one notebook run by hand, one managed cloud job. Three serving stacks. One feature computed four ways. Two teams share a GPU pool by messaging each other.
- The platform team's "data" is an inventory of what each team does and what each one has been burned by — the input to deciding which capabilities are shared pain and which are one team's preference.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A platform is a set of capabilities with a boundary. The shared capabilities are the ones every model team needs and none should build: dataset access with versioning, feature pipelines with a single definition per feature, training job submission with GPU scheduling, experiment tracking, a model registry with promotion policy, a serving path with canary and rollback, and monitoring that joins predictions to outcomes (The Model Registry, Model Monitoring).
- The boundary is what makes it work. The platform team owns the capability — the registry exists, has stages, enforces a policy — and the model team owns what goes in it: the features, the training code, the evaluation criteria, the monitors' thresholds, the on-call for the model's behaviour. The platform is paged when the registry is down; the model team is paged when the model is wrong.
- Each capability is a build-or-buy decision separately. A registry is simple to build and every vendor sells one; GPU scheduling is hard to build and the cloud provides it (Cloud ML Services). The platform team's job is often integration and defaults, not construction.
What is shared, what is owned
The platform question is a boundary question. Every capability an ML system needs can be placed on one side or the other: the platform provides it, or the model team builds it. The list is not the interesting part; the boundary is. A registry the platform runs but whose promotion policy each team sets is a different thing from a registry with one policy for everyone.
The matrix places the capabilities and says where the boundary usually falls — with the note that "usually" is doing work, and that the forecasting team's batch model legitimately needs a different answer for features and serving.
| Capability | Platform provides | Model team owns | Build or buy |
|---|---|---|---|
| Dataset access | Versioned snapshots, access control, a catalogue | Which snapshot, which label definition | Buy the storage; build the snapshot convention |
| Feature pipelines | One definition per feature, computed once, served to training and serving | The feature definitions themselves | Contested — see Feature Stores; a convention plus equivalence tests covers small teams |
| Training jobs | Submission, GPU scheduling, checkpointing, logs | The training code, hyperparameters, the smoke test | Buy the compute; build a thin template |
| Experiment tracking | A place to record runs, with the manifest schema enforced | What each run means and which one to promote | Buy or build; the schema is the value |
| Model registry | Stages, promotion policy enforcement, lineage | The evaluation criteria, the decision to promote | Build — it is small and the policy is company-specific |
| Serving | A path with canary, shadow, rollback and contract tests | The model, its preprocessing, its fallback behaviour | Buy the runtime; build the rollout policy |
| Monitoring | Prediction logging, outcome joins, drift computation | Thresholds, slices, what "wrong" means for this model | Build the joins; buy the dashboards |
| GPU scheduling | A pool with quotas, preemption, priority | How much they need and when | Buy — this is what the cloud is for |
The platform that became a ticket queue
The organisational failure has a mechanism. A platform team that owns the whole path is asked to make every change on it. The queue lengthens, the product teams' delivery slows, and they build a side path. Now the platform serves fewer teams than before and has to justify itself, so it mandates adoption — which lengthens the queue.
The escape is the boundary: the platform owns capabilities and defaults, and a team can deviate at the cost of owning what it deviates from. The golden path is fast because it is opinionated; deviation is allowed because mandates produce side paths anyway.
Every model change is a platform ticket. The platform team tunes monitor thresholds for models it does not understand, is paged when a model is wrong, and gates every deployment. Teams route around it within a year.
The registry, the training template, the serving path and the monitoring joins exist and are fast. A model team using them gets gates and rollback for free and owns its features, evaluation, thresholds and on-call. Deviation is permitted and owned.
Capabilities are shared and stable; models are specific and change weekly. Putting the specific, fast-changing part behind a shared team's queue serialises every team's work through the one team without the domain knowledge to do it.
What must stay true for the platform to keep paying
A platform earns its cost as long as its golden path is faster than the alternative and its boundary holds. Both erode. The golden path slows as it accumulates features for edge cases; the boundary erodes as the platform team, being helpful, takes on things the model teams should own.
The assumption to monitor is not "teams use the platform" but "teams using the platform ship faster and safer than teams that do not". When that stops being true, adoption numbers will keep looking fine for a while.
A model team using the platform's defaults reaches a monitored, canaried deployment faster than a team building its own path, and with the shared gates in place.
holds when The platform is built from the teams' shared pain, the template covers the common case, and deviation is possible without abandoning the whole path.
breaks when The platform grows a release cadence slower than the teams'; a required capability fits one team's needs and not another's; the platform team becomes the approver of model changes.
respond Find the capability the deviating team needed and either add it or explicitly make it a supported deviation. Do not mandate adoption; that produces the side path faster.
How to build it
Most important first.
- Start from the shared pain, not the ideal architecture: the two teams burned by the same skew bug need one feature definition per feature and a serving contract test. Build that. The forecasting team does not need it; do not migrate them to it.
- Define the boundary in writing per capability: what the platform guarantees, what the model team owns, who is paged for what. A platform without a written boundary becomes a ticket queue by default.
- Provide golden paths, not mandates: a model team that uses the platform's training job template gets tracking, registry and canary for free; a team with a reason to deviate can, and owns what it deviates from.
- Buy the hard, undifferentiated pieces — GPU scheduling, object storage, workflow execution — and build the thin, opinionated ones — the promotion policy, the feature-definition convention, the monitoring joins that encode how this company defines outcomes.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Time from a new model team's first commit to a canaried deployment with monitoring, using the golden path. This is what the platform is for.
- Number of distinct implementations of each shared capability across teams. It should fall; if it rises, teams are routing around the platform.
- Platform adoption percentage is the number that looks relevant and is not: a team forced onto an unsuitable path counts as adopted and ships slower.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The capabilities the platform provides are the ones the model teams actually share, checked against the inventory rather than against an ideal architecture.
- The boundary between platform and model team is written down per capability and reflected in who is paged, and it is reviewed when a team routes around the platform.
- The platform's golden path stays faster than the teams' own paths; the day it is slower, teams leave it.
- Offline: onboard a new model onto the golden path and time it. Compare with the last team that built their own. If the golden path is not faster, the platform is not yet a platform.
- Online: audit who was paged for the last five model incidents. If the platform team was paged for a model being wrong, the boundary has eroded.
- Over time: re-inventory the teams' stacks annually. The count of duplicate implementations is the platform's scorecard.
What can go wrong
- The platform becomes the only way to deploy, and its release cadence becomes every model team's release cadence.
- The boundary erodes: the platform team ends up owning monitor thresholds for models they do not understand, and alerts are tuned to silence rather than to the model.
- A capability is built because it is interesting rather than because it is shared pain — a custom training scheduler for two teams who were fine with the cloud one.
- A shared platform is a shared dependency: an outage in the registry stops every team's deployments at once.
- Golden paths encode one set of choices; a model that needs something else — streaming inference, a custom serving runtime — pays the cost of deviating.
- A platform team is headcount that is not shipping models, and its value is diffuse and slow to show; it will be questioned at every planning cycle.
- "Standardise everything and efficiency follows." Standardising a capability that only one team needs slows the others. Standardise shared pain.
- "The platform team owns MLOps." The platform owns capabilities. The model team owns its model's tests, monitors and evaluation. A platform that owns those becomes a bottleneck with no domain knowledge.
- "We need a platform before we can do ML properly." One model needs practices, not a platform. The platform is worth building when enough teams are re-implementing the same practice badly.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- SCALE-SPECIFICBelow roughly three models and two teams there is nothing to share and a platform is overhead; above a dozen models across several teams, the absence of a platform shows up as the same bug fixed four times. The transition, not the architecture, is the decision.
- CONTESTEDA serious position holds that building the platform before the third model is premature — that the first models should be built with good practices on plain infrastructure, and the platform extracted from what they had in common. The opposing view is that extracting a platform from three divergent stacks is harder than starting with one, and that the first team's choices become everyone's constraints either way; it is strongest where the teams are known in advance to have similar needs.
- DOMAIN-SPECIFICReal-time fraud and search need low-latency features and online serving; forecasting and churn are batch jobs over a warehouse. A platform designed for one set is a poor fit for the other, and companies with both often end up with two golden paths rather than one.
Where the depth lives
This domain teaches the model and hands the rest off by name.