The Model Registry
The registry is a state machine over artifacts — Candidate, Registry, Staging, Production, Archived — that stores lineage, metrics, approvals and the feature-definition version, so "which model is live" has one answer.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Three teams retrain the same model on different schedules. How does anyone know which artifact is in production, what it was trained on, and who approved it?
A support-routing team was asked by a compliance reviewer which model version classified a specific ticket in March, what data it was trained on, and who signed off on it. The answer took nine days, involved reading deploy logs and a shared spreadsheet, and turned out to be a version nobody remembered promoting.
Store artifacts in object storage with a sensible naming convention, keep a latest pointer, and let the deploy pipeline pull latest. A spreadsheet records what shipped when. Simple, and it works while one person does all the promoting.
Two pipelines write latest within an hour of each other; the deploy pulls whichever won, and the spreadsheet records the one the engineer thought they promoted.
- Two pipelines write
latestwithin an hour of each other; the deploy pulls whichever won, and the spreadsheet records the one the engineer thought they promoted. - The artifact in production has no manifest, so the compliance question — trained on what, evaluated how — has no answer that is not a reconstruction from commit history.
- A rollback restores the previous image tag, which points at a bucket path whose contents were overwritten by a later run with the same date prefix.
- The feature service was upgraded between training and serving; nothing recorded which feature-definition version the artifact was trained against, so the skew is discovered from the routing accuracy weeks later (Train / Serve Skew).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Route an incoming support ticket to one of a dozen queues; the label is the queue that ultimately resolved it, which is known after the fact and sometimes after a reassignment.
- The organisation needs to answer, for any past decision, which artifact produced it and why that artifact was allowed to.
- Artifacts arrive from three training pipelines — a weekly scheduled retrain, an ad hoc run when a new product launches, and a research branch — each writing to its own bucket with its own naming.
- Each artifact should carry a manifest (What a Model Artifact Contains); in practice the weekly run writes one, the research branch does not, and the launch run writes a different schema.
- Deployment is a Kubernetes manifest with an image tag; the tag encodes a date, not an artifact hash.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A registry is a database of immutable artifact entries plus a state machine over them. An entry is created once, from a content-hashed bundle, and never modified; its *stage* changes — Candidate → Registered → Staging → Production → Archived — and every transition records who, when, and on what evidence.
- The entry stores what promotion and audit need: the lineage (Model Lineage — dataset version, code commit, feature-definition version, parent run), the evaluation metrics with the split they were computed on, the approvals, and the stage history.
- Serving resolves "the production model" by asking the registry for the entry in the Production stage of a named model, and loads it by hash — not by path, not by tag, not by
latest. There is exactly one such entry per model at a time, which is what makes the question answerable. - The registry is not the store. Bytes live in object storage; the registry holds the hash, the location and the state. That separation is what lets the same bytes be referenced from Staging and Production during a rollout without copying.
A state machine, not a folder
The registry's job is to make one question cheap: which artifact is allowed to answer requests for this model right now, and on what evidence. A folder with a latest pointer answers a different question — which file was written most recently — and the two diverge the first time two pipelines write in the same hour.
Modelling the lifecycle as explicit states with guarded transitions is what makes the answer unambiguous. Each transition has a precondition and leaves a record; Production is reached only through Staging, and reaching it demotes the previous occupant to Archived in the same transaction.
- 1Candidate
A training run writes a content-hashed bundle and registers it with its manifest. Loose requirements: the run exists and is identifiable.
fails by The research branch writes to a bucket and never registers; its artifacts are invisible to the gate and get promoted by hand.
- 2Registered
Manifest validated: lineage, metrics on a named split, feature-definition version, replay sample present. Eligible for evaluation.
fails by The manifest schema is enforced loosely so the research branch "can iterate", and lineage fields arrive empty.
- 3Staging
Loaded in the serving image; replay, shadow or canary evidence is collected and attached to the entry.
fails by Staging is treated as a label meaning "probably fine" and entries sit there for months with no evidence attached.
- 4Production
The promotion checklist passed; the entry becomes the unique Production entry and the previous one is archived in the same transaction.
fails by Two entries in Production because demotion is a separate manual step someone forgot.
- 5Archived
Immutable, still loadable for rollback and for answering audit questions about past predictions.
fails by A retention job deletes the bytes but not the entry, so rollback resolves to a hash that no longer exists.
The transitions are the product. The states are just the names of the gaps between them.
What the entry has to hold
The registry stores what the gate and the audit need, and nothing they do not. Lineage answers "trained on what"; metrics on a named split answer "evaluated how"; the feature-definition version answers "against which serving contract"; the approvals answer "who"; and the hash answers "which bytes". A registry missing any one of these will be supplemented by a spreadsheet, and the spreadsheet will be wrong.
The feature-definition version deserves attention because it is the field teams forget. The artifact was trained against a specific computation of each feature; if the serving feature path is at a different version, the weights meet inputs they never saw. The registry cannot check this alone, but it can hold the value the check needs.
1model: ticket-router2version: 373sha256: 4f9c…e14location: s3://models/ticket-router/4f9c…e1/5stage: Production6lineage:7 dataset: tickets@2026-08-10 # dataset version, not a table name8 features: fdef-12 # the feature-definition version served9 commit: 9b7d2e110 parent_run: run-2026-08-11-weekly11metrics:12 split: holdout-2026-08 # named, so the number has a referent13 macro_f1: {value: "…", ci: "…"} # with uncertainty, see metric-uncertainty14 p99_latency_ms: "…"15 memory_mb: "…"16history:17 - {to: Registered, at: 2026-08-11T03:12Z, by: pipeline/weekly}18 - {to: Staging, at: 2026-08-11T09:40Z, by: d.ortiz, evidence: shadow-run-2211}19 - {to: Production, at: 2026-08-13T14:05Z, by: d.ortiz, evidence: gate-2026-08-13, demoted: 36}Notice that the split is named and the metric carries an interval. A metric without a split is a number about nothing, and a comparison without an interval cannot say whether the challenger is better (Metric Uncertainty).
The assumption serving makes
Serving assumes the registry's Production entry is the artifact it should load, and that loading by hash yields the bytes that were evaluated. Both are cheap to check continuously and expensive to discover broken: a serving process that bypassed the registry looks identical to one that did not, until the prediction log and the stage history disagree.
The reconciliation between the hash serving logs and the entry the registry holds is the monitor that makes the registry trustworthy. Without it, the registry describes intent; with it, the registry describes production.
Every prediction served for a model was produced by the artifact the registry held in the Production stage at that instant, loaded by content hash.
holds when Serving resolves the entry through the registry API at rollout, loads by hash, logs the hash per prediction, and no deployment path exists that names a bucket path or image tag directly.
breaks when A hotfix points serving at a path; a rollback restores an image whose baked-in artifact predates the registry; two entries end up in Production because demotion is manual.
respond Treat a mismatch as an incident, not a data-quality note: identify which artifact actually served, register it retroactively with the evidence it lacks, and close the deployment path that bypassed the registry.
How to build it
Most important first.
- Make the registry the only path to production: the serving config names a model and a stage, the deploy resolves it to a hash at rollout, and a bundle that was never registered cannot be loaded (Artifact Integrity).
- Require the manifest at registration. An artifact without lineage, metrics on a named split and a feature-definition version is rejected at the door, which forces the research branch to conform before it can be evaluated for promotion.
- Encode the promotion gate as the transition rule from Staging to Production, with the checklist stored on the entry (Promotion Is a Checklist, Not a Score); a transition without the recorded evidence is not possible, not merely discouraged.
- Log the artifact hash on every prediction (Prediction Logging) so the registry's stage history plus the prediction log answers "which model made this decision" in one join (Tracing a Prediction).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Time to answer "which artifact produced this prediction, trained on what, approved by whom" — the registry exists to make that a query, and if it is still an investigation the registry is decorative.
- The count of production predictions whose logged artifact hash does not match the registry's Production entry at that time. It should be zero; any non-zero value is a serving process that bypassed the registry.
- Not the number of registered models or the size of the stage history. A registry full of entries nobody promotes is a bucket with a UI.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- There is exactly one Production entry per model at any moment, and serving loads that entry by hash rather than by name, tag or path.
- Every stage transition is recorded with its actor and evidence, and no path exists from a training run to a serving process that does not pass through a registered entry.
- The feature-definition version recorded on the entry can be compared to the version the serving feature path is actually running.
- Offline: a test that registers an artifact without a manifest and asserts rejection; a test that attempts Staging → Production without the checklist evidence and asserts refusal.
- Online: a continuous reconciliation between the hash logged by serving and the registry's Production entry, alerting on mismatch within minutes.
- Over time: pick a random past prediction each week and answer the compliance question from the registry and the prediction log alone; if it needs a human's memory, the registry is missing a field.
What can go wrong
- The registry becomes a bottleneck people route around: a hotfix is deployed by pointing serving at a bucket path "just this once", and the stage history now lies.
- Stages are reused as labels rather than states — three entries "in Production" for the same model because the transition rule does not demote the previous champion.
- The registry records the feature-definition version, but the feature service does not expose its own version at runtime, so the mismatch check has nothing to compare against (Feature and Model Versioning).
- A registry is a service with uptime requirements: if serving cannot resolve the Production entry at startup, serving cannot start, so the resolution must be cached or the registry must be as available as the model.
- Requiring a manifest at registration slows research iteration, which is why the research branch resists it; the answer is a Candidate stage with loose requirements and a strict gate later, not a loose registry.
- Immutable entries mean a mistaken registration is archived, not deleted, and storage grows; retention policy becomes a real decision (Dataset Versioning).
- "We use a registry, so we have lineage." A registry stores what it is given. If the training pipeline does not write the dataset and feature-definition versions, the registry holds a hash and a date.
- "The registry is the MLOps platform." It is one table and one state machine. Training orchestration, feature serving and monitoring are separate concerns; a registry that tries to own them becomes the bottleneck people route around (What MLOps Is).
- "Staging means tested." Staging means the entry is eligible to be tested — shadow, canary, replay (Shadow Deployment, Canary Rollout). The evidence those produce is what moves it, and the stage name does not imply the evidence exists.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALOne immutable entry per artifact, one Production entry per model, and every transition recorded — that shape holds whether the registry is a managed service, a database table or a directory of signed manifests.
- SCALE-SPECIFICWith one model and one engineer, a directory convention plus a changelog is a registry and a service is overhead; once several pipelines produce candidates for the same endpoint, or an auditor can ask about a past decision, the state machine has to be enforced by software.
- CONTESTEDA credible position holds that a dedicated model registry duplicates the artifact registry and release tooling the DevOps platform already provides, and that a model should be promoted exactly like any other build artifact — image digest, environment promotion, the same approvals. That is right for the state machine; what the generic registry lacks is the ML-specific evidence the gate needs — metrics on a named split, the feature-definition version, the champion comparison — and teams that go generic tend to keep that evidence in a spreadsheet.
Where the depth lives
This domain teaches the model and hands the rest off by name.