Model Lineage
Raw data → Dataset v12 → Features v7 → Training Run 482 → Model v19 → Production. The graph that answers "which data did the production model learn from" during an incident, recorded by machines rather than remembered by people.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
When a production model misbehaves, how do you trace what it learned from, and what has to be recorded for that trace to exist?
A fraud model started approving a pattern of transactions it used to block. Finance wants to know by end of day whether the model was trained on data from the period when a labelling vendor was known to be mislabelling, and which other models were. The engineer who trained it left in June.
Ask the person who trained it. Look at the notebook. Check the training script's query for the date range. It is one model; someone knows.
The person left. The notebook reads a table that has been backfilled twice. The date range in the script is a variable that was overridden on the command line and the override is not recorded anywhere (Reproducibility).
- The person left. The notebook reads a table that has been backfilled twice. The date range in the script is a variable that was overridden on the command line and the override is not recorded anywhere (Reproducibility).
- Even with the dataset known, whether it contained the vendor's mislabelled period is a question about which raw partitions fed which dataset version, which is the data platform's lineage, and it was never joined to the model's.
- "Which other models" is a reverse query — from a raw source to every artifact downstream — and nothing exists that can answer it; the team polls every model owner on a channel.
- The incident closes with a retrain on "clean" data whose cleanliness is asserted, not traced, and the same question comes back at the next incident (ML Incident Debugging).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding model predicts fraud; the lineage target is that for the artifact serving right now, every upstream input — training run, feature version, dataset and label version, raw sources — can be resolved from recorded metadata in minutes, and the query can be run in reverse: given a bad input, which models are affected.
- The label of "what the model learned from" is a path through a graph, not a sentence someone remembers.
- A registry entry for the production artifact with a digest and, if tracking was done, a run id (The Model Registry).
- A run record naming dataset, label and feature versions (Experiment Tracking); a dataset manifest naming raw partitions; a data platform lineage graph from raw sources to the dataset (Data Lineage).
- The incident: a business-metric question about a specific period of upstream data, arriving with a deadline.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Lineage is a directed graph. Nodes are immutable, identified things: raw partitions, dataset versions, label versions, feature-definition versions, training runs, artifacts, deployments. Edges are "produced from" relations, recorded at the moment each node is produced by the process that produced it.
- The forward query — from a raw source to every downstream model — is what answers "which models are affected". The backward query — from a serving artifact to its raw sources — answers "what did this model learn from". Both are graph traversals if the edges exist and nothing if they do not (Impact Analysis).
- The edges are only trustworthy if each was written by the producing process with resolved identifiers: the dataset manifest lists the raw partitions it read; the run record names the dataset, label and feature versions; the registry entry names the run; the deployment names the artifact digest (Artifact Integrity). A lineage graph filled in by hand after the fact is the notebook problem with better formatting.
The graph, from raw data to production
Each node is something immutable with an identifier; each edge was written by the process that produced the downstream node. The path from the serving artifact back to raw partitions is the answer to "what did it learn from"; the path forward from a raw partition is the answer to "which models are affected".
The Follow One Prediction walkthrough at /ml/prediction traces a single request from the product through features, artifact and outcome; lineage is the same trace applied to the artifact itself — where it came from rather than where its prediction goes.
Recorded by the producer, not remembered by the trainer
Every edge has a natural author: the process that produced the downstream node knows exactly what it consumed. The snapshot job knows which partitions it read; the training entry point knows which dataset version it loaded; the promotion step knows which run it promoted. Writing the edge there costs a few lines and is the only way it is correct.
The two traversals below assume that. Given the edges, the incident question is a query that returns in seconds, and the reverse question — the one that finds the other affected models — is the same query run the other way.
1# edges: (child, parent) written by the process that produced child2# e.g. ("ds-v12", "raw/2026-01-14"), ("run-482", "ds-v12"),3# ("run-482", "feat-v7"), ("model-v19", "run-482"),4# ("deploy-2026-02-20", "model-v19")5 6def upstream(node, edges):7 """What did this model learn from?"""8 seen, stack = set(), [node]9 while stack:10 n = stack.pop()11 for child, parent in edges:12 if child == n and parent not in seen:13 seen.add(parent); stack.append(parent)14 return seen15 16def downstream(node, edges):17 """Which models consumed this source?"""18 seen, stack = set(), [node]19 while stack:20 n = stack.pop()21 for child, parent in edges:22 if parent == n and child not in seen:23 seen.add(child); stack.append(child)24 return seen25 26bad_partitions = [p for p in raw_partitions if vendor_period(p)]27affected = set().union(*(downstream(p, edges) for p in bad_partitions))28affected_models = {n for n in affected if n.startswith("model-")}The traversal is trivial. The work is in the edges existing, being written with immutable identifiers by the producing process, and the deployment node being present — without it the graph knows which models were trained on the bad period but not which one was serving.
What must stay true for the graph to answer
The graph is only as complete as its weakest producer. One model trained outside the pipeline, one deployment by mutable tag, one dataset version that is a table name, and the traversal stops at that node with no answer.
The assumption to monitor is completeness in the production path, checked at promotion — where an incomplete graph can still be refused — rather than at the incident, where it can only be regretted.
For the artifact currently serving, an unbroken chain of recorded, resolvable edges exists back to raw partitions and forward to the deployment, and every node in it is immutable.
holds when Each producing step emits its edge with resolved identifiers; promotion refuses an artifact with a missing or unresolvable upstream; deploys record the artifact digest; retention on nodes is tied to live deployments.
breaks when A hotfix model bypasses the pipeline; the serving config references a tag rather than a digest; a dataset version is a name; run records or manifests are expired by age-based retention.
respond Refuse the promotion or, for a serving model with a broken chain, treat it as untraceable in incident planning — decide now what would be done if a bad source is found, because the graph cannot say whether this model consumed it.
| Trigger | Symptom | Cause | Response |
|---|---|---|---|
| Hotfix trained outside the pipeline | Serving artifact has no run record | The producing process was a laptop | Promotion gate on a resolvable run id; no exceptions for hotfixes |
| Deploy by mutable tag | Graph knows the model, not which artifact served in the window | Deployment node records a name, not a digest | Record digest at deploy; compare to registry on a schedule |
| Dataset version is a table name | Upstream resolves to rows that have since changed | No manifest; the edge points at a moving target | Content-hashed manifest listing raw partitions |
| Run records expired | Chain breaks at a model eight months old | Retention by age, not by deployment | Tie retention to live and recent registry entries |
How to build it
Most important first.
- Make every producing step emit its edge: the snapshot job writes the raw partitions it read; the feature pipeline writes the dataset version it consumed; the training entry point writes the run record; promotion writes the run id into the registry; deploy writes the artifact digest into the serving config.
- Use immutable identifiers on every node — content hashes for data, versions that are never reused for definitions, digests for artifacts — so an edge resolves to the same thing later (Dataset Versioning, Feature and Model Versioning).
- Join the model-side graph to the data platform's lineage at the dataset node, so a question about a raw source can traverse into models and back (Column-Level Lineage).
- Expose both traversals as a query, and rehearse them: "what did model X learn from" and "which models consumed source Y between these dates" should be answerable by someone on call who did not train anything.
- Include deployments and time in the graph: which artifact served which traffic when, so an incident window maps to an artifact and from there upstream (Tracing a Prediction).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Time to answer, for the production artifact, "which dataset, label and feature versions, and which raw partitions" — by someone who did not train it. Minutes means lineage exists; a day means it does not.
- The fraction of registry entries whose run id resolves to a complete run record whose dataset version resolves to a manifest. This is lineage coverage, and gaps are where the next incident is unanswerable.
- Do not measure "we have a lineage tool". A tool with edges drawn by hand answers questions with what someone remembered.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every edge in the graph was recorded by the process that created the node, with identifiers that resolve to immutable things, and no node in the production path was created outside that process.
- The artifact recorded as deployed is the artifact that served — verified by digest, not by name — for the whole window in question.
- The data platform's lineage from raw sources to the dataset version is itself complete and joins to the model graph at a shared identifier.
- Offline: a lineage test at promotion — walk from the candidate artifact back to raw partitions and fail if any edge is missing or unresolvable (Promotion Is a Checklist, Not a Score).
- Online: a periodic drill — pick a serving model, answer both questions from the graph, time it; and a digest comparison between the deployed artifact and the registry entry.
- Over time: lineage coverage per month; an alert when a serving artifact's upstream nodes approach retention.
What can go wrong
- Lineage exists for the training data but the serving config points at an artifact by a mutable tag, so the graph does not know which artifact was actually serving during the incident window.
- The data platform's lineage and the model registry use different identifiers for the same dataset, and the join is a spreadsheet.
- Retention on run records is shorter than the life of the models they describe.
- A hotfix model is trained outside the pipeline "just this once" and deployed with no run record; it is the one serving during the incident.
- Every producing step gains a write to a lineage store, and every identifier has to be immutable, which pushes versioning discipline onto the data platform and the feature team, not just the ML team.
- Retaining run records, manifests and snapshots for the life of every model is storage and governance work with no visible payoff until an incident.
- A hard promotion gate on complete lineage blocks the hotfix that would have shipped in an hour; the argument for the gate is that the hotfix is the model nobody can explain later.
- "We know what data the model used — it is in the training script." The script is a function of parameters that were overridden, reading tables that have changed. It records intent at one moment, not what the run consumed.
- "Lineage is a data-engineering concern." The data platform's lineage ends at the dataset. The edges from dataset to run to artifact to deployment are the model's to record, and without them the platform's graph cannot say which model is affected.
- "We only have three models; we can remember." The incident arrives when the person who remembers has left, and the question is about a model trained eight months ago. Lineage is recorded for the reader who was not there.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALEvery deployed model has an upstream graph whether or not it is recorded; the question "what did it learn from" is task- and model-independent, and the shape of the answer — data version, feature version, run, artifact, deployment — is the same everywhere.
- DOMAIN-SPECIFICIn regulated domains — credit, insurance, health — lineage is an audit requirement with a legal deadline, and the graph must reach back to raw sources with retention to match; in a recommender it is an engineering convenience that becomes an incident tool.
- CONTESTEDA serious position holds that full lineage infrastructure is a platform investment few teams can justify, and that a disciplined run record with immutable dataset and artifact identifiers gives most of the backward query for a fraction of the cost. That is largely right for the backward direction; the counter is that the forward query — which models did this bad source reach — is the one incidents actually ask, and it needs the join to the data platform's graph that run records alone do not provide.
Where the depth lives
This domain teaches the model and hands the rest off by name.