Experiment Tracking
Every run records the code, the data, the features, the configuration, the metrics, the artifacts and the environment. A number without that record is a claim nobody can check.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What must be recorded about a training run for its metric to mean anything a month later?
A team lead asks which of the last quarter's churn models was best. Three engineers each have a notebook with a number in it. Two of the numbers were computed on different validation sets, one was computed before a label fix, and nobody can say which data any of them used. "We need to pick one to ship by Friday."
Keep the notebooks. Put the best number in the summary cell, commit the notebook, and note the run in a spreadsheet with the date and the metric. When it matters, re-run the notebook.
The notebook re-run produces a different number, because the warehouse table it reads was backfilled last week, the feature repo moved on, and the environment has a new version of the boosting library (Reproducibility).
- The notebook re-run produces a different number, because the warehouse table it reads was backfilled last week, the feature repo moved on, and the environment has a new version of the boosting library (Reproducibility).
- Two of the three candidate metrics were computed on different validation sets and cannot be compared; the comparison happens anyway, on Friday, because there is a deadline.
- The model that ships was trained on the pre-fix labels, which nobody can tell from the artifact, and it underperforms in production for a reason that takes a month to find (Model Lineage).
- Six months later an incident asks "what data did the production model learn from" and the answer is a Slack thread (ML Incident Debugging).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The surrounding system predicts thirty-day churn; the tracking system's target is that for any run, the question "what produced this metric" has a complete, machine-readable answer.
- The label being tracked here is the metric itself — and a metric is a property of code, data, split, features and configuration together, not of a model file.
- Runs: each one a training job with a code state, a dataset, a feature pipeline, hyperparameters, a metric on some evaluation set, and output artifacts. Dozens a week per engineer during a project.
- The inputs live in different places — a git repository, a warehouse table, a feature repo, a YAML file, a laptop's Python environment — and nothing joins them except the engineer's memory.
- The evaluation sets change under the runs: a label fix, a new split, a filtered cohort. Two numbers a week apart are not on the same scale.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A training run is a function:
f(code, data, features, config, environment, seed) → (metrics, artifacts). A metric is comparable to another metric only when the evaluation inputs are the same, and reproducible only when all the inputs are recorded precisely enough to be re-obtained. - Tracking means writing the inputs and outputs of that function to a store, per run, with identifiers that resolve to immutable things: a commit hash, a dataset version hash, a feature-definition version, a config file's content, a resolved dependency list, an artifact digest (Dataset Versioning, Feature and Model Versioning).
- The value of the record is in the join: the store lets you ask "all runs on dataset v12 with feature pipeline v7, sorted by validation metric" and get an answer that compares like with like. Without the identifiers the join is impossible and the comparison is a guess.
What a run record contains
The columns are the inputs and outputs of the training function. Each input column holds an identifier that resolves to an immutable thing; each output column holds a value or a digest. A record with any input column empty describes a run that cannot be re-obtained.
Two runs are comparable when their evaluation-set columns match. The table makes that check a filter rather than a conversation.
| Field | Example | Why it must be immutable |
|---|---|---|
| Code version | commit a3f9c1e, clean tree | A dirty tree or a branch name resolves to different code tomorrow |
| Dataset version | Dataset v12, content hash | A table name resolves to whatever the last backfill left |
| Label version | Labels v3 | A label fix changes every metric without touching the model |
| Feature pipeline version | Features v7 | The same feature name with a new definition is a different input |
| Hyperparameters | resolved config, not the defaults file | Overrides on the command line are part of the run |
| Environment | lock file hash, CUDA version, GPU model | A library upgrade can change the number (Reproducibility) |
| Seeds | split, init, shuffle, dropout | Necessary and not sufficient (Random Seeds) |
| Metrics | per named eval set version | A metric without its eval set version is not comparable |
| Artifacts | digest of model + preprocessing | What was actually produced, verifiable later |
Recording from the entry point, not from memory
The record must be written by the code that runs, at the start, with resolved values. A tracking call at the end of a script misses every failed run; a record filled in by hand afterwards captures what the engineer believes was passed.
The resolving is the work. A dataset name has to become a snapshot identifier; a config has to be the merged result of file and overrides; the environment has to be the lock actually installed, not the one in the repository.
1import hashlib, json, subprocess, sys2 3REQUIRED = ["commit", "dataset_version", "label_version", "feature_version",4 "config", "env_lock_sha", "seeds", "eval_sets"]5 6def start_run(cfg, dataset, features, seeds, eval_sets):7 rec = {8 "commit": subprocess.check_output(["git", "rev-parse", "HEAD"]).decode().strip(),9 "dirty": bool(subprocess.check_output(["git", "status", "--porcelain"]).strip()),10 "dataset_version": dataset.snapshot_id, # a hash, not a table name11 "label_version": dataset.label_version,12 "feature_version": features.definition_version,13 "config": cfg.resolved(), # after overrides14 "env_lock_sha": hashlib.sha256(open("uv.lock", "rb").read()).hexdigest(),15 "python": sys.version, "seeds": seeds,16 "eval_sets": {name: es.version for name, es in eval_sets.items()},17 }18 missing = [k for k in REQUIRED if rec.get(k) in (None, "", {})]19 if missing or rec["dirty"]:20 raise RuntimeError(f"refusing to start: missing {missing}, dirty={rec['dirty']}")21 tracker.write(rec) # written before training, so a killed run still has one22 return recThe refusal is the point. A run with a dirty tree or an unversioned dataset is allowed to happen in a notebook; it is not allowed to become a record that a promotion decision might later read.
What must stay true for the record to be worth reading
The record is a set of pointers. Its value depends on the things it points at not changing: a commit is safe, a dataset snapshot is safe if the platform enforces immutability, a "latest" alias is not safe at all.
The failure is a complete-looking record whose identifiers quietly resolve to something else. That is worse than no record, because it is trusted.
Each input identifier in a run record — commit, dataset version, feature version, environment lock, artifact digest — resolves to exactly what the run used, indefinitely.
holds when Dataset and feature versions are content-addressed or immutable snapshots; environments are locked and the lock hash is recorded; artifacts are stored by digest (Artifact Integrity).
breaks when A dataset version is a table name or a date partition that a backfill rewrites; a feature version is a branch; the artifact store allows overwrite by name.
respond Fix the versioning at the source — snapshot the dataset, pin the feature definition — and mark affected runs as non-reproducible rather than deleting them; they are still evidence, just weaker.
The metric and the date in a spreadsheet; the notebook committed with its outputs; the data read live from the warehouse.
Every run writes a record from its entry point with commit, dataset snapshot, feature version, resolved config, environment lock, seeds, eval-set versions, metrics and artifact digests; the registry entry points at the record.
A spreadsheet row cannot be re-obtained and two rows cannot be compared, because the inputs that would make either possible are not in it. The record makes "which was best" a query and "what did it learn from" a lookup, which are the two questions that arrive on Friday and during the incident.
How to build it
Most important first.
- Record automatically, from the training entry point, not by hand: the commit and dirty state, the dataset and label version, the feature pipeline version, the full resolved config, the environment lock, the hardware, the seed(s), every metric on every named evaluation set, and the artifact digests.
- Name evaluation sets and version them like datasets, so a metric is always "metric X on eval set Y version Z" and two runs on different sets cannot be silently compared.
- Make a run that omits any required field fail, rather than log a partial record. A run with a missing dataset version is precisely the one that will matter later.
- Store the record next to the artifact's registry entry so promotion reads it (The Model Registry); the tracking store is the registry's memory.
- Report metrics with the variance across seeds when the difference between candidates is small (Random Seeds, Metric Uncertainty).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The fraction of runs in the last month with a complete record — every required field present and resolvable. This is the number that says whether the tracking works.
- Whether a given production artifact can be traced to a run, and that run to its dataset, feature and code versions, in under a minute, by someone who did not train it.
- Do not measure the number of runs logged. A thousand partial records are a thousand anecdotes.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every identifier in a run record resolves to something immutable: the commit exists, the dataset version is a snapshot that cannot change, the feature version is pinned, the artifact digest matches the file.
- The record was written by the code that ran, at the moment it ran, with the resolved values — not reconstructed afterwards from what the engineer remembers passing.
- The evaluation set named in the record is the one the metric was computed on, and it has not been touched by tuning.
- Offline: pick a run at random, re-obtain every input from the record alone, retrain, and compare the metric within the seed-variance band; a run that cannot be re-obtained from its record is a tracking bug.
- Online: for the model in production, resolve its registry entry to a run record and from there to every input version, as an exercise done by someone new.
- Over time: a monthly count of runs with unresolvable identifiers; a registry promotion that references a run with a missing field should be blocked by the promotion pipeline.
What can go wrong
- The tracking call is at the end of the script, so failed and killed runs — most of the interesting ones — leave no record.
- Engineers log a "dataset name" that points at a mutable table; the record is complete and resolves to something that has changed since.
- The config is logged as the defaults file, not the resolved values after command-line overrides; the recorded run is not the run that happened.
- The tracking store becomes a leaderboard, and the leaderboard drives tuning on the evaluation set until the recorded numbers are optimistic by construction (Never Tune on the Test Set).
- Automatic tracking from the entry point is a constraint on how training is launched — ad-hoc notebook cells stop being runs — and engineers experience that as friction during exploration.
- Immutable dataset and feature versions are storage and a versioning discipline in the data platform that the ML team does not own and has to negotiate for.
- A store of every run with every metric invites selection: the recorded best of a hundred runs on the same evaluation set is a biased estimate, and the tracking makes it easier to produce.
- "We use a tracking tool, so we are tracking." A tool that stores whatever it is given stores partial records. Tracking is the discipline of what must be recorded and the enforcement that it was; the tool is where it goes.
- "The notebook is the record." The notebook is the code state at the last time it was edited, reading data that has since changed, in an environment that has since moved. It records intent, not what ran.
- "The best run in the leaderboard is the best model." It is the run whose metric happened highest on that evaluation set, among many attempts. Without the seed variance and the number of attempts, the leaderboard's top is an optimistic estimate.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a metric is a property of the whole input tuple holds for any model family and any task; the fields to record differ in detail — a fine-tuning run adds the base model's version, a forecasting run the cutoff date — but the shape is the same.
- SCALE-SPECIFICA single engineer with one model can hold the record in their head for a while; the discipline becomes load-bearing at the second engineer, the second model, or the first incident, whichever comes first.
- CONTESTEDA serious position holds that heavy tracking slows exploration — most runs are throwaway, and forcing a complete record on every one taxes the phase where speed matters most. The counter is that the throwaway run is indistinguishable from the important one until later, and that automatic recording from the entry point costs nothing per run once set up; the friction is real only when tracking is manual.
Where the depth lives
This domain teaches the model and hands the rest off by name.