The Model Supply Chain
Pretrained weights and public datasets are dependencies: unpinned, unhashed, unsigned, and loaded by formats that execute code. Pin, hash, sign, use safe formats, and record provenance in the registry.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A pretrained model was downloaded from a public hub, fine-tuned and deployed. What did the team just add to their dependency tree, and how would they know if it changed?
We fine-tune a public text encoder for our product-search model. The training script downloads "the latest" weights from a hub by name. Last month the fine-tuned model's quality dropped after a retrain and it took a week to find that the upstream weights had been updated. Our security team then asked what format the weights are in and whether loading them can run code, and we did not know.
Pretrained weights are a file, like an image asset. Download the latest, fine-tune, ship. The hub is reputable and the model card says what the model is.
The file changed under the same name. An upstream update — a legitimate one — altered the encoder's representation; the fine-tuned model trained on it was worse for this task, and nothing in the pipeline recorded that the input had changed (Reproducibility, Model Lineage).
- The file changed under the same name. An upstream update — a legitimate one — altered the encoder's representation; the fine-tuned model trained on it was worse for this task, and nothing in the pipeline recorded that the input had changed (Reproducibility, Model Lineage).
- The file is code. A serialisation format that reconstructs objects runs whatever the file says on load (Unsafe Deserialization). The training job loaded it with broad storage permissions; a tampered file would have had them too.
- The public evaluation dataset was also pulled by name, so the evaluation number is against a moving target, and a quality change could be the model, the weights or the benchmark.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The search model ranks products for a query; the label is a click within the session. The supply-chain target is that every third-party input to training — weights, datasets, libraries — is pinned to a specific hashed version, verified before use, loaded through a format that cannot execute code, and recorded in the registry so a change is a visible event rather than a mystery.
- A pretrained encoder pulled by name at training time; a public evaluation dataset pulled the same way; a pinned Python environment for libraries but not for model files. The registry records the fine-tuned artifact and the training commit, not the upstream weights' version or hash.
- The weights are in a serialisation format that reconstructs arbitrary objects on load, which is how many older model files were distributed.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A pretrained model is a dependency with the properties of a library — it changes behaviour when it changes — and the properties of data — it is large, opaque and produced by a process you did not run. Libraries get pinned versions, lockfiles, hashes and vulnerability scans (Dependency Pinning). Model files and datasets usually get none of those by default, because they are fetched by name from a hub at training time.
- The load path is the sharpest edge. Some serialisation formats store a program that rebuilds the object; loading them executes that program. A format that stores only tensors and metadata cannot. The choice of format is the difference between "a tampered file gives the attacker code execution inside the training job" and "a tampered file produces a bad model".
- Provenance closes the loop. The registry should record, for every artifact, the upstream weights' identifier and hash, the dataset snapshot hashes, and the environment lockfile — so a retrain that used different inputs is visibly a different lineage, and a quality change can be attributed (Artifact Integrity, The Model Registry).
What was added to the dependency tree
A training job's inputs are code, libraries, data and — when fine-tuning — someone else's weights. The first two are under the discipline software engineering built for dependencies. The last two usually are not, and they have larger effects on the output than any library upgrade.
The matrix lists each third-party input, how it is typically fetched, what can change without anyone noticing, and the control that makes the change visible and the file safe.
| Input | Typical fetch | What changes silently | Control |
|---|---|---|---|
| Pretrained weights | By name or tag from a public hub at training time | The file under the name; the representation; the licence; in a code-executing format, what runs on load | Pin by hash; mirror after verification; tensor-only format; provenance in the registry |
| Public dataset | By name from a hub or a URL | Rows added, removed or relabelled; the benchmark you evaluate against | Snapshot, hash and mirror; record the snapshot in the run manifest (Dataset Versioning) |
| Libraries | Package manager with a lockfile | Usually pinned — the exception is the model-loading library whose safe-format default changed between versions | Lockfile with hashes; scan; pin the loader version and its safe-load setting explicitly |
| Tokeniser and preprocessing assets | Bundled with the weights, fetched the same way | Vocabulary and normalisation rules, which silently change every feature | Hash with the weights; bundle in the artifact (Preprocessing Lives in the Artifact) |
| Your own fine-tuned artifact | Written to the registry, read by serving | Anything, if the registry write is not signed and the loader does not verify | Sign on write, verify on load; the same discipline as a build artifact (Artifact Integrity) |
A file that runs on load
The format question is the one the security team asked and the one worth answering first. A serialisation format that stores instructions for rebuilding an object gives whoever wrote the file the ability to run code in whatever process loads it. For a model file, that process is the training job — with its storage credentials — or the serving container.
The defence does not require imagining an attacker. It is a property of the format: choose one that stores tensors and metadata and nothing executable, enforce it in the loader, and convert legacy files once in an isolated environment. Security Engineering's Unsafe Deserialization lesson covers the general class; the ML-specific point is that model hubs distributed code-executing formats for years and many files still are.
1import hashlib2 3SAFE_SUFFIXES = (".safetensors",) # tensor-only formats; take the list from current docs4PINNED = {5 "text-encoder-base": {6 "revision": "a1b2c3d4", # a specific upstream revision, never "latest"7 "sha256": "e3b0c442...", # verified once, recorded in the registry8 "mirror": "object-store://mirror/text-encoder-base/a1b2c3d4/model.safetensors",9 },10}11 12def fetch_weights(name, store):13 spec = PINNED[name] # a name not in the pin table is an error, not a download14 if not spec["mirror"].endswith(SAFE_SUFFIXES):15 raise ValueError(f"{name}: refusing a format that can execute code on load")16 blob = store.read(spec["mirror"]) # the team's own mirror; the public hub is never contacted here17 digest = hashlib.sha256(blob).hexdigest()18 if digest != spec["sha256"]:19 raise ValueError(f"{name}: hash mismatch — upstream or mirror changed")20 return blob, {"upstream": name, "revision": spec["revision"], "sha256": digest}21 # the returned provenance dict goes into the run manifest and the registry recordThree refusals, in order: an unpinned name, an unsafe format, a wrong hash. Each one turns a silent change into a failed job with a reason.
Provenance is what makes a change visible
The quality drop took a week to diagnose because nothing recorded that the upstream weights had changed. With the upstream revision and hash in the registry record, the diagnosis is a diff between two artifacts' lineage. With them structured rather than free text, the reverse question — which of our models used a revision now known to be bad — is a query.
The assumption device names what has to stay true after the controls are in place, and how the team would notice if the fetch path was bypassed.
A training run can only use upstream weights and datasets that were fetched by hash from the verified mirror in a tensor-only format, and the registry record names them.
holds when The pin table is the only fetch path; the loader's format allow-list is enforced in code; the registry rejects an artifact whose provenance fields are empty; training jobs have no egress to public hubs.
breaks when A notebook fetches by name for an experiment and the resulting artifact is promoted; a loader library upgrade changes its safe-load default; the mirror job starts fetching "latest" to save effort; a provenance field becomes optional to unblock a release.
respond Treat a bypass as a supply-chain incident: identify every artifact trained through it, re-verify their inputs, and close the path — usually by removing hub egress rather than by asking people to stop.
How to build it
Most important first.
- Pin every third-party input by content hash, not by name or tag. The training script fetches a specific revision, verifies the hash before use, and fails if it does not match.
- Load weights only through formats that cannot execute code — tensor-only formats — and refuse the others in the training and serving loaders. Convert once, verify, and store the converted file in your own registry.
- Mirror upstream weights and datasets into your own artifact store after verification, so the training job never fetches from the public hub at run time and an upstream change is a deliberate update on your side.
- Record provenance in the registry: upstream identifier, hash, licence, the date it was verified and by whom, and the environment lockfile. Sign your own artifacts on write and verify on load, the same as any build artifact (Software Supply Chain Security in Security Engineering owns the general discipline).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Fraction of production artifacts whose full lineage — upstream weights hash, dataset hashes, lockfile — is recorded and verifiable. This is the number that says whether a change would be visible.
- Count of code-executing model formats in the loaders' allow-list: zero.
- The hub's download count or reputation is the number that looks relevant and is not: it says nothing about whether the file you loaded is the one you evaluated.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every third-party weight file and dataset used in training is fetched from the team's own verified mirror by hash, and the public hub is never contacted at training time.
- The training and serving loaders accept only tensor-only formats, and the allow-list is enforced in code rather than by convention.
- The registry's provenance fields are structured, so an upstream identifier or hash can be queried across all artifacts when an upstream problem is announced.
- Offline: change one byte of a mirrored weight file in staging and run the training job; it must fail at hash verification. Attempt to load a code-executing format through the loader; it must refuse.
- Online: audit the network egress of a training job; a connection to a public hub is a fetch that bypassed the mirror.
- Over time: when an upstream model or dataset is updated or withdrawn, query the registry for every artifact that used the old revision. If the query cannot be answered, provenance is not structured.
What can go wrong
- The hash is pinned in the training script and the mirror is populated by a separate job that fetches by name, so the mirror drifts and the hash check fails only when someone looks.
- The loader refuses code-executing formats, and a convenience script for "trying out" a new model bypasses it on a laptop with production credentials.
- The provenance record exists and is free text, so a lineage query for "which models used upstream revision X" cannot be answered when X is found to be compromised.
- Mirroring and hashing upstream weights is storage and a verification step that delays trying a new model by hours, which researchers will resent.
- Refusing code-executing formats excludes some older models until they are converted, and conversion is itself a step that needs a trusted environment.
- Structured provenance is a schema the registry must enforce, and enforcement means a retrain fails when a field is missing.
- "The hub is reputable, so the file is safe." Reputation is about the publisher. The file you loaded is whatever was at that name on that day, in a format that may execute on load. Pin, hash and use a safe format regardless of publisher.
- "We pin our Python packages, so we are pinned." Packages are one class of dependency. Model files and datasets fetched by name at training time are unpinned by default and larger in effect.
- "The quality drop was a training regression." It was an input change. Without upstream provenance in the registry, the two are indistinguishable, which is the finding.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a fetched model file is a dependency with library-like effects on behaviour and data-like opacity holds for every pretrained model and public dataset, on any hub, in any framework.
- FRAMEWORK-SPECIFICWhich serialisation formats execute code on load and which are tensor-only is a property of the framework and its versions, and the safe-format allow-list has to be taken from current documentation rather than from memory; the principle — refuse formats that reconstruct arbitrary objects — is stable.
- SCALE-SPECIFICA team fine-tuning one public model can verify and mirror it by hand once; a team that evaluates dozens of upstream models a quarter needs the mirror, hash check and format refusal automated in the fetch path, or the convenience script will bypass them.
Where the depth lives
This domain teaches the model and hands the rest off by name.