Artifact Integrity
Hashes and signatures prove the bytes serving loads are the bytes that were evaluated. Deserialisation formats that execute code on load, and a serving process that loads the wrong file, are the two ways that proof gets skipped.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Between the registry and the serving process the artifact is copied, cached and loaded. How do you know the file serving opened is the file that was promoted — and that opening it is safe?
A recommendations team investigating an odd week of results found that one of six serving replicas had been running a different model for eleven days. A node had been rebuilt, the init container pulled from a stale cache, and the file it got was an older version with the same name. Separately, the security review asked why the serving process loaded a pickle from a bucket that four teams could write to.
The registry says which path is production; the init container downloads that path; the serving process loads it. Object storage is durable and the path is unique, so the file serving gets is the file the registry means.
A path is not an identity. The same path held three different files over its life, the node cache kept the first, and a replica served a model the registry had archived — indistinguishable from the outside because the version string inside the file was also stale.
- A path is not an identity. The same path held three different files over its life, the node cache kept the first, and a replica served a model the registry had archived — indistinguishable from the outside because the version string inside the file was also stale.
- The prediction log records a model *name* and *version string*, both read from the loaded file, so the log confirmed the wrong replica was running the right version.
- A pickle-based artifact runs code on load. Anyone able to write to the bucket can replace the file with one that behaves identically and also does something else at load time, and the serving process cannot tell (The Model Supply Chain).
- A dependency of the preprocessing pipeline was pulled from a public package index at image build time without a pinned hash; the artifact was intact and the code that deserialised it was not.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Rank items for a home feed; the target of the surrounding system is engagement, and the artifact under discussion is the ranking model's bundle.
- The property being protected is not the model's quality but its identity: that what serves is what was promoted, and that loading it cannot run anything but the model.
- Artifacts are written to object storage by the training pipeline, referenced by the registry, pulled by an init container into a node-local cache, and loaded by the serving process at startup.
- The serialisation is a framework-native pickle-based format that reconstructs arbitrary Python objects on load, including the preprocessing pipeline.
- The bucket is writable by the training pipeline's role and, historically, by anyone who has ever needed to "fix a file quickly".
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A content hash is a fixed-length digest of the bytes. Any change to the bytes changes the digest; comparing the digest of what was loaded to the digest the registry recorded at promotion proves they are the same bytes. A path, a version string or a filename proves nothing, because all three can be attached to different bytes.
- A signature adds authorship: the promoting pipeline signs the digest with a key only it holds, and serving verifies the signature before loading. A hash catches accidental substitution; a signature also catches deliberate substitution by anyone who cannot sign.
- Deserialisation formats differ in what loading means. A pickle or a framework format built on it reconstructs objects by executing stored instructions, which can be arbitrary code. A tensor-only or interchange format carries arrays and a graph description and loads by parsing, not executing. The second cannot carry an arbitrary preprocessing object; that limitation is the safety property (Preprocessing Lives in the Artifact).
- Provenance is the chain: training run → registered digest → signed promotion → serving verification → digest logged per prediction. Every link that is skipped is a place where the file can be swapped without anyone knowing.
A path is not an identity
The registry said production was s3://models/feed-ranker/v41/model.pkl. That was true in the sense that the path was the one promoted. It was not true in the sense that mattered: the bytes at that path had changed twice, a node cache held the first version, and the replica that rebuilt from the cache served it for eleven days. Nothing checked the bytes because everything trusted the name.
The chain below is the provenance a serving system needs, and each arrow is a place where the file can be swapped. The digest is what makes the arrows checkable: it is recorded once at registration and compared at every subsequent hop.
v41 beat v38 on the held-out engagement metric at promotion, on the same slice and threshold policy; the registry recorded the gate as passed.
Fleet-level engagement moved by a fraction of the expected gain, and one region's metrics were flat; nothing in the model dashboards was out of range for eleven days.
- 1One of six replicas loaded v38 from a stale node cache after a rebuild; its predictions were plausible and its version string, read from inside the file, said what the file said.
- 2Aggregated metrics diluted a one-in-six regression to a small shortfall that looked like a noisy week, and the prediction log recorded a name, not a digest, so no reconciliation could flag the replica.
- 3The gate was correct about v41. The gap was not model quality but identity — the evaluated bytes were not what one replica served.
Loading by parsing versus loading by executing
A general object serialiser stores instructions for rebuilding objects, and the loader follows them. That is what makes it able to carry a fitted preprocessing pipeline with arbitrary custom classes, and what makes loading it equivalent to running whatever the writer chose. A tensor-only format stores named arrays and metadata; the loader parses and allocates. It cannot carry a custom class, which is why teams reach for the general format, and it cannot run code, which is why they should not.
The comparison is not "secure versus insecure" but "what can this format carry, and what does loading it cost you". The right design puts the weights in a parse-only format and the preprocessing state in explicit data — means, vocabularies, fill values — that a small, reviewed piece of code reconstructs.
Preprocessing object and estimator serialised together with a general object serialiser; loaded from a bucket several teams can write to; identified by path.
Weights in a tensor-only or interchange format; normaliser statistics, vocabularies and fill values as JSON; a manifest listing every member with its digest, signed at promotion; loader verifies before it parses.
Loading the second cannot execute anything but the reviewed loader, substitution of any member is detected before parsing, and the preprocessing state is readable and diffable — which also makes train/serve skew inspectable rather than hidden inside an opaque object.
The check that makes the log trustworthy
The loader computes the digest over the bytes it is about to parse, compares it to the digest the registry holds for the Production entry, verifies the signature over that digest, and only then parses. The digest it verified is what it logs on every prediction. The reconciliation job compares logged digests against the registry's stage history and alerts on any row that does not match.
Everything in this chain is a few dozen lines. The reason it is often missing is not difficulty but that the failure it prevents looks, from the outside, like a slightly worse model.
On every replica, after every restart, the bytes the loader parsed have the digest the registry recorded at promotion and a signature from the promotion key — and loading them executed nothing but the reviewed loader.
holds when Verification happens in the serving process over the fetched bytes, after any cache; the signing key is held only by the promotion pipeline; the weights are in a parse-only format and the serving image is built from a hash-locked manifest.
breaks when Verification is done in an init container and the cache is read afterwards; the key is stored beside the artifacts; a code-bearing format is reintroduced "for one preprocessing object"; a dependency is pulled unpinned at build time.
respond Take the replica out, identify what it actually loaded, and close the path that let unverified bytes reach the parser before restoring capacity.
1import { createHash } from 'node:crypto'2 3async function loadProduction(model: string) {4 const entry = await registry.production(model) // { digest, signature, location, manifest }5 const bytes = await fetchBytes(entry.location) // from cache or storage — untrusted either way6 const digest = createHash('sha256').update(bytes).digest('hex')7 if (digest !== entry.digest) {8 throw new Error(`refusing ${model}: digest ${digest} != registry ${entry.digest}`)9 }10 if (!verifySignature(entry.digest, entry.signature, PROMOTION_PUBLIC_KEY)) {11 throw new Error(`refusing ${model}: signature does not verify against the promotion key`)12 }13 const bundle = parseBundle(bytes, entry.manifest) // parse-only: tensors + JSON preprocessing state14 return { bundle, digest } // the digest travels into every prediction log row15}The digest is computed over the bytes actually fetched, after any cache, and the thrown error takes the replica out rather than letting it serve. A health check has to surface that refusal, or the fleet shrinks silently.
How to build it
Most important first.
- Identify artifacts by digest everywhere: the registry records it at registration, the serving loader computes it after download and refuses on mismatch, and the prediction log carries the digest rather than a version string read from inside the file (Prediction Logging).
- Sign at promotion with a key held by the promotion pipeline only, verify at load, and treat an unsigned or wrongly-signed bundle as unloadable — not as a warning.
- Prefer a serialisation that loads by parsing over one that loads by executing; where the preprocessing state genuinely needs a code-bearing format, keep it in a separately signed, minimal object and never load either from a location writable by humans.
- Pin every dependency of the serving image by hash and build from a locked manifest; the artifact is only as trustworthy as the code that deserialises it.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The count of predictions whose logged digest does not match the registry's Production digest at that time. Zero is the only acceptable value; the recommendations incident was eleven days of a non-zero count nobody computed.
- The count of load attempts refused for digest or signature mismatch. Non-zero is a healthy signal that the check exists and something upstream is wrong; it is not a reason to relax the check.
- Not "the model version in the health endpoint". It is read from the loaded file, which is the thing in question.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The digest serving computes over the bytes it actually loads matches the digest the registry recorded at promotion, on every replica, after every restart and cache rebuild.
- The key that signed the promoted artifact is held only by the promotion pipeline, and no writable path exists from a human or a training job to the location serving loads from.
- Loading the artifact executes nothing beyond the deserialiser and the model runtime, and the deserialiser's own dependencies are pinned by hash.
- Offline: a test that alters one byte of a promoted bundle and asserts the loader refuses; a test that presents an unsigned bundle and asserts refusal; an audit that lists every principal with write access to the artifact location.
- Online: the digest-versus-registry reconciliation over the prediction log, per replica, alerting within minutes of a mismatch; a periodic re-verification of the local cache on long-running replicas.
- Over time: rotate the signing key and confirm serving rejects bundles signed with the retired key; rebuild the serving image from the lock file and confirm the digest of the image is reproducible.
What can go wrong
- The digest check is implemented in the init container and the serving process loads from the local cache afterwards; a cache entry that predates the check is never verified.
- The signing key is in the same bucket as the artifacts "for convenience", so anyone who can replace the file can re-sign it.
- The loader verifies the digest of the archive and then extracts a member with a path that escapes the target directory, or a member the manifest did not list; verify the manifest's listed members, not the container.
- A migration to a parse-only format drops the preprocessing object because the format cannot carry it, and serving recomputes the normaliser from live data — an integrity fix that introduced skew (Train / Serve Skew).
- Digest verification adds a full read of the bundle at startup, which for a large model is seconds to minutes and lengthens every scale-out (Model Serving Architecture).
- Moving away from code-bearing serialisation forces the preprocessing state into an explicit data format, which is real engineering work and is the point.
- A hard refusal on mismatch means a corrupted cache takes a replica out rather than serving stale predictions; that is the correct failure, and it needs a health check that reports it (Serving Fallbacks).
- "The bucket is private, so the artifact is safe." Private to whom? Four teams and every past hotfix have write access. Integrity is about who can change the bytes between promotion and load, not about who can read them.
- "We log the model version, so we would have seen it." A version string inside the file describes the file it is inside. Only a digest computed over the loaded bytes and compared to an external record can detect substitution.
- "Signing is for supply-chain attackers; our threat is bugs." The recommendations incident was a bug. The same digest check that catches an attacker catches a stale cache, and it is the cheaper of the two to explain.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat bytes are identified by digest and not by path holds for every artifact in every domain; the ML-specific part is that the artifact includes fitted state whose serialisation format decides whether loading is parsing or executing.
- FRAMEWORK-SPECIFICWhich formats execute code on load and which only parse arrays varies by framework and version; the safe assumption is that any format built on a general object serialiser executes, and any tensor-only or graph-interchange format does not — check the specific loader rather than the file extension.
- SCALE-SPECIFICWith one replica loading from one path the wrong-file failure is rare and obvious; with dozens of replicas, node-local caches and rolling rebuilds it is common and invisible, because the fleet looks healthy in aggregate while one replica serves an archived model.
Where the depth lives
This domain teaches the model and hands the rest off by name.