ArtifactsGENERALFRAMEWORK-SPECIFICMODEL-SPECIFIC

What a Model Artifact Contains

A weights file alone is not a model. The artifact is parameters, architecture, fitted preprocessing, feature order, version and training metadata — and serving needs all of it.

Target & dataWhat to measureWhat must stay true▶ Follow One Prediction

The problem, the obvious approach, and why it breaks

Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.

The question

The training run finished and produced a file. What has to travel with that file for a serving process to reproduce the predictions the evaluation measured?

The problem

A pricing team hands the platform group "the model" — a single .pt file on a shared drive — and asks for it to be served by Monday. Two weeks later the served prices are systematically lower than the ones the notebook produced for the same listings, and nobody can say which of four candidate files in the drive is the one that was evaluated.

The obvious approach

A model is its learned parameters. Save the weights, load the weights, call predict. The framework serialises the tensor state and that is what the training run produced, so that is the model.

Why it breaks

The serving process loads the weights into an architecture rebuilt from the current code, which has one more hidden layer than the code that trained the file; the state dictionary loads with a warning, and half the weights land in the wrong layer.

How it breaks — usually after the offline metric looked fine
  • The serving process loads the weights into an architecture rebuilt from the current code, which has one more hidden layer than the code that trained the file; the state dictionary loads with a warning, and half the weights land in the wrong layer.
  • Serving standardises mileage with the mean of the last week's listings instead of the training mean, so every input is shifted and the prediction is shifted with it — a bias with no error message.
  • A region added after training is unknown to the encoder; serving maps it to the all-zeros vector, which the model learned to read as the most common region during training.
  • Four files in the drive share a name pattern; the one that ships is the last one written, which was a hyperparameter experiment that was never evaluated.
Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

What is being predicted, and from what data

This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.

Target
  • Predict the sale price of a used-vehicle listing from its attributes; the label is the realised sale price, known once the listing closes.
  • The prediction feeds a suggested-price widget, so a systematic bias in one direction costs sellers money on every listing and is not obviously wrong on any single one.
Data
  • One example is one closed listing: make, model, year, mileage, region, a free-text condition field, and the closing price.
  • The notebook standardised mileage and year using the training-set mean and standard deviation, one-hot encoded region with the training vocabulary, and filled missing mileage with the training median. None of that is in the .pt file.
  • The columns were fed to the network in the order the DataFrame happened to have them after a merge; the order was never written down.

How it actually works

Precisely enough to predict its behaviour — not a framework API.

  • A trained model is a function f(x; θ) where θ are the weights and x is the tensor built from a raw record by a fixed sequence of transformations. The evaluation measured the composition of both halves. Shipping θ alone ships half of the function.
  • Every fitted transformation carries state learned from the training fold: a normaliser's means and standard deviations, an encoder's vocabulary and unknown-token policy, an imputer's fill values, a tokenizer's merges. Those numbers are as much "learned parameters" as the weights are, and they must be the training-fold values, not recomputed at serving time (Preprocessing Lives in the Artifact, Train / Serve Skew).
  • The architecture is the shape the weights are poured into. A state dictionary is a map from layer names to tensors; if the receiving architecture does not match exactly, the load either fails loudly or, worse, succeeds partially.
  • Metadata — feature order, framework and library versions, the dataset and code versions, the metrics and the threshold — is what makes the artifact identifiable and checkable rather than a blob with a filename (Model Lineage).

The file is half the function

The evaluation ran predict(transform(raw)) and reported a number about that composition. The .pt file holds the parameters of predict. Everything transform learned — the mean of mileage in the training fold, the vocabulary of regions, the median used to fill a missing odometer reading, the order the columns arrived in — lived in Python objects that died with the kernel.

So the artifact has to be defined as the whole function. The diagram is the minimum: not a file but a bundle, with a manifest that names every part and a hash that identifies the bundle as a whole. Serving loads the bundle or refuses; it does not reconstruct the missing half from whatever data is nearby.

load or refuseweights θarchitecturefitted preprocessingfeature order + typesthresholdmanifest: version, hash, lineage, metricsserving loader (strict)
UserLLMAgentToolDataDecisionHumanGuardrail

What a strict loader checks

A loader that accepts whatever it is given turns every packaging mistake into a quiet bias. A loader that checks the manifest turns the same mistake into a failed deploy, which is the cheap place to find it. The checks are mechanical: keys in the state dictionary match the architecture exactly, the preprocessing state has the fields the manifest says, the feature list in the request matches the manifest in name and order.

The replay sample is the check that catches everything the others miss. The training run stores a few hundred validation rows and the predictions it made for them; the serving image reproduces them or it does not ship.

A manifest, and the load that refuses to guess
1manifest = {
2    "version": "price-v14",
3    "sha256": "…",
4    "framework": {"name": "torch", "version": "2.3.1"},
5    "features": [                      # order is part of the contract
6        {"name": "year", "type": "float", "transform": "standardize"},
7        {"name": "mileage", "type": "float", "transform": "standardize", "fill": "median"},
8        {"name": "region", "type": "category", "transform": "onehot", "unknown": "reject"},
9    ],
10    "preprocessing": "preprocessing.json",   # training-fold means/stds/vocab/medians
11    "threshold": None,                        # regression: no operating point
12    "lineage": {"dataset": "listings@2026-07-31", "features": "fdef-9", "commit": "a1b2c3d"},
13    "eval": {"split": "val-2026-08", "replay": "replay_sample.parquet"},
14}
15 
16def load(bundle):
17    missing, unexpected = model.load_state_dict(bundle.weights, strict=True)  # raises on mismatch
18    prep = Preprocessing.from_json(bundle.read(manifest["preprocessing"]))
19    for spec in manifest["features"]:
20        assert spec["name"] in prep.state, f"no fitted state for {spec['name']}"
21    sample = bundle.read_parquet(manifest["eval"]["replay"])
22    preds = model(prep.transform(sample.drop(columns=["expected"])))
23    assert (preds - sample["expected"]).abs().max() < 1e-5, "serving path does not reproduce training outputs"
24    return model, prep

The replay assertion is the one that matters. Key matching catches an architecture mismatch; only reproducing stored predictions catches a normaliser refitted on the wrong data or a column silently reordered.

What must stay true once the bundle is deployed

The artifact is a promise that serving computes the same function the evaluation measured. The promise is kept by three things staying true: the bundle serving loads is the evaluated bundle, the transformations use the stored training-fold state, and the input schema is the manifest's schema. Each of those can be checked by a machine, and each breaks for a mundane reason — a redeploy, a dependency bump, a new upstream column.

None of them is checked by the validation metric, which was computed before the bundle existed. That is the general pattern of this domain: the number is true about a system that is not the one that shipped.

must stay trueThe artifact is the evaluated function

For any raw record, the serving process produces the prediction the training process would have produced for the same record, because it runs the same weights, the same fitted preprocessing and the same feature order.

holds when The bundle is complete and content-hashed, the loader is strict, and the replay sample reproduces in the serving image before every rollout.

breaks when A serving dependency is upgraded and deserialises preprocessing state differently; the feature service adds or reorders a column; the loader falls back to defaults for a missing field; someone deploys the wrong file from a directory of similar names.

how you would know The artifact hash logged with every prediction and compared to the registry entry; a schema check on every request; the replay diff run in CI against the serving image, not the training container.

respond Roll back to the last bundle whose replay passed, then fix the packaging or the loader. Do not retrain — the weights were never the problem.

How to build it

Most important first.

  • Define the artifact as a directory or archive with a manifest, not a file: weights, an architecture description or a self-describing serialised graph, the fitted preprocessing state, the ordered feature list with types, the operating threshold, and a metadata document.
  • Give the artifact a content hash and a version, and make serving log both on every prediction so a prediction can always be traced to the exact bundle that produced it (Artifact Integrity, Prediction Logging).
  • Record the training provenance in the manifest: dataset version, feature-definition version, code commit, hyperparameters, random seed, and the evaluation metrics on the named validation set (Reproducibility, Feature and Model Versioning).
  • Make loading strict: refuse a state dictionary with missing or unexpected keys, refuse a request whose feature names or order differ from the manifest, and refuse a category the encoder has no policy for.

What to measure

Which number actually maps to the decision — and which numbers look relevant and are not.

  • A replay check: run a fixed sample of validation rows through the serving path and compare predictions to those the training run stored in the manifest. Bitwise or within a stated tolerance — this is the number that says the artifact is complete.
  • The share of serving requests whose feature vector fails the manifest's schema — wrong order, unknown category, out-of-range value. Anything above zero after rollout is an incomplete artifact.
  • Not the validation metric. It was computed inside the training process where every transformation was in memory, so it cannot detect anything missing from the exported bundle.

What must stay true after deployment

The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.

Assumptions
  • The serving process reconstructs exactly the architecture, preprocessing state and feature order recorded in the manifest, and rejects a bundle it cannot reconstruct exactly.
  • Every fitted transformation applied at serving time uses the training-fold state stored in the artifact, never a value recomputed from serving traffic.
  • The bundle that serving loads is the bundle that was evaluated, and a hash on both sides can prove it.
How to verify — offline, online, and over time
  • Offline: build the serving image, load the artifact into it, replay the manifest's stored validation sample and diff predictions against the stored outputs. Run this in CI for every candidate (Serving Contract Tests).
  • Online: on rollout, log the artifact hash on every prediction and assert it matches the registry's production entry; alert on any request rejected by the schema check.
  • Over time: whenever a serving dependency is upgraded, rerun the replay check before the upgrade reaches production; a library bump is a change to the model.

What can go wrong

Failure modes in production
  • The manifest is written by hand and drifts from the code: the feature list is updated, the manifest is not, and the strict loader now rejects every request until someone edits the manifest to match — without checking which one is right.
  • The bundle is complete but the library that deserialises the preprocessing state is a different major version in serving, and the pickled normaliser silently loads with default attributes.
  • The replay check passes because it was run against the same bundle in the same container the training used, not against the serving image.
What the recommended approach costs
  • A complete artifact is larger and slower to build, and the manifest is one more thing to keep in sync with the code that produces it.
  • Strict loading turns a silent bias into a hard outage; that is the right trade almost always, but it means a schema change anywhere upstream is now a deploy blocker.
  • Framework-native serialisation carries the most state with the least effort and ties the artifact to one library version; interchange formats are portable and carry less (Preprocessing Lives in the Artifact).
Misreads
  • "The weights loaded without error, so the model is the same." A partial load with a warning, or a full load into a re-declared architecture with different defaults, produces different predictions without an error.
  • "We can recompute the normalisation from the data at serving time." That is a different model. The weights were fitted against the training-fold statistics, and any other statistics shift every input.
  • "The notebook is the source of truth." The notebook has state the file does not — in-memory encoders, a column order, an implicit threshold. The artifact must carry what the notebook remembers.

Where this applies

ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.

  • GENERALThat the evaluated function is weights composed with fitted transformations holds for every model family; a gradient-boosted tree carries a categorical vocabulary and a feature order as surely as a network carries a normaliser.
  • FRAMEWORK-SPECIFICWhat "the architecture" means differs by framework: a state dictionary needs the class definition re-declared exactly, while a self-describing graph format serialises the structure and needs only a compatible runtime. The manifest has to say which.
  • MODEL-SPECIFICPipeline objects in tabular libraries can bundle preprocessing and estimator in one serialised object, which hides the problem rather than solving it; deep-learning training code usually keeps preprocessing outside the model object, so it is forgotten more often.

Where the depth lives

This domain teaches the model and hands the rest off by name.

Data Engineeringfeature-pipelines
DevOpsartifactsartifact-digestsbuild-provenance