ML Security
Six ways an ML system is exposed that a service is not: poisoned training data, tampered artifacts, memorised sensitive data, adversarial inputs, an untrusted supply chain and abusable inference. Defensively, at the level of what to trust.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A standard security review of the model service found nothing. What does an ML system expose that the review was not looking for?
Our security team reviewed the model serving service — authentication, TLS, input schema, dependency scanning — and signed off. A week later a researcher showed us that our support-ticket classifier would reproduce a customer's address if prompted with the start of a specific ticket. The review had no category for that. I need the list of what is different about an ML system so the next review has one.
A model service is a service. Authenticate it, validate the request schema, scan the dependencies, encrypt in transit. That is what the security checklist covers, and it passed.
The checklist treats the model as code. The model is also a compressed copy of its training data, and a text model fine-tuned on tickets memorises rare strings — an address that appears in one ticket is a sequence the model can reproduce (ML Privacy).
- The checklist treats the model as code. The model is also a compressed copy of its training data, and a text model fine-tuned on tickets memorises rare strings — an address that appears in one ticket is a sequence the model can reproduce (ML Privacy).
- The dependency scan covered packages. It did not cover the pretrained weights, which were a file downloaded from a hub and loaded with a deserialiser that executes code on load. That file was a dependency with no hash, no signature and no scan (The Model Supply Chain).
- The input schema was validated. The input *content* was not: a ticket crafted to flip the classifier's routing is a valid JSON string (Adversarial Inputs). And the endpoint's probabilities let anyone measure how the model responds to changes, which is the raw material for extraction (Inference Abuse).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The classifier routes support tickets to teams; the label is the team that resolved the ticket. This lesson's target is a threat inventory for the system around it: the six exposures that are specific to a model, and the boundary each one needs.
- Training data: two years of support tickets including free text, which contains whatever customers typed — addresses, order numbers, occasionally card fragments. Labels from the ticketing system. A pretrained text encoder downloaded from a public hub and fine-tuned. An endpoint that returns the top three teams with probabilities.
- The review covered the endpoint as a service. It did not cover the training data as an input, the pretrained weights as a dependency, or the predictions as an output that leaks.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A service has inputs, code and outputs. An ML system has inputs that were partly authored by the public (training data), code that was partly produced by a process nobody reviewed line by line (the weights), and outputs that reveal the training data and the decision function. Each of those is a trust boundary a service review does not draw.
- The six exposures follow from that. Data poisoning: training data from a source an adversary can influence teaches the model wrong behaviour (Data Poisoning). Artifact integrity: the weights file is a dependency that must be hashed, signed and loaded safely (Artifact Integrity). Sensitive-data leakage: a model memorises rare training examples, and membership can be inferred from confidence — at the concept level, the model is a lossy copy of its data. Adversarial inputs: small changes to a valid input change the prediction. Supply chain: pretrained weights, datasets and libraries from third parties. Inference abuse: an endpoint that answers freely can be extracted, probed or run up a bill.
- Security Engineering owns the attacker model, threat modelling, input validation, rate limiting and secrets — this module does not re-teach them. It says where the ML-specific boundaries are and what each one needs validated, signed, minimised or limited.
Three boundaries a service review does not draw
The service review draws the boundaries a service has: the network edge, the request, the dependencies, the secrets. A model system has three more. Its training data crosses a boundary from sources it does not control. Its weights cross a boundary from a training process — or a third party — into a loader that trusts the file. Its outputs cross a boundary to callers who can learn from them.
The diagram places the six exposures on those boundaries. Every one of them is a place to validate, sign, minimise or limit — the same verbs as the rest of security engineering, applied to parts of the system a service does not have.
The model is a lossy copy of its data
The address came back because a model with enough capacity fits rare training examples closely, and a text model that predicts the next token has, for a rare sequence, only one example to fit. The reproduction is not a bug in the serving code; it is the model doing what training asked of it. The same mechanism lets an outside party estimate whether a given record was in the training set by how confidently the model treats it — membership inference, at the level of concept.
Offline evaluation cannot see this. A memorising model generalises fine on the validation split, because memorising rare strings and learning the routing task are not in conflict. The offline/online gap here is between the metric and a property the metric was never about.
Held-out routing quality comparable to the previous model; no evaluation of what the model would reproduce, because no test asked.
A researcher prompts the endpoint with the opening of a real ticket and receives the customer's street address in the model's continuation.
- 1Fine-tuning a text model on raw tickets fits rare sequences — a specific address — closely enough to reproduce them.
- 2The training text was not scrubbed, on the argument that addresses sometimes carry routing signal.
- 3The endpoint exposed a generative continuation path that the routing consumer never needed.
The threat inventory, with a control per row
The table is the deliverable for the next review: each ML-specific exposure, the boundary it lives on, the defensive control, and the lesson that goes deeper. The attacker's side is deliberately absent; Threat Modeling: The Process and Attack Surface in Security Engineering cover how to think about an adversary, and nothing here requires one to justify the controls.
The assumption device closes with the one that is easiest to forget: the boundaries move. A new data source, a new pretrained model or a new output field each adds a row, and a threat model that was complete last quarter is not complete now.
Each training source, each third-party artifact, the serving loader and each output field is listed in the threat model with a control and an owner.
holds when The threat model is revisited when a data source, a pretrained model or an output field is added, and the review checklist includes the six rows above.
breaks when A team adds a public dataset "to improve coverage"; a new endpoint returns embeddings for a search feature; a fine-tuning run switches to a hub model nobody pinned.
respond Add the row before the release, not after; if a boundary shipped uncovered, treat it as a finding with the same severity as an unauthenticated endpoint.
| Exposure | Boundary | Defensive control | Lesson |
|---|---|---|---|
| Training data poisoning | Sources → training set | Provenance per source; validation; slice evaluation; caution with user-influenced labels | Data Poisoning |
| Artifact integrity | Training / registry → serving | Hash on write, verify on load; signed artifacts; a loader that cannot execute code | Artifact Integrity |
| Sensitive-data leakage | Training set → outputs | Scrub or exclude sensitive text; canary tests for memorisation; minimal outputs; privacy techniques where warranted | ML Privacy |
| Adversarial inputs | Callers → predictions | Input validation on content; robustness tests; ensembles; confidence monitoring | Adversarial Inputs |
| Supply chain | Third parties → training | Pinned, hashed, signed weights and datasets; safe formats; provenance in the registry | The Model Supply Chain |
| Inference abuse | Callers → endpoint | Authentication; rate limits; return decisions not probabilities; cost limits on GPU endpoints | Inference Abuse |
How to build it
Most important first.
- Add the ML boundaries to the threat model explicitly: training data sources, the artifact and its loading path, the prediction output, and the third-party weights and datasets. A review that draws those boundaries has a category for the address that came back.
- Treat training data as untrusted input: validate it, record provenance, and evaluate on slices so that a poisoned source shows up as a slice anomaly (Data & Feature Tests, Evaluation Slices).
- Treat the artifact as a signed dependency: hash on write, verify on load, use a format that cannot execute code, record provenance in the registry (The Model Registry).
- Treat the output as sensitive: return the decision the consumer needs rather than the full probability vector, rate-limit and authenticate the endpoint, and scrub or exclude sensitive strings from training text before the model can memorise them.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Coverage of the ML boundaries in the threat model: is each of the six listed with an owner and a control. This is a checklist, and its absence is the finding.
- For leakage specifically: the rate at which a canary string inserted into training data can be reproduced from the endpoint. If it can, the model memorises, and the scrubbing is not sufficient.
- The service review's pass rate is the number that looks relevant and is not: it measures the boundaries a service has, which are necessary and not sufficient.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every training data source is listed with its provenance and trust level, and a source that users can influence is validated and slice-evaluated separately.
- The serving path loads only artifacts whose hash matches the registry record, using a loader that cannot execute code from the file.
- The endpoint returns the minimum the consumer needs, under authentication and rate limits, and the returned confidence is not a raw probability unless a consumer has a stated need for it.
- Offline: insert a synthetic canary string into a small fraction of training text, train, and test whether the endpoint reproduces it. Repeat after scrubbing. This is a memorisation test, and it is the one the review lacked.
- Online: attempt to load an artifact whose hash does not match the registry in staging; the serving path must refuse. Attempt a burst past the rate limit with a valid key; it must be limited.
- Over time: rerun the threat model when a new data source, a new pretrained model or a new output field is added — each is a new boundary.
What can go wrong
- The training text is scrubbed for known patterns — card numbers, emails — and memorises a customer's unusual name and street combination, which no pattern matches.
- The artifact is signed, and the loading code verifies the signature and then deserialises with a format that executes code on load, so a compromised registry write still runs code.
- The endpoint is rate-limited per key, and a single legitimate customer with a high quota is the one whose key leaks.
- Scrubbing training text loses signal; an address in a ticket is sometimes exactly what routes it to the right team.
- Returning a decision instead of probabilities removes information a legitimate downstream consumer may have relied on, and they will ask for it back.
- Signing and verifying every artifact adds a step to the promotion path and a key to manage (Secrets Management is where that lives).
- "The service passed the security review." The review covered the boundaries a service has. A model has three more — its data, its weights and its outputs — and none was in scope.
- "We validate all inputs." The schema was validated. A poisoned training row and an adversarial ticket are both schema-valid; the validation that matters is of content and provenance.
- "ML security is about attacks, and we are not a target." Every endpoint on the internet is probed; every public data source is influenceable. The defensive controls are cheap relative to the exposure and do not require assuming an attacker.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a model is trained on partly untrusted data, loaded from a file that is a dependency, and exposes its decision function through its outputs holds for every model family; which exposure dominates depends on the data and the deployment.
- DOMAIN-SPECIFICA fraud or spam model faces adversarial inputs by nature and poisoning through user feedback; a medical or support model faces memorisation of sensitive text; an internal forecasting model on trusted warehouse data faces mostly the supply-chain and artifact exposures. The threat inventory is the same; the priorities are not.
- MODEL-SPECIFICLarge text models memorise rare sequences far more readily than a gradient-boosted classifier on tabular features, so the sensitive-data leakage exposure is severe for generative and fine-tuned text models and mild for most tabular models.
Where the depth lives
This domain teaches the model and hands the rest off by name.