Inference Abuse
An endpoint that answers anyone reveals its decision function, its training data and its cost structure. Authenticate, rate-limit, return decisions rather than probabilities, and bound spend on GPU endpoints.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A public prediction endpoint was built to be helpful: fast, unauthenticated, returning full probabilities. What can a caller learn or cost you by calling it a lot, and what do you withhold without breaking the product?
Our pricing-recommendation endpoint returns a full probability distribution over price bands for any listing. A competitor appears to have rebuilt our model: their recommendations track ours within a day of each change. Separately, our image-classification endpoint on GPU had a weekend bill ten times normal from a single API key sending garbage. Both endpoints passed the security review, because both are "just APIs".
A prediction endpoint is a read-only API. It cannot change anything, so it needs less protection than a write endpoint. Return everything the model produces; consumers can ignore what they do not need.
Read-only is not harmless. Each response is a sample of the decision function, and enough samples over a chosen grid of inputs let a caller fit a model that approximates yours. The full probability vector gives far more information per call than a decision would — the competitor needed fewer calls than anyone estimated.
- Read-only is not harmless. Each response is a sample of the decision function, and enough samples over a chosen grid of inputs let a caller fit a model that approximates yours. The full probability vector gives far more information per call than a decision would — the competitor needed fewer calls than anyone estimated.
- The image endpoint's cost was bounded by autoscaling, which is to say unbounded: a caller sending garbage at volume was met with more GPUs. Autoscaling protected availability and converted the abuse into a bill (Inference Cost).
- Confidence on specific inputs also reveals training membership at the concept level — a model tends to be more confident on examples it trained on — and a public endpoint with raw probabilities exposes that to anyone who asks (ML Privacy).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The pricing model recommends a price band; the image model classifies listing photos. The defensive target is an endpoint that gives each legitimate consumer what their decision needs and no more, under authentication and limits that bound what any single caller can learn or spend.
- The pricing endpoint: no authentication beyond an API key handed out freely, no rate limit, full probability vector in the response, no logging of per-key volume. The image endpoint: authenticated, GPU-backed, autoscaled, no per-key spend cap, no input validation beyond "is an image".
- Prediction logs exist for the pricing model but are not joined to caller identity, so a caller's query pattern — a grid over listing features — was never visible as a pattern.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An inference endpoint is an oracle for the model's function. Every call returns a point on that function; the richer the output — probabilities, embeddings, explanations — the more of the function each call reveals. Model extraction is the systematic use of that oracle; membership inference is its use to learn about the training set; cost abuse is its use to make the provider pay for compute. None requires a vulnerability; all use the endpoint as designed.
- The defences are the ordinary ones for any API, plus output minimisation, which is specific to models. Authentication ties calls to a principal. Rate limiting bounds the calls per principal per window, which bounds extraction and cost (Rate Limiting as a Security Control). Output minimisation returns the decision the consumer needs — a band, a class, a yes — rather than the probabilities, embeddings or explanations that reveal the function. Spend caps bound the compute a principal can trigger on GPU endpoints, independent of autoscaling.
- Generative endpoints add a class of abuse — prompt-style manipulation of the model's instructions — that belongs to Agentic Engineering; see Prompt Injection and Input and Output Guardrails there rather than here. The endpoint controls in this lesson apply to generative endpoints too and are the floor under those.
What a caller learns per call
The output field is the exposure. A class label reveals which side of the boundary an input is on. A probability reveals how far. An embedding reveals the representation. An explanation reveals which features moved it. Each step up gives the caller more of the function per call, and the number of calls extraction needs falls accordingly.
The matrix pairs each output with what it reveals and with who legitimately needs it. The last column is the design question: for each consumer, what is the least output that supports their decision?
| Output returned | What each call reveals | Who legitimately needs it | Default |
|---|---|---|---|
| Decision only (band, class, yes/no) | Which side of the boundary; the least per call | Most product consumers | Public endpoint |
| Calibrated probability | Distance from the boundary; membership hints on trained examples | Consumers who set their own thresholds; risk systems | Authenticated, rate-limited, logged path |
| Full distribution over classes | The shape of the function around the input | Rarely anyone outside the model team | Internal only |
| Embeddings | The learned representation; enables near-complete reuse of the model | A search or retrieval system you operate | Never public; service-to-service authentication |
| Feature attributions | Which inputs move the decision and by how much | Reviewers, regulators, a debugging tool | Internal, per request, audited (Explainability) |
The weekend the autoscaler paid for
The image endpoint did everything a well-run service does: it stayed up, it stayed fast, it scaled. The exposure was not availability, and the controls that protect availability made the cost exposure worse. A GPU endpoint is a way to convert requests into spend, and an autoscaler with no per-principal cap is a promise to convert as many as arrive.
The offline/online device is the capacity plan against the bill. The plan was right about capacity; it never modelled a single principal as a load source.
Capacity plan from expected legitimate traffic: autoscaling keeps latency within budget at the projected peak, with headroom.
One principal sends garbage images at volume for two days; latency stays within budget; the bill is a multiple of a normal month.
- 1No per-principal rate limit or spend cap, so one key could trigger unbounded compute.
- 2No input validation before the GPU; garbage images cost the same inference as real ones.
- 3Monitoring watched availability and latency, both of which autoscaling kept perfect, and nobody watched spend per principal.
The public endpoint, minimised
The serving code below is the shape of a public endpoint after the controls: authenticated, limited per principal, validated before the model, returning a decision and logging the call with its principal. The probability path exists and is elsewhere. The point is not any single check; it is that the response carries what the consumer acts on and nothing else.
The assumption device closes with what must stay true: that the rich output stays on the controlled path, and that the limits stay tied to legitimate usage rather than drifting up to capacity.
Each public response carries the consumer's decision and nothing richer, and every richer output is behind a separate authenticated, rate-limited, logged path with a named consumer.
holds when The public handler returns the decision only; the internal path requires a distinct credential; limits are derived from legitimate usage and reviewed; per-principal volume and input patterns are monitored.
breaks when A product team adds confidence to the public response "for the UI"; the internal path accepts the public key; the rate limit is raised to capacity after a complaint; key issuance becomes self-service without verification.
respond Remove the field or move the consumer to the controlled path; rotate the shared key; and if a sweep is found, treat the model as partially extracted — the response is a business decision, not a retrain.
1export async function recommendBand(req: Request, deps: Deps): Promise<Response> {2 const principal = await deps.auth.principal(req) // no principal, no prediction3 if (!principal) return json({ error: 'unauthenticated' }, 401)4 5 if (!(await deps.limits.allow(principal, 'pricing', { perMinute: 60 })))6 return json({ error: 'rate_limited' }, 429) // bounds extraction and cost7 8 const input = parseListing(await req.json()) // schema and range validation9 if (!input.ok) return json({ error: 'invalid_input' }, 400)10 11 const probs = await deps.model.predict(input.value) // full distribution stays here12 const band = argmax(probs) // the decision the consumer acts on13 14 deps.log.prediction({ principal, modelVersion: deps.model.version, input: input.value, band })15 // no probabilities, no embeddings, no attributions in the public response16 return json({ band, modelVersion: deps.model.version })17}The line that does the security work is the return statement. Everything the model computed beyond band is available to a caller only through a separate path with its own authentication and its own reason to exist.
How to build it
Most important first.
- Return decisions, not distributions. If the consumer acts on a price band, return the band. Keep a separate, authenticated, rate-limited internal endpoint for the consumers who genuinely need probabilities, and log who uses it.
- Authenticate every caller and rate-limit per principal with limits derived from legitimate usage, not from capacity. A limit set at capacity is not a limit.
- Cap spend per principal on GPU endpoints — requests per window and compute per window — and validate inputs before they reach the GPU: size, format, and a cheap classifier that rejects obvious garbage (Serving Fallbacks for what to return when rejecting).
- Log calls per principal and monitor query patterns: a caller sweeping a grid of inputs, or a caller whose volume rose sharply, is the signal that the oracle is being used systematically (Prediction Logging).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Calls per principal per window against the legitimate baseline, and the distribution of inputs per principal — grid-like patterns are the extraction signature.
- GPU spend per principal per day, with a cap. This is the number that turned a weekend into a bill.
- Endpoint availability and latency are the numbers that look relevant and are not: autoscaling kept both perfect while the abuse ran.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Every endpoint returns the minimum its consumers need for their decision, and any richer output is on a separate, authenticated, monitored path with a stated consumer.
- Rate limits and spend caps are per authenticated principal, derived from legitimate usage, and key issuance is controlled so a limit cannot be escaped by rotation.
- Per-principal call volume and input patterns are logged and reviewed, so extraction-shaped usage is a visible pattern rather than a competitor's product launch.
- Offline: estimate how many calls it takes to fit a surrogate model of the endpoint from its outputs, with probabilities and with decisions only. The difference is what output minimisation buys; the number is what the rate limit must stay below.
- Online: send a burst past the rate limit with a valid principal and confirm it is limited; send oversized and malformed inputs to the GPU endpoint and confirm they are rejected before inference.
- Over time: review per-principal volume and input-pattern reports weekly. A principal whose inputs form a grid is the review that finds extraction.
What can go wrong
- Output minimisation is applied to the public endpoint, and the internal probability endpoint is reachable with the same freely distributed API key.
- Rate limits are per key, and the abuser rotates keys through a self-service sign-up that issues them without verification.
- The cheap input-rejection classifier is itself the target: inputs are crafted to pass it and still cost the GPU model the full inference.
- Returning a decision instead of a distribution removes information some consumers relied on, and the internal path for them is another endpoint to secure.
- Rate limits derived from legitimate usage will occasionally limit a legitimate spike, and the first complaint is the argument to raise them to capacity.
- Input validation before the GPU adds latency to every call to reject a minority; the cheap classifier is a second model to maintain.
- "It is read-only, so it is low risk." Each read reveals part of the model. Enough reads reproduce it, and reads on a GPU endpoint cost real money.
- "Autoscaling handles load." Autoscaling converts abuse into spend. Availability was never the exposure; the bill was.
- "Consumers need the probabilities." Some do. Give them a separate, authenticated path and find out how few there are.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat an endpoint is an oracle for the model's function, and that richer outputs reveal more per call, holds for every model and serving mode; the controls are the same for a tabular classifier and an image model.
- SCALE-SPECIFICAn internal endpoint behind service authentication with a handful of consumers needs output minimisation mostly for the privacy exposure; a public endpoint at volume needs all four controls, and a GPU-backed one needs the spend cap before it needs anything else.
- CONTESTEDA serious position holds that output minimisation is the wrong trade for many products: probabilities let consumers set their own thresholds, and a decision-only API forces the provider to choose a threshold for everyone and to re-deploy when it changes. That is a real cost, and the reply is not to withhold from every consumer but to make the rich output an authenticated, rate-limited, logged path with a named consumer rather than the public default.
Where the depth lives
This domain teaches the model and hands the rest off by name.