Online Inference
Request → Features → Model → Prediction → Response, inside a latency budget. The feature fetch is usually the latency, and the timeout and fallback are part of the model's quality, not an infrastructure detail.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A prediction is needed inside a request that a user is waiting on. Where does the time go, and what is served when the model cannot answer in time?
A marketplace shows a fraud check before confirming a checkout. The model itself is fast, yet checkout p99 has climbed past the page budget, and during a recent incident a third of high-value orders were approved by a fallback rule nobody had reviewed in a year.
Put the model behind an endpoint, have the checkout service call it, and measure the endpoint's latency. The model is fast, so the check is fast. If the endpoint is down, approve — a false decline loses a customer, and outages are rare.
The endpoint's reported latency is the model's. The check's latency is the feature fetch plus the model plus the network, and the three feature stores each contribute a tail; at p99 one of them is slow almost every time (Latency Breakdown).
- The endpoint's reported latency is the model's. The check's latency is the feature fetch plus the model plus the network, and the three feature stores each contribute a tail; at p99 one of them is slow almost every time (Latency Breakdown).
- The fallback approves. During a feature-store incident every checkout for twenty minutes is approved with no score, and the fraud that arrives is concentrated in exactly the window an attacker would notice.
- The checkout service retries a timed-out call, doubling load on a feature store that timed out because it was overloaded; the incident lengthens itself (Serving Fallbacks).
- The offline evaluation never saw a timeout. It scored every example with complete features, so the quality it reported was for a system in which the fallback never runs.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a checkout is fraudulent; the label is a chargeback or a confirmed-fraud flag, arriving weeks later.
- The decision is approve, step-up verification, or decline, made synchronously inside the checkout request with a fixed budget for the whole check.
- One example is one checkout with buyer aggregates (orders in the last hour, day, month; chargeback history), device and session signals, and seller features. The aggregates are served from a low-latency store fed by the event stream.
- The model is a modest gradient-boosted ensemble whose own compute time is a small fraction of the budget.
- The feature service calls three stores — buyer, seller, device — and each has its own tail.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- An online prediction is a chain: the request arrives, features are gathered (some from the request, most from stores keyed by entity), the vector is assembled in the artifact's order, the model runs, a threshold and policy produce a decision, and the response returns. Each link has a latency distribution, and the request's latency is their sum along the critical path.
- The feature fetch is usually the largest term and always the one with the fattest tail: it is network calls to stores under other teams' load. The model is a local computation with a nearly fixed cost. Optimising the model shortens the shortest link.
- A timeout converts a slow answer into no answer, and the fallback decides what "no answer" means to the user. That decision has a false-positive and a false-negative rate like any classifier, and it is applied to the requests the system was least able to score — which are not a random sample.
- Freshness is a second constraint on features: the aggregate served must reflect events up to the request, or the model sees a buyer who has not yet made the five orders they made in the last minute (Feature Freshness).
Where the milliseconds go
The endpoint dashboard shows the model taking a few milliseconds and everyone concludes the check is fast. The checkout service sees something else: three concurrent feature fetches, the slowest of which sets the pace, then the model, then the policy, then the response. At the median the fetch is quick. At the tail one of three stores is slow on almost every request, and the tail is where the budget is spent.
The chain is worth drawing because the place people optimise — the model — is the link with the least variance. The place that decides the p99 is the fan-out to stores owned by other teams, and the design levers there are timeouts, concurrency and degradation, not model compression.
The evaluation never saw a timeout
Offline, every example had every feature. The metric described a system in which the feature stores are always fast and the fallback never runs. Online, a few percent of requests time out, and those requests are not random: they arrive during load spikes, during dependency incidents, and — in fraud — when someone is deliberately generating them. The fallback decides those, and the fallback was never evaluated.
The gap between offline and online quality here is not the model getting worse. It is a second, unmeasured decision-maker handling the hardest slice of traffic.
Counts are illustrative, for the requests that fell back in one quarter. Recall is zero by construction. This is the matrix nobody drew, because the fallback was written as infrastructure rather than as the model it is on this slice of traffic.
Validation precision and recall at the operating threshold on a held-out month, with complete features for every example; comfortably better than the rule system it replaced.
Chargeback rate on approved orders noticeably higher than validation predicted, concentrated in a few short windows; checkout p99 above the page budget on weekday evenings.
- 1Requests that timed out on a feature store were approved by a fallback rule with recall near zero; those windows coincided with attack bursts, so the fallback handled the worst traffic.
- 2Degraded vectors — one store timed out, its features defaulted — were scored by the model as if the buyer had no history, which the model reads as low risk.
- 3Peak-hour load on the buyer store pushed its tail past the per-store timeout, so the fallback rate at peak was several times the daily average and nobody had sliced by hour.
Budget, timeout, degrade — in code
The serving code enforces a total budget, fans out to the stores concurrently with per-store deadlines, and assembles whatever came back. A missing store marks the vector degraded rather than failing the request; the policy decides whether a degraded vector is scoreable or goes to step-up. Nothing retries inside the budget. Every response records which path produced it.
The assumption this encodes — that the check stays inside its budget and the fallback rate stays below a ceiling — is what has to hold after deployment, and both are numbers the log makes visible.
Under peak load and dependency degradation, the check's p99 remains inside the page budget and the fallback handles less than a stated fraction of traffic.
holds when Per-store timeouts are shorter than the budget, fetches are concurrent, circuit breakers skip a degraded store, and the fallback ceiling is alerted on.
breaks when A store slows gradually; traffic mix shifts toward entities whose features are cold; a new feature adds a fourth store to the fan-out without re-allocating the budget.
respond Fix the slow link or re-allocate the budget; re-evaluate the fallback as a model and replace it if its error rate is unacceptable. Do not retrain the primary model for a latency problem.
1const BUDGET_MS = 150, STORE_MS = 602 3async function fraudCheck(req: Checkout): Promise<Decision> {4 const t0 = performance.now()5 const [buyer, seller, device] = await Promise.all([6 withDeadline(buyerStore.get(req.buyerId), STORE_MS), // resolves to undefined on timeout, no retry7 withDeadline(sellerStore.get(req.sellerId), STORE_MS),8 withDeadline(deviceStore.get(req.deviceId), STORE_MS),9 ])10 const degraded = [buyer, seller, device].filter((x) => x === undefined).length11 if (degraded > 0 && !policy.scoreableWhenDegraded(degraded)) {12 return record({ path: 'fallback:step-up', latencyMs: performance.now() - t0 })13 }14 const x = assemble(req, buyer, seller, device, artifact.featureOrder) // missing -> shipped fill values, flagged15 const p = artifact.predict(x)16 const decision = policy.decide(p, { degraded }) // threshold from the artifact17 return record({ path: degraded ? 'model:degraded' : 'model', p, decision, latencyMs: performance.now() - t0 })18}The path field is the important output. Without it, fallback decisions and degraded-vector decisions are indistinguishable from model decisions in the log, and their error rate can never be measured separately.
How to build it
Most important first.
- Set the budget for the whole check, then allocate it: feature fetch, model, policy, network, with headroom. Fetch features concurrently with per-store timeouts shorter than the total, and let a missing store degrade the vector rather than fail the request.
- Design the fallback as a model decision with a measured error rate: a conservative rule from features that are still available, or step-up verification, never a blanket approve. Record which fallback was used on every response so its quality can be measured later.
- Do not retry a timed-out feature call inside a request that has a budget; use a circuit breaker so an overloaded store is skipped fast instead of waited on repeatedly.
- Log the full feature vector, the model digest, the latency of each link and the decision path for every request (Prediction Logging); the fallback rate and its outcomes are the numbers this mode is judged by.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- p99 of the whole check, decomposed by link, against the budget. The decision is where to spend engineering effort, and the decomposition is what points at the feature store rather than the model.
- Fallback rate, and precision and recall of fallback decisions on realised outcomes, separately from the model's. A fallback that runs on five percent of traffic with a much worse error rate can dominate the loss.
- Not the model's inference time in isolation, and not the mean latency; the mean hides the tail where the fallback runs.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The end-to-end p99 of the check stays inside the budget, and the fallback rate stays below a stated ceiling, under peak load and during dependency degradation.
- Features fetched at request time reflect events up to the request, within a freshness bound the model was trained to expect.
- The fallback's decisions are logged, distinguishable from model decisions, and their error rate on realised outcomes is known and accepted.
- Offline: replay logged requests with the recorded per-link latencies and simulate the timeout policy, to see what fraction would fall back and what the fallback would have decided against the realised outcomes.
- Online: a load test at peak shape with one feature store injected to be slow, asserting the check's p99 and the fallback rate; continuous per-link latency histograms in production.
- Over time: when labels arrive, compute precision and recall separately for model-scored and fallback-scored requests, and for requests scored with a degraded vector.
What can go wrong
- Degraded vectors — a store timed out, its features defaulted — are scored as if complete, and the model treats a missing aggregate as zero orders, which looks like a new buyer with no history.
- The per-store timeout is tuned once and the store gets slower over months; the fallback rate creeps up, and because fallback outcomes are not measured separately, the loss shows up as "model decay".
- Health checks report the model endpoint healthy while the feature store is down; the load balancer keeps sending traffic to a service that can only fall back.
- The fallback rule is written under incident pressure and never evaluated; it becomes the highest-volume model in the system during every partial outage.
- Online inference buys freshness with infrastructure: a low-latency feature store, a service with an SLO, an on-call, and a fallback whose quality must be managed like a second model.
- A tight per-store timeout lowers the tail and raises the fallback rate; the right point depends on the fallback's error cost, which the offline evaluation never measured.
- Concurrent feature fetching reduces latency and multiplies load on the stores; during their incidents the model service is the amplifier (Throughput vs Latency).
- "The model is fast, so latency is not our problem." The model is the shortest link. The check's latency is the feature fetch, and that is the model team's problem because the fallback that runs when it is slow decides the model's quality.
- "Fail open — a false decline costs a customer." A blanket approve is a classifier with recall of zero on exactly the traffic that arrives during incidents. Its cost is a number; measure it before choosing it.
- "Retry the feature call; it usually succeeds the second time." Inside a request budget a retry doubles load on the dependency that just timed out for load. Skip it, degrade, or break the circuit.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat request latency is the sum of links along the critical path, and that the feature fetch is the fat-tailed one, holds for any model served synchronously against remote features; only the proportions change.
- SCALE-SPECIFICAt low volume features can be fetched from a single database and the timeout policy hardly matters; at high volume with several stores the tail is the product, and the fallback becomes a model in its own right.
- DOMAIN-SPECIFICIn fraud the fallback must be conservative because the traffic during incidents is adversarially selected; in recommendation a fallback to popular items costs a little engagement and fail-open is the right call.
Where the depth lives
This domain teaches the model and hands the rest off by name.