Choosing the Inference Mode
How fresh must the prediction be, can it be precomputed, does it need live features, what is the latency budget, what is the volume — those five questions decide batch, online, streaming or hybrid. Online is often unnecessary, and the churn case shows why.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Before drawing the serving architecture: which questions decide whether predictions are computed in a nightly job, on request, on every event, or some combination?
A product team asks for "real-time churn prediction". Pressed, they want a risk score visible in the account page and a daily list for the retention desk. A second team asks for "recommendations" and needs a home feed that reflects what the user did thirty seconds ago, for tens of millions of users. Both were about to be built as one online endpoint.
Build an online endpoint. It is the general case: any consumer can call it whenever, and if the batch list needs scores it can call it in a loop. One architecture, one team, one deployment.
The churn endpoint is called a few million times each night by the CRM and a few hundred times a day by the account page; it is sized for the burst, idle otherwise, and every score it returns could have been computed from yesterday's snapshot for a fraction of the cost (Batch Inference).
- The churn endpoint is called a few million times each night by the CRM and a few hundred times a day by the account page; it is sized for the burst, idle otherwise, and every score it returns could have been computed from yesterday's snapshot for a fraction of the cost (Batch Inference).
- The recommendation endpoint cannot score millions of candidates per request inside the page budget, so the team caps candidates at whatever is fast, and the feed quality is decided by an arbitrary limit rather than by the model (Candidate Generation vs Ranking).
- Neither team asked how fresh the prediction needs to be. Churn was given seconds of freshness it cannot use; recommendations were given a slow endpoint that cannot deliver the freshness it needs.
- The "general" architecture is the most expensive mode for the churn case and the wrong mode for the recommendation case, and it looks like one system on the diagram.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The churn system predicts thirty-day cancellation per subscriber; the recommendation system ranks catalogue items per user. Neither target says anything about when the prediction must be computed — that comes from the decision.
- The decision for churn is a daily call list and a slowly-moving badge; the decision for recommendations is a feed rendered on open that must reflect recent activity.
- Churn features are daily aggregates in the warehouse; nothing the model uses changes faster than a day, and the population is a few million subscribers.
- Recommendation features include per-user long-term history, which changes slowly, and the last few interactions, which change by the second; the candidate space is a catalogue of millions per user.
- Both teams have a warehouse, an event stream and a backend; neither has a low-latency feature store yet.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Five questions decide the mode. Freshness: how old may the prediction be when acted on? Precomputability: is the set of entities to score known in advance, and are their features known in advance? Live features: does the prediction depend on anything only knowable at request time? Latency budget: how long may the consumer wait? Volume: how many predictions per unit time, and how bursty?
- Batch fits when the entities and features are known ahead and the freshness tolerance is longer than the cycle. Online fits when the request carries information the prediction needs, or the set of entities is not enumerable ahead, and the budget allows a feature fetch plus the model. Streaming fits when the prediction must change on every event and the event rate defeats per-event fetching (Streaming Inference).
- Hybrid is the common answer for large candidate spaces: a batch job precomputes the expensive, slowly-changing part — candidates per user, long-term embeddings — and an online step scores a small set with live context inside the budget. Each part runs in the mode its freshness and volume demand.
- The mode is a property of the decision, not of the model. The same weights serve a daily list from a table and an on-request score from an endpoint; what changes is when the features are read and where the cost is paid.
Five questions, four answers
The decision is a tree rooted in freshness. If the prediction may be hours old, it can be precomputed, and batch is the answer unless the set of entities cannot be enumerated ahead. If it must reflect the request itself — the transaction in flight, the query just typed — online is forced, and the remaining questions size its budget. If it must change on every event at a rate that defeats fetching, streaming. If the expensive part is slow-moving and the fresh part is small, hybrid.
The device below is the decision; the interactive tool at /ml/inference walks the same questions with reasons and costs for each leaf. The point of both is that "online" has to be earned by an answer, not assumed.
Given freshness, precomputability, live features, budget and volume, where is the prediction computed?
when Freshness tolerance is longer than a practical cycle; entities and features are known ahead; no request-time information is needed.
cost A staleness window by construction; a scheduled job with dependencies; a prediction table to publish atomically and retain.
when The prediction needs request-time information or the entity set is not enumerable ahead; the budget allows a feature fetch plus the model; volume is within what a service can sustain.
cost A low-latency feature path, a service SLO, timeouts and a fallback evaluated as a model, an on-call.
when The prediction must update on every event; features are per-key event history; the event rate defeats per-event fetching.
cost Stateful operators, checkpoints, watermark and late-event policy that training must match, partition balance.
when A large, slow-moving part can be precomputed and a small, fresh part must be scored with live context inside the budget — the candidate-generation-then-rerank shape.
cost Both modes' operational costs plus a boundary that has to be evaluated and scaled separately.
Churn: the case for not being online
The retention desk works a list each morning; the account page shows a risk badge that nobody expects to change mid-session. Every feature the model uses is a daily aggregate. The freshness answer is "a day"; the entities are every subscriber; nothing in the request matters. Batch scores everyone overnight, publishes a table, and the account page reads the table through a thin lookup — which is the "endpoint" the product asked for, serving a precomputed number in a millisecond with no model involved (Designing a Churn Prediction System).
What was almost built instead: an online endpoint fetching daily aggregates from a feature store per request, called millions of times each night by the CRM, idle by day, with a fallback nobody designed. Same weights, same predictions, an order of magnitude more infrastructure, and a fresh score for a decision that is made once a day.
Per-request feature fetch from a store fed by the warehouse; CRM loops over every subscriber nightly; account page calls on view; endpoint sized for the nightly burst; timeout and fallback unspecified.
Nightly job scores every subscriber from one snapshot and publishes a keyed table; the CRM reads the table; the account page reads one row by key; the fallback for a missing row is yesterday's row, by construction.
The decision has a daily freshness tolerance and no request-time inputs, so batch delivers identical predictions with a reproducible snapshot, a durable record of every score acted on, and no service to keep inside a latency budget.
Recommendations: the case for hybrid
The home feed must reflect what the user did thirty seconds ago and choose from millions of items for tens of millions of users. No single mode fits: batch cannot deliver the freshness for the live part, and online cannot score millions of candidates inside the page budget. Split at the point where the features change speed. Long-term taste and the candidate set move slowly — precompute a few hundred candidates per user in batch. Session context moves by the second — rerank those candidates online with the last few interactions as features.
The boundary is a design decision that can be evaluated: the batch side is judged on candidate recall, the online side on ranking quality given those candidates, and the freshness of each side is set by its own answer to the first question.
- 1Nightly candidate generation
For every user, retrieve a few hundred candidates from long-term history and embeddings; publish to a keyed store.
fails by Candidate recall is never measured, so the online reranker is judged for items it was never given.
- 2Stream keeps session features fresh
Recent interactions are aggregated per user into a low-latency store within seconds.
fails by Freshness bound is unmeasured; the reranker sees a session from a minute ago and looks like a worse model.
- 3Online rerank on open
Fetch the user's candidates and session features, score a few hundred items inside the budget, return the top of the list.
fails by Timeout on the session store falls back to the batch order — acceptable if it is measured, invisible if it is not.
- 4Log which part did what
Every impression records the candidate source, the model digests on both sides and whether the rerank ran or fell back.
fails by One quality number for the feed, so a regression cannot be attributed to either side.
The batch side answers "what could this user want", the online side answers "what now". Each runs in the mode its freshness demands, and neither has to do the other's job.
How to build it
Most important first.
- Answer the five questions in writing before choosing, with numbers: freshness in minutes or hours, volume per hour at peak, budget in milliseconds. "Real-time" is not an answer to the freshness question; a number is.
- Default to batch when the answers allow it. It is the cheapest, most reproducible and most debuggable mode; the burden of proof is on the mode that needs infrastructure.
- When part of the prediction needs freshness and part does not, split it: precompute the slow part in batch, serve the fast part online over a small candidate set, and make the boundary explicit so each side can be evaluated and scaled on its own.
- Re-ask the questions when the product changes. A badge that becomes a push notification, or a list that becomes an in-session trigger, changes the freshness answer and therefore the mode.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Quality of the decision as a function of prediction age, on realised outcomes. For churn, list precision by age is flat over a day, which is the evidence batch loses nothing; for recommendations, engagement falls within minutes of staleness, which is the evidence the live part must be online.
- Cost per thousand predictions actually acted on, per mode. Online churn scoring pays a feature fetch and an idle endpoint for a score nobody reads for hours (Inference Cost).
- Not "requests per second the endpoint can handle". Capacity is a consequence of the mode, and measuring it first mistakes the answer for the question.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The freshness tolerance, latency budget and volume that decided the mode are still the product's actual requirements; a new consumer with different answers has not been quietly attached.
- For a hybrid, the boundary between the precomputed and the live parts still matches where the features change speed: the batch side stays slow-moving and the online side stays small enough for the budget.
- The chosen mode's freshness is actually delivered — the batch cycle publishes on time, the online path stays inside its budget, the stream keeps its lag bounded.
- Offline: simulate each mode's freshness against the historical decisions — score with features as of an hour, a day and a week before each decision and measure quality at each age; the curve is the freshness answer.
- Online: for a hybrid, log which part produced each element of the response so the batch and online contributions can be evaluated separately when outcomes arrive.
- Over time: re-run the age-versus-quality analysis quarterly and whenever a new consumer is attached; a curve that has steepened is a product change that the mode has to follow.
What can go wrong
- The freshness answer is given by the loudest stakeholder rather than measured; "real-time" is chosen, and the cost is paid forever for a decision made on a daily rhythm.
- The hybrid boundary is drawn where the engineering was convenient rather than where freshness changes; the online step re-computes features the batch step already had, and the batch step is too stale for its part.
- A batch mode chosen correctly is later consumed by a new surface that needs freshness, and nobody re-asks the questions; the product experiences a stale prediction as a wrong one.
- Choosing batch commits to a staleness window and a scheduled job; choosing online commits to a feature store, a service SLO and a fallback; choosing streaming commits to stateful operators. Each is a different operational shape, and switching later is a rebuild.
- A hybrid gets each part's benefits and both parts' operational costs, plus a boundary that has to be evaluated on its own.
- Answering the five questions with numbers takes a measurement most teams have not done, and the honest answer is often "we do not know yet" — which argues for batch first and instrumentation.
- "Real-time is better, so build online and batch is free." Online scoring of a daily decision buys freshness the decision cannot use, at the cost of a feature store, an endpoint, an on-call and an unmeasured fallback. Freshness is a requirement to be measured, not a virtue.
- "The data arrives as a stream, so we need streaming inference." The data's arrival mode and the prediction's mode are independent; a stream landing in a table scored nightly is a batch system.
- "One endpoint can serve every consumer, so the mode question is premature." It can, and it makes every consumer pay the most expensive mode. The question is answered by the consumers' decisions, and they differ.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat the mode follows from freshness, precomputability, live features, budget and volume — and not from the model or the data's arrival pattern — applies to any predictive system; the answers differ per product, the questions do not.
- DOMAIN-SPECIFICIn fraud and bidding the request carries the information the prediction needs, so online is unavoidable and the question is the budget; in retention, credit scoring and demand planning decisions are made on a rhythm and batch nearly always wins.
- CONTESTEDA credible position holds that building online from the start is right even for batch-shaped decisions, because products reliably grow surfaces that need on-demand scores, and retrofitting an endpoint onto a batch system later is a rebuild under pressure. That is a real cost; the counter is that the online system carries its infrastructure and fallback costs for every month the on-demand surface does not exist, and a batch table with a thin lookup endpoint covers most "on-demand" needs until the freshness curve says otherwise.
Where the depth lives
This domain teaches the model and hands the rest off by name.