Batch Inference
Dataset → Model → Predictions, on a schedule. When the prediction can be precomputed, batch is the cheapest and most debuggable mode — and the staleness window is a property to design, not a defect.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The product needs a prediction for every customer, but not right now. When can predictions be computed ahead of time, and what does the delay between scoring and use cost?
A retention team wants a list every morning of the customers most likely to cancel this month, so agents can call them. The first version of the system was a real-time API nobody called in real time — the CRM pulled it once a day for every customer, took four hours, and timed out on Mondays.
Deploy the model behind an HTTP endpoint — that is what "deploy a model" means — and let the CRM call it whenever it needs a score. Online serving is the general solution; batch is a special case of it.
The CRM needs every score at once, so the endpoint receives millions of sequential requests every morning; the feature fetch per request dominates, and the run takes hours (Latency Breakdown).
- The CRM needs every score at once, so the endpoint receives millions of sequential requests every morning; the feature fetch per request dominates, and the run takes hours (Latency Breakdown).
- The scores are not reproducible. Each request fetched features at a slightly different time, so two customers scored an hour apart were scored against different states of the same upstream table.
- Nobody can answer "what did we score customer X at on the 3rd" because online predictions were not stored; the call list was built and discarded.
- The endpoint is sized for the morning burst and idle for twenty-three hours, and the on-call is paged for a service whose only consumer runs a cron job.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a subscriber will cancel within the next thirty days; the label is the cancellation event, known at the end of the window.
- The decision is which customers appear on today's call list, so the prediction is consumed once a day in bulk and ranked, not looked up per request.
- One example is one subscriber as of a scoring date, with usage aggregates over the trailing weeks, billing history, support contacts and tenure. All of it lives in the warehouse and is refreshed nightly.
- Training examples were built by picking historical scoring dates and joining features as of each date to the cancellation outcome in the following thirty days (Point-in-Time Correctness).
- There are a few million subscribers, the model is a gradient-boosted ensemble, and scoring everyone takes minutes on a handful of cores.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Batch inference is a job: read a dataset of entities and their features as of a cutoff, run the model over all of them, write a table of predictions keyed by entity and scoring time. The model is loaded once, the features come from one consistent snapshot, and the output is durable.
- Because all inputs come from one snapshot, every prediction in the run is about the same moment. That is what makes the run reproducible, and it is also what defines the staleness window: a prediction made from Monday's snapshot describes Monday, however long it is used.
- The staleness window is the time between the snapshot and the moment the prediction is acted on. If the decision does not depend on anything that happened inside that window — the call list does not change because a customer logged in this morning — batch loses nothing. If it does, the window is a quality cost with a measurable size.
- The consumer reads predictions by joining the prediction table back to the entity — a lookup by key, no model involved. That join is the serving path, and it is a database read.
A job, a table, a join
The shape is three steps. A scoring job reads one consistent snapshot of features as of a cutoff, runs the model across every entity, and writes predictions to a table keyed by entity, scoring time and model digest. The consumer joins that table to the entity when it needs a score. There is no endpoint, no per-request feature fetch, and the model loads once per run.
Each step has a place it fails. The job can start before its inputs landed; the table can be overwritten or half-published; the join can pick the wrong scoring date. None of those is a model problem, and all of them look like one from the product.
- 1Wait for inputs
The orchestrator checks that every feature table's watermark is past the cutoff before the scoring task is eligible.
fails by A time-based schedule instead of a dependency: the job runs at 03:00 whether the features landed or not, and scores yesterday.
- 2Snapshot features
Read every entity's features as of the cutoff, in one query, so all predictions describe the same moment.
fails by Reading a live table mid-refresh, so entities early in the scan see old features and late ones see new.
- 3Score
Load the artifact once by digest, apply the shipped preprocessing, predict, attach scoring time and digest to every row.
fails by One malformed entity raises and the whole run fails; isolate bad rows to a quarantine table and publish the rest.
- 4Check and publish
Assert row count, null rate and score distribution against the previous run; write to staging; swap atomically.
fails by Overwriting in place, so the consumer reads a partial table and past runs are unrecoverable.
- 5Serve by join
The consumer reads the latest published run for each entity; a suppression join removes entities whose state changed since the cutoff.
fails by Joining on entity only, so a consumer silently reads an older run when today's has not published.
The staleness window is the time from step two to step five. It is a number, and it should be measured against quality rather than assumed to be a problem.
Where the training set can lie to a batch job
Batch inference is where a particular leak becomes visible. The training set was built from historical scoring dates joined to features "as of" those dates; if one feature was computed from a table that is rebuilt monthly, its value for a mid-month scoring date includes events after the date — including the cancellation itself. Offline the model looks excellent. The batch job, scoring today from today's tables, cannot see the future and the feature is suddenly weak.
The same feature is fine when its value at scoring time is computed only from events before the scoring time, and that is exactly what the batch job does. The leak was in how the training rows were built, not in the feature.
looks like A count of support contacts in the trailing thirty days, present in the training table and in the nightly feature table with the same name and a sensible distribution.
why it leaks The training table was built from a monthly snapshot of the support system, so a customer scored on the 10th carries contacts through the 30th — including the cancellation call. The model learns that a high count means cancellation, because for training rows it often follows it.
fix Build training features with the same as-of computation the batch job uses — a point-in-time join against event-level data, never a periodic snapshot rebuilt after the fact (Temporal Leakage).
The staleness window is the assumption
Batch inference works because the decision does not depend on what happened since the snapshot, or because the cost of ignoring it is small. That is an assumption about the product, and it can stop holding without anyone changing the model — a new feature in the app makes yesterday's usage aggregate a poor description of today, or the retention team starts calling customers within hours of a support complaint.
The way to know is to measure quality as a function of prediction age on realised outcomes. A flat curve says the window is free; a falling one says how much a shorter cycle would buy.
The value of the prediction to the decision does not fall materially between the snapshot cutoff and the moment of action.
holds when The decision is made on a daily rhythm, the features move slowly relative to the cycle, and a suppression join handles the few entities whose state changed since the cutoff.
breaks when The decision starts reacting to intra-day events; a new product surface makes the features move faster; the cycle lengthens because the job got slower and nobody re-measured.
respond Shorten the cycle if the curve falls within the day; add an online rerank over the batch candidates if it falls within the hour (Choosing the Inference Mode); do not retrain — the weights are not what aged.
How to build it
Most important first.
- Schedule the scoring job in the orchestrator after the feature tables it depends on have landed, with the cutoff as an explicit parameter so a rerun for a past date produces the same scores (Reproducibility).
- Write predictions to a table keyed by
(entity, scoring_time, model_digest), append-only, so every score ever acted on can be found and so the future training set can use realised outcomes joined to the score that was live (Prediction Logging). - Publish atomically: write the run to a staging table and swap, so a consumer never reads half a run, and include the row count and score distribution as run-level checks that block publication when they move (Prediction Drift).
- Define the staleness window as a product requirement and measure it: how old is the prediction the agent sees, and how much does quality fall with age? If the fall is steep, the answer is a shorter cycle or a hybrid, not "go online" (Choosing the Inference Mode).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Precision of the call list at the list size the team can work, measured on realised cancellations thirty days later. This is the number the retention team is buying.
- Age of the prediction at the moment of action, and quality as a function of that age — the number that says whether the batch cycle is short enough.
- Not endpoint latency. There is no endpoint; the consumer's read latency is a database property and is not where the quality is decided.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The decision the prediction feeds does not depend on events that occur inside the staleness window, or the cost of ignoring them has been measured and accepted.
- The feature snapshot the job reads is complete and as-of the intended cutoff every run — the upstream tables landed before the job started.
- Every prediction acted on is stored with its scoring time and model digest, so outcomes can be joined back to the score that was live.
- Offline: rerun the job for a past cutoff and diff against the stored predictions — identical output is the reproducibility check; a scheduled check that the feature tables' watermark is past the cutoff before scoring starts.
- Online: publish-time assertions on row count, null rate and score distribution against the previous run; an alert when the job has not published by the time the consumer reads.
- Over time: join realised cancellations to the score that was live and compute list precision by prediction age; a steep decline with age is the argument for a shorter cycle.
What can go wrong
- The scoring job runs before the feature refresh lands, scores yesterday's features, and publishes a plausible table; every downstream check passes because the distribution is the same as yesterday — because it is yesterday (Feature Freshness).
- A customer cancels at 09:00 and is called at 10:00 from a list scored at 03:00; the staleness window is the visible symptom, and the fix is a suppression join at read time, not online inference.
- The prediction table is overwritten in place each morning, so when the model is found to have been wrong in March there is no record of what March's scores were.
- The batch is huge and one entity with a malformed feature fails the whole run; the morning list is missing and the agents call nobody.
- Predictions are stale by construction; the window is designed, but it exists, and any decision that needs the last hour's events cannot be served this way.
- Scoring every entity every cycle spends compute on entities nobody will look at; the cost is usually small, but at very large populations the job becomes a data-engineering problem in its own right.
- A durable prediction table is storage that grows without bound and holds personal data; retention and access policy become part of the model's design.
- "Batch is the legacy option; a real ML system is online." Batch is the correct mode whenever the prediction can be precomputed, and it is cheaper, more reproducible and easier to debug. The churn call list has no use for a sub-second score.
- "The predictions are a day old, so they are wrong." They describe the state a day ago. Whether that is wrong depends on how much the decision changes in a day, which is a measurement, not a feeling.
- "We'll score in batch and serve from the table, so there is no serving problem." The join back to the entity is the serving path. If it reads a half-published run, or the wrong scoring date, the product sees wrong predictions from a correct model.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat a prediction which does not depend on events since the snapshot can be precomputed holds for any model and task; the model family only changes how long the job takes.
- SCALE-SPECIFICScoring a few million rows on a schedule is a single job; scoring billions of pairs, as in candidate generation for a large catalogue, becomes a distributed data-processing problem whose design lives in the data platform rather than in the model.
- TASK-SPECIFICRanking lists and risk scores consumed by people tolerate a daily window; a fraud decision on a transaction in flight, or a recommendation that must reflect what was clicked a minute ago, does not — those decisions need the online or streaming modes, or a hybrid.
Where the depth lives
This domain teaches the model and hands the rest off by name.