The Questions Before the Boxes
Ten questions — target, latency, batch or online, freshness, volume, label delay, model size, fallback, retraining, cost — each of which decides a part of the architecture before any model is chosen.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Which questions have to be answered before an ML system can be designed, and what does each answer decide?
A senior engineer is asked in a design review to "sketch the ML architecture for the new risk feature". They have a whiteboard, forty minutes, and a product manager who has said only that "it should use ML". They need to know what to ask before drawing anything.
Ask what the data looks like, pick the model family that usually wins on that data, then work out how to deploy it. Model first, because the model is the hard part and everything else is plumbing.
The model that wins offline needs a GPU to serve at the required latency, and the latency budget was never asked. The deployment is redesigned around the model instead of the model around the decision.
- The model that wins offline needs a GPU to serve at the required latency, and the latency budget was never asked. The deployment is redesigned around the model instead of the model around the decision.
- The label arrives ninety days after the prediction and nobody planned a proxy, so for a quarter the only evidence the system works is that it is returning numbers.
- The feature that carried the model needs a value fresher than the batch pipeline produces. Serving computes it live from a different source; training never saw that source.
- Volume was assumed to be "the users", which meant every user every hour, which meant a cost that a weekly batch over the active accounts would have been a hundredth of.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- The target of the surrounding system is undefined — that is the first question. Until someone can say what event is predicted, at what horizon, and what action follows, there is no architecture to draw (Target Definition, Decision Before Model).
- What this lesson optimises is the order of the questions: each answer constrains the next, and answering them out of order forces later answers by earlier accidents.
- The data question is folded into several of the ten: what exists at prediction time (freshness), when the label arrives (delay), and how many predictions are made (volume). Each is a fact about the data platform the design has to accept or change.
- One training example is defined by the target and the horizon together — an entity at a moment, with features from before and a label from after — and the freshness question decides whether that moment is a batch snapshot or a request timestamp (What Is One Example?).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Each question maps to a design decision, and the mapping is nearly mechanical once the question is asked. Latency and volume together decide the inference mode (Choosing the Inference Mode); freshness decides whether features are looked up or computed (Feature Freshness); label delay decides what monitoring can see and when (Ground-Truth Delay); model size and latency decide the hardware (Inference Cost); the cost of a wrong prediction decides the threshold and the fallback (Serving Fallbacks).
- The questions are ordered by how much each constrains the rest. The target comes first because it defines the label, the horizon and the decision; nothing else can be answered without it. Latency and batch-versus-online come next because they fix the shape of serving. Label delay, freshness and volume follow because they are facts about the data the design must live with. Model size, fallback, retraining and cost are answered last because they are consequences.
- The reason the order matters is that a late answer cannot undo an early one. A model chosen before the latency budget will be served on hardware chosen to fit it; a feature chosen before the freshness question will be computed on a path chosen to produce it. Every "how did we end up with this architecture" story is a question answered late.
Ten questions, what each decides, where the depth lives
The matrix is the lesson. Each row is a question that must be answered before a box is drawn, the design decision that the answer fixes, and the lesson that treats the decision in depth. Read the "decides" column as the reason the question cannot be skipped: an unanswered question is a decision that will be made by accident.
The order is deliberate. Rows near the top constrain rows below them; a row answered out of order is usually answered wrong, because the constraint it depended on was not yet known.
| Question | What the answer decides | Depth |
|---|---|---|
| What is the target? | The label, the horizon, the decision, and whether ML applies at all | Target Definition, When Not to Use ML |
| What latency does the decision need? | Online vs batch, hardware, and whether the model may be large | Choosing the Inference Mode, Latency Breakdown |
| Batch or online? | The whole serving architecture and the on-call footprint | Batch Inference, Online Inference |
| How fresh must features be? | Lookup vs live computation; whether a stream is needed | Feature Freshness, Streaming Inference |
| What is the prediction volume? | Cost, batching, and whether precomputation is cheaper | Inference Batching, Inference Cost |
| When does the label arrive? | What monitoring can see, and which proxies must exist at launch | Ground-Truth Delay, Model Monitoring |
| How large may the model be? | CPU vs GPU, quantization, and the memory budget | CPU or GPU for Inference, Quantization |
| What happens when the model is unavailable? | The fallback, and the rule baseline it usually is | Serving Fallbacks, The Rule Baseline |
| How often must it be retrained? | Pipeline shape: triggered vs scheduled; the promotion gate | Retraining Strategies, Champion / Challenger |
| What may it cost? | The ceiling on everything above; per-prediction cost as a promotion criterion | Inference Cost, ML Cost Optimisation |
Latency, volume and freshness decide the mode
Three of the ten questions interact to fix the inference mode, and the interaction is worth making explicit because it is the most common place designs go wrong. When the decision is made — page load, nightly, weekly — sets the latency; how many entities need a prediction sets the volume; whether the prediction depends on something that happened seconds ago sets freshness.
The decision below is the same one the Choosing the Inference Mode lesson and the inference lab work through. The point here is that all three inputs are facts about the product, discoverable by asking, and none of them is a property of the model.
When is the prediction consumed, for how many entities, and does it depend on the last few seconds?
when The decision is consumed on a schedule (a call list, a nightly report), the population is enumerable, and features change slowly relative to the schedule.
cost Predictions are stale by up to one schedule interval; a consumer who later wants a fresh score has to be told no, or given a separate design.
when The decision is made inside a request and depends on request-time context that cannot be precomputed — the transaction being scored, the query being typed.
cost A model server, a feature lookup path with its own freshness, a latency budget, a fallback and an on-call rotation — for every prediction, whether or not anyone would have noticed staleness.
when Most of the signal is precomputable and a small part is request-time — a nightly candidate set re-ranked with session context.
cost Two pipelines with a join between them; the precomputed half has one freshness, the online half another, and the boundary must be logged or it cannot be debugged.
The answers are assumptions, and they move
Every answer written down at design time is an assumption the deployed system depends on. The retention team's capacity sets the threshold; if the team shrinks, the threshold is wrong and no monitor on the model will say so. The label delay sets the monitoring lag; if the billing system starts reporting cancellations earlier, the proxies are now stale rather than early.
So the design document is not an artifact of the design phase. It is the list of assumptions the monitors should check, and the section that ages worst is the one about the decision consumer, because that is the one the ML team does not control.
The consumer, the latency budget, the capacity behind the threshold and the label delay are the ones written down at design time.
holds when The application that calls the model is the one the design named, its use of the prediction has not changed, and the business process behind the decision is intact.
breaks when A second application starts calling the same endpoint with a different latency need; the retention team is halved; a new billing system changes when cancellations are recorded; the product adds a real-time surface to a batch-designed model.
respond Re-answer the questions for the new consumer as a separate design. Do not stretch the existing system to serve a decision it was not designed for and then blame the model.
How to build it
Most important first.
- Ask the ten questions in order and write the answers down before the architecture. The matrix in the sections below is the template: question, what it decides, and the lesson where the depth lives.
- Answer with the decision, not the model: "batch, weekly, top-200 list" is an answer; "gradient boosting" is not. The model family is chosen last, from the answers, and the candidate is the simplest thing that meets them (Baselines Are Mandatory).
- For each answer, record what it rules out — online inference ruled out by the weekly decision, deep models ruled out by the interpretability requirement — so a later reviewer can see the constraint rather than re-litigate it.
- Treat "we do not know yet" as an answer that names a data-collection task, not as permission to skip the question. An unknown label delay is a project blocker, not a detail.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The number that maps to the decision is the cost of a wrong prediction in each direction, in the business's units — that is what the threshold, the fallback and the metric are derived from (Business Metrics vs Model Metrics).
- For the design itself, count the questions that were answered with a fact about the decision rather than a preference about the model. Ten is a design; four is a model with plumbing.
- Offline model metrics are not a measurement of the design. They are a measurement of one box, on data whose relevance the other nine questions decide.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The answers to the ten questions remain true after deployment — the decision consumer, the latency budget, the label delay and the capacity the threshold was set for have not moved without the design being revisited.
- The decision the system serves is still the decision it was designed for; a second consumer with a different latency or freshness need is a second design, not a config change.
- The cost of each kind of mistake, as written down at design time, is still what the business would say today.
- Offline: have someone outside the ML team read the ten answers and confirm each is a fact about the product rather than a preference of the modeller.
- Online: check each answer against production after the first month — actual p99 latency against the budget, actual label delay against the assumed one, actual volume against the estimate.
- Over time: revisit the answers on every consumer change. A new application calling the model is a new set of answers, and the old design may not serve it.
What can go wrong
- The questions are answered by the ML team alone and the answers about the decision are guesses; the retention team's real capacity turns out to be a third of the assumed one and the threshold is wrong from day one.
- The retraining question is answered "monthly" as a placeholder and becomes policy; the pipeline retrains on data contaminated by its own interventions because nobody revisited it.
- The fallback question is answered "return the previous score" and the previous score is in a cache that expires; the first outage returns nulls to an application that treats null as zero risk.
- Ten questions take time to answer honestly, and several require conversations with people who are not engineers; the pressure to skip to the model is real and the cost of skipping is invisible for a quarter.
- Designing to the decision rather than to the model can leave accuracy on the table — a simpler model chosen for a latency budget may be measurably worse offline, and the design has to defend that.
- "Which model? XGBoost." Which data, what latency, how interpretable, which metric, at what cost — the answer to "which model" is downstream of nine other answers, and a reflex answer means none of them were asked (Which Model Should We Use?).
- "We will decide batch versus online once we see how fast the model is." The mode is decided by when the decision needs the prediction, not by how fast the model runs. A slow model in a batch system is a cost problem; a fast model in an online system that did not need to exist is an on-call rotation.
- "The label delay is a monitoring detail." It is the design constraint that decides what the team can know and when, and therefore what proxies must be built before launch.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe ten questions apply to any predictive system; the answers, and which questions dominate, are entirely system-specific — fraud is dominated by latency and label delay, churn by capacity and horizon.
- SCALE-SPECIFICAt small volume several questions collapse — cost, model size and volume all resolve to "it fits on one machine" — and the design conversation is mostly about the target, the label delay and the fallback; at large volume every question has a dollar answer.
- CONTESTEDSome experienced teams argue the questions cannot be answered before a prototype exists, because the product owner does not know the decision until they see predictions; they build a throwaway model first and ask the questions second. That is a legitimate discovery process, provided the throwaway is actually thrown away rather than deployed with the questions still unanswered.
Where the depth lives
This domain teaches the model and hands the rest off by name.