Which Model Should We Use?
The interview question whose red-flag answer is a product name. A strong answer asks about the data, the latency, the interpretability, the metric and the cost before naming anything.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Someone asks "which model should we use?" — what has to be known before any family can be named, and what does each family cost once it is chosen?
A product manager, in the kickoff meeting: "We want to predict which trial users will convert so sales can call the right ones. Which model should we use? Someone said XGBoost."
Use XGBoost. It wins tabular benchmarks, handles mixed types and missing values, needs no scaling, and every team knows how to run it. Tune it, ship it, move on.
Nobody asked what the prediction is for. The sales team wanted a ranked list of fifty accounts a day with a reason attached, and the model produced a probability per account with no reason. The reps ignored it within a week.
- Nobody asked what the prediction is for. The sales team wanted a ranked list of fifty accounts a day with a reason attached, and the model produced a probability per account with no reason. The reps ignored it within a week.
- Nobody asked when the prediction is made. The model was trained on full fourteen-day usage and scored accounts on day two, where most of its features are still zero — a temporal mismatch the offline metric could not see because validation used the same full-window features (Temporal Leakage).
- Nobody asked about the baseline. A rule — "company size above X and more than three active days" — captured most of the lift, and the ensemble's improvement over it was within the noise of a forty-thousand-row validation set (Metric Uncertainty).
- Nobody asked about the metric. The tuned model maximised AUC across all accounts; the business metric is conversions in the top fifty per day, which depends only on the head of the ranking and was not what the tuning optimised (Business Metrics vs Model Metrics).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a trial account converts to paid within the trial window. The label is the conversion event, known at trial end — fourteen days after signup.
- The decision is which accounts a small sales team calls. The team can call a fixed number per day, so the output is a ranking of accounts, and the metric that matters is conversions among the accounts called.
- One example is one trial account: signup source, company size band, product-usage counts over the first few days, support-ticket count, and whether it converted. Around forty thousand trials over two years.
- Usage features are only meaningful after a few days, so a prediction on day one has almost no signal and a prediction on day ten has much more — the target must say *when* the prediction is made (Target Definition).
- Sales has been calling accounts by their own judgement for two years, so the historical conversions are partly caused by the calls (Selection Bias).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A model family is a choice of inductive bias and cost profile. Linear models assume additive effects and give a coefficient per feature; trees partition the space along axes and handle interactions and missingness natively; boosting stacks many small trees fit to residuals; neural networks learn the representation and need the data to pay for it; k-NN defers everything to a distance (Raw Features vs Learned Representations).
- Data decides more than any benchmark: modality (tabular, text, image, sequence), size, label quality, and whether the features already carry the signal or must be learned. On a few hundred thousand tabular rows with engineered features a tuned boosted ensemble is usually strongest; on raw pixels or free text a pretrained network's representation wins because the hand features do not exist.
- The serving constraints decide the rest: a latency budget of a millisecond on a CPU excludes a large network; a requirement to explain each prediction to a rep or a regulator favours a linear model or a shallow tree; a team that must retrain nightly favours something cheap to fit; a compliance rule may forbid certain features entirely.
- The question "which model" is therefore answered last. It follows from the decision, the data, the metric at the operating point, the latency and cost budgets, the interpretability requirement, and what the simplest baseline already achieves (Decision Before Model, Baselines Are Mandatory).
The questions that come before the family
The strong answer to "which model" is a list of questions, and the interviewer is listening for whether the candidate asks them unprompted. What decision does the prediction drive, and at what moment? What data exists at that moment, and how much of it? Which metric at which operating point maps to the decision? What are the latency, throughput and cost budgets? Who has to understand a prediction, and how deeply? How often will it retrain?
The red flag is a product name given before any of those are known. It is not that boosting is wrong for the trial-conversion problem — it may be the right choice — it is that the answer arrived without the constraints that would justify it, and the same reasoning would give the same answer to every problem.
What do the constraints say, and which family do they point to first?
when Additive effects, a coefficient per feature must be shown, sub-millisecond CPU serving, or under a few thousand rows.
cost Misses interactions unless engineered by hand; needs scaling and encoding reproduced exactly at serving time.
when The prediction must be explained as a rule path to a non-technical user, and a few points of accuracy can be traded for that.
cost High variance; small data changes reshape the tree; usually beaten by an ensemble on the metric.
when Tabular data with interactions, tens of thousands to millions of rows, batch or modest-latency serving, explanations via attribution acceptable.
cost Tuning surface; extrapolates flat outside the training range; per-prediction explanations are approximate.
when The signal is in raw text, images, audio or sequences, or an existing pretrained representation can be adapted; data and compute budget to match.
cost Training and serving cost, GPU dependency, the least interpretable option, and on small tabular data usually no gain.
when The product is "similar cases", or retrieval over embeddings is the mechanism; the reference set is data infrastructure.
cost All the cost at inference; distance quality depends entirely on scaling or the learned representation; needs an index at scale.
The families against the constraints
A matrix invites reading it as a ranking, so read the caveat first. The scores below are for a tabular problem of the trial-conversion shape — forty thousand rows, engineered features, nightly batch scoring, a rep who wants a reason. Change the data modality and the neural row jumps; change the row count to four million and the k-NN and SVM-like options fall off; add a regulator and the linear row is the only one whose explanation is exact.
What the matrix cannot say is which axis dominates. For sales calls, interpretability and the top-k metric dominate and quality differences within validation noise do not; for a fraud model scoring every transaction, latency and quality dominate and the rep's reason code does not exist.
| Option | Quality | Latency | Cost | Interpretability | Data needed | Operational | Note |
|---|---|---|---|---|---|---|---|
| Linear | Coefficients are the reason codes; misses the usage × company-size interaction unless engineered. | ||||||
| Single tree | A rule path per account; unstable between retrains, which reps notice. | ||||||
| Boosting | Likely strongest on the metric; reason codes via attribution, which are approximate. | ||||||
| Neural | No representation to learn here; the extra tuning and serving cost buy nothing at this size. | ||||||
| k-NN | "Accounts like this one converted" is a good reason; the distance on mixed tabular columns is not. |
caveat Every score is a rank on this problem at this size and would be wrong for another. The quality column in particular is a guess inside validation noise: at forty thousand rows the linear model with two interaction features and the boosted ensemble may be indistinguishable, and the matrix has no cell for "the difference is not measurable".
The honest position on neural networks and tabular data
"Neural networks are always better" is the slogan this lesson exists to make falsifiable. Made precise, the claim is: given enough data and compute, a network can learn a representation at least as good as hand-engineered features. On raw text, images, audio and long sequences that claim is borne out and the network is the default. On small tabular data with features an analyst already engineered there is no representation to learn, and the tree ensemble or the linear model usually wins on the metric while being cheaper to train, faster to serve and easier to explain.
The reverse slogan — "boosting for tabular, always" — is the red flag in the other direction. It is the right default on tabular data with tens of thousands to a few million rows and no hard latency or explanation constraint, and a wrong answer wherever one of those constraints binds. The model is chosen by the constraints, and the constraints have to be asked for.
The data size, modality, latency budget, interpretability requirement and retrain cadence that justified the chosen family are the ones in force in production.
holds when The choice was recorded with the reasons; the reasons are reviewed when the data grows, the serving path changes, or a new consumer of the predictions appears.
breaks when The data grows past the family's comfortable range, a real-time use case appears for a batch model, a regulator asks for exact explanations, or the sales process changes so the baseline lift evaporates.
respond Re-ask the questions, not "tune harder". A changed constraint usually calls for a different family or a different decision layer, and occasionally for no model at all (When Not to Use ML).
How to build it
Most important first.
- Ask the questions first, in this order: what decision does the prediction drive, and when is it made; what data exists at that moment; which metric at which operating point maps to the decision; what latency, throughput and cost budget applies; who needs to understand a prediction and how much; how often will it retrain.
- Establish the baseline before any family is discussed: a rule, a mean or majority predictor, then a linear model on scaled features. The candidate must beat the baseline on the operating-point metric by more than the validation noise (Beating the Baseline).
- Choose the family from the constraints, and write down which constraint chose it. "Boosting, because tabular with interactions, forty thousand rows, per-account reason codes acceptable via SHAP, batch scoring nightly" is an answer; "XGBoost" is not.
- Use the decision tree at
/ml/decideas the checklist: every leaf names the trade-off, the simpler thing to try first, and how the chosen approach fails. - Plan the successor. State what change in data size, latency or interpretability requirement would make the chosen family wrong, so the choice is revisited on evidence rather than on fashion.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Conversions among the accounts actually called, versus the rule the sales team used before — measured in an experiment where the model's list and the rule's list are given to comparable reps (A/B Testing Models).
- Precision in the top fifty per day on a time-based validation split, with features computed as of the scoring day. This is the offline number that maps to the decision; AUC over all accounts does not.
- The lift over the rule baseline with its uncertainty — a confidence interval, not a point estimate — because at forty thousand rows the difference between families is often inside it.
- Do not measure "which model has the best validation AUC" and call that a model choice. It is one input, at one operating point that is not the business's.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The constraints that chose the family — data size and modality, latency budget, interpretability requirement, retrain cadence — are still the constraints in force.
- The features available at the scoring moment in production are the features the model was validated on, computed as of the same point in the trial.
- The baseline still performs as measured, so the model's lift over it is still a lift; a change in the sales process can erase it without the model changing.
- The operating-point metric in production is being measured, not only the offline metric the family was selected on.
- Offline: rule, linear and candidate family on the same time-based split with as-of features, reported as top-k precision with confidence intervals.
- Online: an experiment where half the reps get the model's list and half the rule's; the gap in conversions per call is the only number that justifies the more expensive model.
- Over time: a quarterly re-ask of the constraint questions, recorded with the model's lineage, so a family chosen for forty thousand rows is reconsidered at four hundred thousand (Model Lineage).
What can go wrong
- The chosen family fits the constraints today; a year later the data has grown tenfold and latency requirements tightened, and nobody re-asks the questions because the model "works".
- The interpretability requirement was met with a post-hoc explainer whose explanations the reps stopped trusting after two contradicted each other (Explainability).
- The model ranks well but the sales team calls the same top-ranked accounts every day because the list does not change; the top fifty needed exclusion of already-called accounts, which is a decision-layer concern the model choice ignored.
- The baseline rule was never deployed as a fallback, so when the model's feature pipeline broke the sales team had no list at all (Serving Fallbacks).
- Asking the questions first delays the first model by days and produces a baseline the team may find embarrassing to beat; it also produces a fallback, a metric that maps to the decision, and a defensible choice.
- Choosing on constraints rather than benchmarks may pick a family that scores lower on the leaderboard metric and higher on the business one, which is hard to explain to a team that reads leaderboards.
- Writing down the constraint that chose the family creates an obligation to revisit when it changes, which is more discipline than most teams keep.
- "XGBoost wins on tabular data, so the choice is made." It wins on many tabular benchmarks, at their metric, with their data sizes, with no latency or interpretability constraint. Which of those match your problem?
- "Neural networks are always better." On small tabular data a tree ensemble or a linear model usually wins, needs less tuning, and serves cheaper. Networks win where the representation must be learned — text, images, audio, sequences — and where the data pays for it.
- "The model with the best validation AUC is the best model." It is the best at ranking every account against every other, which is not the decision. The top fifty per day, with as-of features, is.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe questions — decision, timing, data, metric, latency, interpretability, cost — apply to every supervised problem regardless of family; only the answers, and therefore the family, change from problem to problem.
- DATA-SPECIFICThe tabular default of "boosting beats networks" holds up to a few million rows with engineered features and flips on raw text, images or audio, where a pretrained representation makes even a linear head on top of it stronger than any tree on hand features.
- CONTESTEDA serious position holds that on tabular data the choice barely matters: a tuned boosted ensemble, a tuned random forest and a well-regularised linear model with interaction features land within validation noise of each other on most business datasets, so time spent choosing a family is better spent on labels, features and the split. That is right more often than practitioners admit; the counter is that the families differ enormously in serving cost, retraining cost and what they can explain, and those differences decide the project even when the accuracy does not.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — the rule baseline deployed as a fallback is the reliability property that makes the model choice safe to get wrong; keeping it exercised and its lift measured is a resilience practice this domain names and another owns.