Regression
The output is a number. The loss decides which errors that number is allowed to make, and the business rarely agrees with squared error about which errors are expensive.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
The model outputs a continuous value. Which errors is it trained to avoid, and are those the errors that cost the business money?
A food-delivery operations lead says: "Tell customers when their order will arrive. If we are late they cancel; if we say sixty minutes for a twenty-minute order they do not order at all."
Minimise mean squared error on duration. Squared error is differentiable, standard, and gives the conditional mean, which is the best single guess. Report RMSE on a held-out set and show the predicted mean to the customer.
The conditional mean is pulled up by the two-hour tail, so the typical order is quoted several minutes later than it will arrive. Customers see a slow service; the model is doing exactly what MSE asked.
- The conditional mean is pulled up by the two-hour tail, so the typical order is quoted several minutes later than it will arrive. Customers see a slow service; the model is doing exactly what MSE asked.
- Where the tail is thin — short distances, quiet hours — the mean is close to the median and the quote is early about half the time, and every early quote is a cancellation risk.
- The dropped cancellations were the slowest orders, so the training distribution is faster than reality and the model is optimistic exactly on the orders that will be cancelled again.
- RMSE on the held-out set looked fine because held-out orders were also survivors, and the metric weights the tail heavily in a way the customer never sees.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict delivery time in minutes from order confirmation to handover. The label is the observed duration from the courier app, which exists only for completed deliveries.
- The decision is the number shown to the customer, and the cost of that number is asymmetric: an estimate that is ten minutes early costs a cancellation, one that is ten minutes late costs a little demand (Prediction vs Decision).
- One example is one completed order: restaurant, distance, hour, day, courier load at confirmation, kitchen prep history, weather, and the realised duration.
- Cancelled orders have no duration and are dropped, which removes exactly the orders that ran longest (Survivorship Bias).
- The duration distribution has a long right tail: most orders take thirty to forty minutes, a few take two hours because a kitchen lost the ticket.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A regression model outputs a point estimate; the loss decides what that point is. Squared error is minimised by the conditional mean; absolute error by the conditional median; a pinball loss at quantile q by the q-th conditional quantile. Choosing the loss is choosing which statistic of the outcome distribution the model reports.
- The mean is the right statistic when the cost of error is symmetric and quadratic. Delivery quotes have a cost that is asymmetric — early is worse than late — and roughly linear, which points at a quantile above the median, not the mean.
- Every regression metric summarises the residuals with its own weighting: RMSE amplifies large errors, MAE counts them linearly, MAPE divides by the true value and explodes near zero (MSE, RMSE and MAE, MAPE and Its Caveats). The metric that maps to the decision is the one whose weighting matches the cost.
The loss picks the statistic
A regression model outputs one number per input, and the training loss decides which number. Squared error makes it the conditional mean, absolute error the conditional median, and the pinball loss at level q the q-th conditional quantile. These are three different models of the same data, and they differ most where the outcome distribution is skewed.
Delivery durations are skewed. The mean sits above the median because of the long tail, so the MSE model quotes later than typical for every order and still early for the slow ones. The quantile model can be asked to be early rarely, which is what the business wants.
1import numpy as np2 3def mse(y, p): return np.mean((y - p) ** 2) # minimised by the conditional mean4def mae(y, p): return np.mean(np.abs(y - p)) # minimised by the conditional median5def pinball(y, p, q): # minimised by the q-th conditional quantile6 r = y - p7 return np.mean(np.maximum(q * r, (q - 1) * r))8 9# durations for one restaurant segment: skewed, long right tail10y = np.concatenate([np.random.normal(34, 4, 950), np.random.normal(95, 20, 50)])11print(y.mean(), np.median(y), np.quantile(y, 0.8))12# mean > median: an MSE model quotes later than the typical order13# the 0.8-quantile is later still, and is early on only one order in fiveThe three numbers on the last line are three defensible quotes for the same segment. Which one the customer sees is a decision about the cost of being early against the cost of looking slow, and the loss function makes it whether or not anyone did.
The survivors are fast
Cancelled orders never got a realised duration, so they are missing from the training set. But they were cancelled because they were slow, so the missing rows are the tail of the distribution. The model learns a world where slow orders are rarer than they are, and quotes accordingly.
This is the regression version of leakage: the label exists only for the rows where a certain outcome did not happen, and the model is optimistic on exactly the rows that will produce that outcome again.
looks like A clean training table with one duration per order and no nulls, produced by an inner join to the courier handover event.
why it leaks The inner join keeps only orders that completed. Completion is correlated with speed, so the label is available only for the fast part of the distribution and the model never sees the slow part it will be asked about.
fix Keep cancelled orders with their elapsed time as a right-censored observation, or re-weight completed orders by the inverse of their estimated completion probability.
Which number maps to the decision
The decision is a quote, and the business outcome of a quote is whether the order arrived on or before it and whether the customer placed the order at all. So the numbers to watch are late-rate and cancellation-rate per segment, and the quantile coverage that connects them to the model.
The assumption behind the quote is that the quantile the model was trained at is still the right trade between the two costs, and that it still holds per segment.
The share of orders arriving on or before the quote is still close to the target quantile in every high-volume segment, and that quantile still reflects the business's cost of early against late.
holds when Per-segment coverage is checked monthly against the target and is within tolerance; the cost ratio in the model documentation matches what operations currently believes.
breaks when A restaurant's prep-time distribution shifts; courier supply changes so the tail lengthens; the business decides cancellations matter less than demand and the quantile should move.
respond If coverage slipped in one segment, look at that segment's data before retraining. If the cost ratio changed, re-derive the quantile and retrain at the new level; the old model is not wrong, it is answering a different question.
How to build it
Most important first.
- Write the cost of an error as a function of its sign and size, and pick the loss whose minimiser is the statistic that cost prefers. For delivery quotes, a quantile loss at the level that balances cancellations against lost demand (Choosing a Regression Metric).
- Model the tail explicitly or report an interval: a point estimate cannot say "usually thirty-five, occasionally ninety", and the customer would rather know.
- Include the cancelled orders with their elapsed time as a censored lower bound, or at minimum re-weight, so the training distribution is not the survivors' distribution (Dataset Construction).
- Evaluate on the residual distribution by segment — distance band, hour, restaurant — not one global number, because the tail lives in specific segments (Evaluation Slices).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The share of orders arriving after the quoted time, and the cancellation rate as a function of quoted minus realised. Those are the decision numbers.
- Residual quantiles per segment: the median residual says whether the quote is systematically early or late for that segment; the ninetieth percentile says how bad the tail is.
- Global RMSE is dominated by the two-hour orders and moves for reasons the customer never experiences. Track it as a training diagnostic, not as the product metric (R² (Coefficient of Determination) has the same problem).
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The relationship between the cost of early and late errors that fixed the quantile is still the business's cost, and the quantile is re-derived if it changes.
- The training distribution of durations includes the slow orders — via censoring or re-weighting — and the retraining pipeline preserves that handling.
- Per-segment residual distributions stay close to those at validation time, so a quantile fitted on the whole population still holds in each segment.
- Offline: coverage of the chosen quantile on a time-based holdout, per segment — if the q-th quantile is quoted, a share q of orders should arrive on or before it in every segment.
- Online: late-rate and cancellation-rate per segment on live orders, with the quote logged alongside the realised duration.
- Over time: a monthly check of residual quantiles per restaurant, and an alert when the median residual for any high-volume restaurant moves beyond a stated tolerance.
What can go wrong
- The quantile is raised to reduce cancellations and the quoted times get so long that demand drops; the metric that was optimised improved and revenue fell.
- A restaurant changes its kitchen process and its prep-time distribution shifts; the model keeps quoting the old quantile and nobody notices until the segment's late-rate alarms, if there is one.
- The censored-cancellation handling is dropped in a refactor and the model becomes optimistic again over the next retrain.
- A quantile loss is a deliberate bias: the model is wrong on purpose in one direction, which is harder to explain than "we predict the average".
- Per-segment evaluation multiplies the number of things to watch and requires segments with enough volume to estimate a tail quantile.
- Censored-data handling makes the training pipeline more complicated and easier to break silently on the next refactor.
- "RMSE improved, so the quotes are better." RMSE is dominated by the two-hour orders. The quotes for the typical order may have got worse while the tail got a little less wrong.
- "Predict the mean, it is the best guess." It minimises squared error. The customer does not experience squared error; they experience an early quote as a cancellation.
- "Regression is the easy task, there is no threshold." There is a choice of statistic, which is a threshold in disguise, and it is made by the loss function whether or not anyone chose it.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThat the loss decides which conditional statistic a regression model reports, and that the metric must match the cost of error, holds for linear models, trees and networks alike.
- TASK-SPECIFICThe asymmetry argument is specific to outputs consumed as commitments — delivery quotes, demand forecasts, capacity plans. For a regression whose output feeds a symmetric downstream calculation, such as a price elasticity estimate, the conditional mean is usually right and a quantile loss adds bias for nothing.
- SIMPLIFIEDAny durations and percentages mentioned here describe the shape of the argument; a real quote model would set its quantile from measured cancellation and demand curves, not from this lesson.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Testing & Reliability Engineering — asserting that the censored-order handling survives a refactor is a data-pipeline test, and a regression that quietly drops it is the kind of failure a pipeline test suite exists to catch.