Linear Regression
ŷ = w·x + b. A weight per feature, a bias, a loss that says which mistakes hurt — and a set of assumptions the weights only make sense under.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
A linear model gave a coefficient of 40 minutes per kilometre and an RMSE the team liked. What did the model actually assume, and what happens when the assumptions stop being true?
A delivery company wants to quote a delivery time at checkout. Right now dispatchers guess from experience and the quotes are wrong often enough that customers complain. "Give us a number we can show, and tell us what drives it, because ops will ask."
Fit ŷ = w·x + b by least squares on every completed delivery. Read the weights as "each kilometre adds w₁ minutes, each kilogram adds w₂ minutes". Report the RMSE on a held-out set and ship the equation, which is easy to serve because it is a dot product.
The customer-was-out deliveries dominate the squared loss. The fitted line is pulled towards a tail the checkout quote should never promise, so the typical delivery is over-quoted and the quote looks worse than the dispatcher's guess for most customers.
- The customer-was-out deliveries dominate the squared loss. The fitted line is pulled towards a tail the checkout quote should never promise, so the typical delivery is over-quoted and the quote looks worse than the dispatcher's guess for most customers.
- The "minutes per kilometre" coefficient was learned on a feature set where distance and depot are correlated — the rural depot has long routes. The weight on distance is partly the depot's effect in disguise, so ops' "what drives it" reading is wrong (Attribution Is Not Causality).
- Queue length at dispatch was reconstructed from the dispatch log. The migration changed how queue length was recorded, so the feature means one thing before the boundary and another after, and the model learned a blend of both.
- A new depot opens in a city with a distance profile outside the training range. The line extends without complaint; the quotes there are confidently wrong from day one (Residuals & Assumptions).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict the minutes between order confirmation and the courier marking the parcel delivered. The label is that elapsed time on completed deliveries, so cancelled and lost parcels are not in the training set at all (Survivorship Bias).
- The quote is shown to the customer and used by ops to schedule couriers, so the prediction is consumed as a number, not as a rank — the scale of the error matters, not just its order.
- One example is one completed delivery: distance in kilometres, parcel weight, hour of day, day of week, the courier's current queue length at dispatch, and the depot.
- Distance comes from the routing service as of dispatch. Queue length is reconstructed from the dispatch log, which was migrated between two systems eight months ago and disagrees with itself across the boundary.
- The label distribution is long-tailed: most deliveries take under an hour, a few take a day because the customer was out.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The model is ŷ = Σᵢ wᵢxᵢ + b. Each weight is the change in prediction per unit change in that feature, *holding the other features fixed*, which is a statement about the training data's joint distribution rather than about the world.
- Least squares picks the weights that minimise Σ(y − ŷ)² over the training set. Because the loss is squared, a residual of 600 minutes costs as much as 100 residuals of 60 minutes, which is why the tail owns the fit. The mean absolute error Σ|y − ŷ| weighs every minute of error the same and pulls the fit towards the median instead of the mean (MSE, RMSE and MAE).
- The closed-form and gradient-descent solutions find the same weights; the closed form (XᵀX)⁻¹Xᵀy fails or becomes unstable when columns are collinear, because XᵀX is then not invertible in any meaningful sense — the same weight can be split between two correlated features in infinitely many ways.
- The weights are only interpretable under the assumptions the fit makes: the relationship is roughly linear over the range, examples are independent, the residual spread is roughly constant, and no feature is a proxy for the label (Residuals & Assumptions).
The loss decides which mistakes count
Least squares is the default because it has a closed form and a clean statistical story, not because squared error is what a checkout page cares about. Written out, the two losses make the difference obvious: the squared loss multiplies a residual by itself, so the 600-minute delivery contributes 360,000 to the sum while a 6-minute miss contributes 36.
The absolute loss treats each minute of error equally. Its minimiser is the conditional median rather than the conditional mean, which is what "the typical customer gets an honest quote" means. Neither is right in general; the cost of the error decides.
1import numpy as np2 3def predict(X, w, b):4 return X @ w + b # y_hat = w·x + b5 6def mse(y, y_hat):7 r = y - y_hat8 return np.mean(r * r) # a 600-minute miss counts 10,000x a 6-minute miss9 10def mae(y, y_hat):11 return np.mean(np.abs(y - y_hat)) # every minute of error counts once12 13def mse_gradient(X, y, w, b):14 r = y - predict(X, w, b)15 return -2 * X.T @ r / len(y), -2 * r.mean() # dL/dw, dL/dbThe gradient is a weighted sum of the residuals. Whichever examples have the biggest residuals steer the weights, and under MSE the biggest residuals are the tail.
A coefficient is a statement about the training set
The weight on distance is the change in predicted minutes per kilometre *with the other features held at their training-set relationship to distance*. In this dataset long distances mostly come from the rural depot. Some of the depot's slowness is therefore absorbed by the distance weight, and the depot indicator gets the remainder.
Nothing is wrong with the fit. The prediction is as good as it can be. What is wrong is reading the weight as a property of kilometres. This matters twice: when ops explains the model to customers, and when the depot mix changes and the "kilometre" effect moves without the roads changing.
looks like A column in the dispatch table, populated for every delivery, with a large and stable coefficient. It is the routing service's estimated driving time for the assigned route.
why it leaks It does not leak the label directly, but it is written by the routing service *after* the courier is assigned and re-planned mid-route, so the stored value is the final route, computed with knowledge of the delays that happened. It is a collinear proxy for the answer.
fix Use the routing estimate as it existed at order confirmation, reconstructed with a point-in-time join, or drop it (Point-in-Time Correctness).
What the weights assume after they ship
A linear model has no non-linearity to hide behind, so its assumptions are unusually easy to state and unusually easy to break without noticing. The model will happily quote a delivery at a distance five times anything it saw, because a line extends. It will happily accept a feature in metres that it learned in kilometres, because a dot product does not check units.
These are not exotic failures; they are what happens on the first Tuesday after a depot opens or a routing service upgrades. The defence is to write the assumptions down with the artifact and check them at the boundary, which a linear model makes cheap.
Every feature reaches the model in the unit it was trained on and within (or near) the range the training set covered.
holds when The feature pipeline is versioned with the artifact and the served population looks like the training population — same depots, same distance profile, same parcel mix.
breaks when A new depot, a new service region, a unit change upstream, or a schema migration that redefines a column while keeping its name.
respond Do not retrain first. Find whether the input moved or the world moved. A unit change is a pipeline bug; a new depot is a training-range gap that needs new examples, not a new loss.
| Option | Quality | Latency | Cost | Interpretability | Data needed | Operational | Note |
|---|---|---|---|---|---|---|---|
| Dispatcher rule | Whatever the model must beat. Zero serving cost, fully explainable, and wrong in ways nobody has measured. | ||||||
| Linear regression | A dot product to serve. Coefficients readable with care; interactions have to be hand-built. | ||||||
| Gradient-boosted trees | Learns the rush-hour interaction itself. Explanations are approximations; the artifact needs a runtime. |
caveat The quality column is a guess until both are fitted on this data with this loss; the point of the matrix is the columns other than quality, which are what the linear model wins on and which the team will feel every day.
How to build it
Most important first.
- Choose the loss from the cost of the error, not from the default. A checkout quote that should be right for the typical customer wants MAE or a Huber loss; a capacity plan that must cover the tail wants MSE or a quantile loss (Choosing a Regression Metric).
- Establish the mean predictor and the dispatcher's rule as baselines and report the model against both, on the same held-out set (Majority Class and Mean Predictor, The Rule Baseline).
- Standardise features before reading coefficients, and read them as "effect per standard deviation in this dataset", never as causes (Bucketing & Normalisation).
- Record the training range of every feature with the artifact and refuse or flag predictions outside it — the line has no idea it is extrapolating.
- Plot residuals against the prediction before anyone sees the RMSE. The plot shows the tail, the heteroscedasticity and the missing non-linearity that the single number averages away.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- MAE on the held-out set, in minutes, alongside the same number for the mean predictor and the dispatcher rule. The gap is the value of the model; the MAE alone is not.
- The fraction of quotes within a tolerance the product owner names — "within 15 minutes" — because that is the promise the checkout page makes.
- RMSE looks relevant and is dominated by the customer-was-out tail; a model can improve RMSE by getting worse on the deliveries the quote is for.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Each feature arrives at serving time in the same unit and roughly the same range as in training; a prediction outside the training range of any feature is flagged rather than trusted.
- The relationship between features and delivery time is close enough to linear over the served range that the residuals show no systematic curve.
- The features that were correlated in training remain correlated in the same way in production; a coefficient learned against a proxy is only right while the proxy behaves.
- Offline: residual-vs-prediction plot on the held-out set; MAE against the mean predictor and the rule baseline; the coefficients on standardised features reviewed by someone from ops for sign and magnitude.
- Online: per-feature range checks at the serving boundary with a counter for out-of-range requests; the fraction of quotes within tolerance on completed deliveries as labels arrive.
- Over time: MAE by depot and by week, because a single aggregate MAE hides a new depot that is wrong every day (Evaluation Slices).
What can go wrong
- A feature gets rescaled upstream — distance switches from kilometres to metres — and the dot product silently produces quotes a thousand times too large (Train / Serve Skew).
- Two collinear features are supplied by different services; one goes stale and the weights, which were balanced against each other, now produce a large spurious shift.
- The model is retrained on a period with a new depot and the distance coefficient moves, so ops' explanation of "why the quote changed" changes with it and trust erodes.
- A linear model cannot represent the interaction "distance matters more during rush hour" without someone constructing the interaction feature by hand, and every hand-built feature is a serving-time transformation to reproduce.
- MAE and Huber losses give up the closed-form solution and need an iterative solver, which is trivial in cost but means the fit now has a convergence check.
- Flagging out-of-range requests means some customers get no quote; that is a product decision the model forces.
- "The coefficient on distance is 40, so a kilometre costs 40 minutes." It costs 40 minutes *in this dataset, holding the correlated features at their training relationship*. Change the depot mix and the number changes without the roads changing.
- "RMSE went down, so the quotes got better." RMSE is dominated by the tail; the typical quote may have got worse. Check MAE and the within-tolerance fraction.
- "It is only a linear model, so it cannot overfit." With enough one-hot depot columns and interaction features it has as many parameters as it needs to memorise; regularise it like anything else (Regularized Linear Models).
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe loss choice and the coefficient-reading caveats hold for any model that is a linear function of its inputs, including a neural network's final layer; only the assumptions about linearity are specific to using it as the whole model.
- DATA-SPECIFICOn a few thousand rows with a handful of features a linear model with hand-built interactions is often the strongest thing you can defend; on hundreds of features with interactions a tree ensemble usually wins and the coefficients stop being the point (Tree Ensembles: When and When Not).
- SIMPLIFIEDThe delivery example and any coefficient values in it are for the shape of the argument; the residual behaviour is what to look for, not the specific numbers.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Statistics — the assumptions under which least-squares coefficients have confidence intervals (independent errors, constant variance, correct functional form) are the same ones the residual plot checks; this domain uses the plot and links the theory.