Logistic Regression
Linear score → sigmoid → probability, trained by gradient descent on the log loss. The threshold that turns the probability into a decision is a different step, owned by someone else.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does a logistic regression actually compute, how is it trained, and why is "the model said yes" always two decisions dressed as one?
A subscription product wants to know which customers are likely to cancel next month so the retention team can call them. "We have about two hundred people we can call a week. Tell us who, and tell us how sure you are, because a call to a happy customer annoys them."
Fit a logistic regression, which is "linear regression for yes/no". Predict 1 when the output is above 0.5 and 0 otherwise. Report accuracy. Hand the retention team the list of 1s.
At a 0.5 cut almost nobody is predicted to cancel — with a 3% base rate the model rarely reaches 0.5 — so the call list is nearly empty and the team calls no one. Accuracy is excellent (Accuracy Under Imbalance).
- At a 0.5 cut almost nobody is predicted to cancel — with a 3% base rate the model rarely reaches 0.5 — so the call list is nearly empty and the team calls no one. Accuracy is excellent (Accuracy Under Imbalance).
- When the list is instead "top 200 by probability", it is a threshold chosen by capacity, and its precision depends on where in the score distribution the 200th person sits. Nobody measured that, so the team has no idea what fraction of their calls are wasted.
- The probabilities were trained on subscriber-months with a 3% base rate; the retention team reads 0.4 as "40% likely to cancel". Whether that is true depends on calibration, which was never checked (Calibration).
- The model was trained with class weights to "fix the imbalance" and the scores all moved up; the 0.5 cut now flags a third of customers. Same model family, different intercept, and the threshold that made sense last week makes none now (Sigmoid & Probability).
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a subscriber cancels within the 30 days after the prediction date. The label is the cancellation event in that window, built from the billing system, so a payment failure that is later recovered must be decided on: cancellation or not (Label Construction).
- The consumer is a weekly call list of fixed size. The model produces a probability; the list is the top of the ranking cut at capacity, which is a threshold nobody chose explicitly.
- One example is one subscriber on one prediction date: tenure, plan, logins in the last 7 and 30 days, support tickets, last payment status, and a discount flag.
- The training set is built from monthly snapshots over the last year, so the same subscriber appears twelve times with a label each time (Entity Leakage).
- Cancellations are around 3% of subscriber-months, so the positive class is rare and the intercept will be strongly negative.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The model computes a linear score z = w·x + b, exactly as in regression. The sigmoid σ(z) = 1 / (1 + e^(−z)) maps the score onto (0, 1). The output is a probability *in the sense that the training objective rewards calibrated probabilities on the training distribution* — not by construction.
- Training minimises the log loss −[y·log p + (1−y)·log(1−p)] averaged over examples. Its gradient with respect to the score is simply (p − y): the same form as the squared-error gradient, which is why one gradient-descent loop trains both models and why the Gradient Descent Visualizer applies here.
- The log loss is convex in w, so gradient descent converges to the one global minimum for any learning rate small enough. That is the practical meaning of "logistic regression is well behaved": no initialisation luck, no local minima, a reproducible fit.
- The decision boundary p = 0.5 is the hyperplane z = 0. Any other threshold t is the hyperplane z = log(t / (1 − t)) — a parallel plane, shifted. Moving the threshold moves the plane; it does not change the model.
Two steps that look like one
The model service computes z = w·x + b and returns σ(z). That is the whole model. Somewhere else — in the same function, if nobody separated them — a comparison against 0.5, or a sort-and-take-200, turns that number into "call this person". The first step is learned from data. The second is a business decision about costs and capacity that happens to be written as a constant.
Drawing the two steps separately is the single most useful habit in this module. It makes the threshold visible, gives it an owner, and stops a model retrain from silently changing a business policy.
The loss and its gradient
The log loss penalises confident mistakes without bound: predicting p = 0.01 for a customer who cancels costs log(100). Its gradient with respect to the score is p − y, so each example pushes the score in proportion to how wrong the probability was. Because the loss is convex, gradient descent with a sensible learning rate reaches the same weights every run.
Written out, the training loop is a few lines, and everything a framework adds — regularisation, solvers, class weights — is a modification of these lines that you should be able to name.
On a held-out month, precision at the top-200 cut looked strong and log loss beat the base-rate predictor comfortably.
The retention team reports most calls going to customers who were never going to leave, and a cluster of cancellations from customers the list never contained.
- 1The validation split was random over subscriber-months, so the same customer's snapshots appeared in both train and validation and the offline precision was inflated (Entity Leakage).
- 2The score distribution in production is shifted from validation because the plan mix changed after a pricing update; the 200th score now sits lower and precision fell with it.
- 3The called customers who stayed were recorded as negatives with high-risk features, and the first retrain learned from them.
1import numpy as np2 3def sigmoid(z):4 return 1.0 / (1.0 + np.exp(-z))5 6def log_loss(y, p, eps=1e-12):7 p = np.clip(p, eps, 1 - eps)8 return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))9 10def step(X, y, w, b, lr):11 p = sigmoid(X @ w + b)12 err = p - y # dL/dz — the whole gradient story13 w -= lr * (X.T @ err) / len(y)14 b -= lr * err.mean()15 return w, b16 17# the decision is not in this file:18# decide = lambda p, t: p >= t # t is a business parameter, see /ml/thresholdEvery example's pull on the weights is (p − y)·x. A confident wrong answer pulls hard; a correct one barely pulls. Class weighting multiplies err per class, which mostly moves b — see Sigmoid & Probability.
What must stay true once the list is live
The model's weights encode the training-time relationship between features and cancellation, and the threshold encodes the training-time score distribution. Both can move independently of each other and of the code.
The one a linear model makes easy to check is the score distribution: a histogram of weekly scores is cheap, needs no labels, and moves the moment the served population changes. It is the earliest signal that the capacity cut no longer means what it meant.
The 200th-highest weekly score sits at the same probability, and therefore the same precision, as it did on validation.
holds when The served population and the base rate match validation; the model version and the decision config were tuned together.
breaks when The subscriber mix changes, a pricing change shifts behaviour, a retrain (or a class-weight change) shifts the intercept, or the capacity changes without re-deriving the threshold.
respond If only the score scale moved, re-derive the cut from the new distribution and check calibration; if the features moved, that is drift to diagnose before any retrain (Drift Is Not Failure).
How to build it
Most important first.
- Separate the two steps in code and in ownership: the model service returns a probability and a model version; the decision service applies a threshold or a capacity cut that the business owns (Prediction vs Decision, Thresholding).
- Choose the operating point from the confusion matrix at the capacity the team actually has — 200 calls — and report precision and recall *there*, not at 0.5 (Threshold Selection).
- Split by subscriber and by time so that the same customer's twelve snapshots do not straddle train and validation (Group Split, Time-Based Split).
- Standardise features before fitting; regularise; check calibration on a held-out period before anyone reads a probability as a percentage (Regularized Linear Models).
- Store the score, not just the decision, for every prediction, because the threshold will change and the analysis will need the scores (Prediction Logging).
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Precision among the 200 called per week, once the 30-day labels arrive — this is the number the retention team feels as "wasted calls".
- Recall at that capacity: what fraction of next month's cancellations were on the list. This is the number the business feels as "we never called them".
- Log loss on a held-out later month, as the training-objective check, and a reliability curve if the probability is shown to anyone.
- Accuracy is the number that looks relevant and measures the 97% of customers who stayed and were not called.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The score distribution in production matches the validation score distribution closely enough that the capacity cut lands at the same precision; a monitor on the weekly score histogram would show it moving.
- The base rate of cancellation on the served population is close to the training base rate, or the probabilities have been corrected for the difference; a drift in observed cancellations per month tests this.
- The decision step reads the model version it was tuned against; a model redeploy without a threshold review is a silent change to the call list.
- Offline: precision and recall at the top-200 cut on a held-out later month; log loss against the majority-class baseline; a reliability curve (Majority Class and Mean Predictor).
- Online: weekly histogram of scores on the served population against validation; the position of the 200th score; precision on the called list as labels arrive.
- Over time: the observed cancellation rate against the mean predicted probability, monthly; a widening gap is either drift or a feedback loop from the calls themselves (Ground-Truth Delay).
What can go wrong
- The call itself changes the outcome — a called customer who stays becomes a training-set negative with high-risk features — and the next model learns that high-risk features mean staying (Feedback Loops).
- The score distribution drifts as the plan mix changes and the top-200 cut now sits at a different probability; precision moves and nobody notices because the list is still 200 long.
- Someone "improves" the model with class weights and the serving-side threshold, tuned to the old score scale, is now wrong by a constant.
- A separate decision service is a second deployable with its own config and its own way to be wrong, in exchange for the threshold being changeable without a model release.
- Logistic regression cannot learn "recent login drop matters only for annual plans" without a hand-built interaction; a tree would find it, at the cost of readable weights.
- Storing every score is a privacy and retention question; not storing it means the threshold can never be revisited on past data.
- "Logistic regression outputs probabilities." It outputs a sigmoid of a linear score, trained by an objective that rewards calibration on the training distribution. Whether 0.4 means 40% is an empirical question answered by a reliability curve.
- "The model predicted 1 for these customers." The model predicted 0.61; something else predicted 1. Ask who chose the threshold and against what cost.
- "Accuracy is high, so the classifier works." At a 3% base rate the majority-class predictor has the same accuracy. Report precision and recall at the operating point.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe score → sigmoid → probability → threshold → decision chain is how every binary classifier is consumed, whatever produces the score; the convexity and the readable weights are what logistic regression specifically adds.
- MODEL-SPECIFICLogistic regression's log-loss training tends to produce reasonably calibrated scores on the training distribution; boosted trees and SVMs produce scores whose calibration is much worse by default and must be checked or corrected before being read as probabilities.
- SIMPLIFIEDThe 3% base rate, the 200-call capacity and any precision figures are for the shape of the argument; the mechanism of the capacity cut being an implicit threshold is what transfers.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Optimisation theory — convexity of the log loss is what guarantees gradient descent finds the global minimum; the Gradient Descent Visualizer lets you push the learning rate until it diverges on exactly this surface.