The Confusion Matrix
Four counts — TP, FN, FP, TN — and four business outcomes with four different prices. Every classification metric is a way of reading this table; read the table first.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Before any metric, what did the classifier actually do to each of the four kinds of case, and what does each cell cost the business?
A hospital system flags patients at admission for a sepsis-prevention protocol. Nursing staff say the alert "fires all the time"; the quality team says several sepsis cases last quarter were never flagged. Both are looking at the same model and asking whether it is any good.
Report one number — accuracy — and call the model good if it is high. When staff and quality disagree, average their complaints and adjust the sensitivity a little.
Accuracy is dominated by the true negatives, which are the patients who were never at risk. It says nothing about either complaint, because both complaints are about the other three cells.
- Accuracy is dominated by the true negatives, which are the patients who were never at risk. It says nothing about either complaint, because both complaints are about the other three cells.
- The nurses' complaint is the FP cell; the quality team's complaint is the FN cell. Lowering the threshold shrinks one and grows the other. Without the four counts in front of both teams, every adjustment looks like a fix to one and a regression to the other.
- Nobody put a price on the cells. A false alarm costs minutes and an antibiotic course; a miss costs hours of untreated sepsis. Treating them as equal is a clinical decision made by omission.
- The label is contaminated by the alert: flagged-and-treated patients often never meet the code, so they are counted as false positives when some were true positives that the protocol prevented. The FP cell is inflated by the model's own success.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an admitted patient will develop sepsis within 48 hours. The label is a clinical diagnosis code recorded later, which is itself applied inconsistently across wards (Label Quality).
- The decision is binary — start the protocol or not — and the two mistakes land on different people: a false alarm costs nursing time and antibiotic exposure; a miss costs a patient hours of delay.
- One example is one admission: vitals at intake, lab results within the first six hours, age, admission source, and comorbidity flags.
- Sepsis occurs in a small minority of admissions, so the table is dominated by true negatives whatever the model does.
- Alerts that were acted on change the outcome — a patient treated early may never meet the diagnosis code — so the label for flagged patients is partly a consequence of the flag (Feedback Loops).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- For a binary decision at a fixed threshold, each case lands in one cell. Rows are the actual class, columns the predicted class: TP (actual +, predicted +), FN (actual +, predicted −), FP (actual −, predicted +), TN (actual −, predicted −). The row sums are the actual class counts; the column sums are how many the model flagged and how many it cleared.
- Every classification metric is a ratio of these cells. Accuracy = (TP + TN) / total. Precision = TP / (TP + FP) reads down the predicted-positive column. Recall = TP / (TP + FN) reads across the actual-positive row. Specificity = TN / (TN + FP) reads across the actual-negative row. Each answers a different question and no single one answers all of them (Precision, Recall & F1).
- The matrix is a function of the threshold. Lower it and cases migrate from the predicted-negative column to the predicted-positive one: FN becomes TP (good), TN becomes FP (bad). The matrix at one threshold is one row of a threshold sweep (Threshold Selection).
- Attaching a cost to each cell turns the matrix into a number the business recognises: expected cost = FP·cost_FP + FN·cost_FN (TP and TN usually cost nothing or save something). That number, not accuracy, is what the two teams are arguing about.
Four cells, four prices
Lay the table out with actual class as rows and predicted class as columns. The top-left is the model doing its job; the bottom-right is the model correctly staying quiet — the cell that makes accuracy look good. The other two are the two ways to be wrong, and they are not the same mistake. Every metric in this module is a way of reading this table, and reading the table directly is the one step nobody should skip.
Write the price next to each cell before computing anything. In the sepsis case the FP price is nursing time and an antibiotic course; the FN price is hours of untreated sepsis. Once those are on the table, "is the model any good" becomes a sum, and "which way should the threshold move" becomes arithmetic.
Illustrative counts. Accuracy here is dominated by the bottom-right cell and would barely change if the top row were halved; the two complaints are the 690 and the 31, and they move in opposite directions when the threshold moves.
The matrix is one slice of a sweep
At a lower threshold some of the 31 misses would have been flagged — and so would many more of the 9,195 quiet patients. The matrix at any threshold is one frame of a film; the threshold sweep is the film. The cells are computed from the score list and the labels, and the code is short enough that there is no excuse for reading someone else's summary of it.
Reading it yourself also surfaces the label contamination: if flagged-and-treated patients rarely meet the diagnosis code, the FP cell contains the model's successes, and the sweep is built on a label the model is bending.
The cost of a false alarm and the cost of a miss, as recorded with the model, are the costs the hospital currently bears.
holds when Protocol, staffing and the available tests are unchanged since the costs were agreed.
breaks when A cheaper confirmatory test arrives, staffing changes make an alert more or less expensive, or a policy change raises the penalty for a missed case.
respond Re-derive the threshold from the new costs and the current score distribution; the model itself has not changed and does not need retraining.
1import numpy as np2 3def confusion(y, p, threshold):4 flagged = p >= threshold5 tp = int(np.sum(flagged & (y == 1)))6 fn = int(np.sum(~flagged & (y == 1)))7 fp = int(np.sum(flagged & (y == 0)))8 tn = int(np.sum(~flagged & (y == 0)))9 return tp, fn, fp, tn10 11def expected_cost(cells, cost_fp, cost_fn):12 tp, fn, fp, tn = cells13 return fp * cost_fp + fn * cost_fn # TP and TN cost nothing here14 15# a sweep is just this at many thresholds:16# [(t, expected_cost(confusion(y, p, t), c_fp, c_fn)) for t in np.linspace(0, 1, 101)]Accuracy would be (tp + tn) / len(y). Notice that tn is the only cell that appears nowhere in the cost — and it is the cell accuracy is mostly made of.
What the label does not know
The quality team counts a miss when a patient with the diagnosis code was not flagged. The nurses count a false alarm when a flagged patient never met the code. But a flagged patient who was treated early may never meet the code *because* they were flagged. The matrix is filled from a label that the model's own output changes, and the FP cell inflates as the protocol works.
This is not a metric problem with a metric fix. It is a labelling decision — exclude treated patients, treat their label as censored, or estimate the counterfactual — and it has to be made before the matrix means anything (Label Leakage, Feedback Loops).
On the pre-rollout held-out period the matrix showed a precision the clinicians accepted and a recall that beat the previous manual screen.
Measured precision has fallen every quarter since rollout while alert volume is flat; the quality team's FN count is unchanged.
- 1Flagged-and-treated patients increasingly avoid the diagnosis code, so true positives are being counted as false positives — the model's successes are inflating the FP cell.
- 2The admission mix shifted towards a lower-risk source, lowering the base rate; the same threshold now flags proportionally more negatives.
- 3Coding practice changed on two wards, so the label itself moved independently of the patients.
How to build it
Most important first.
- Report the matrix before any metric, at the deployed threshold, on a held-out later period. Put the four counts in front of both teams with the costs written next to the cells.
- Get the costs from the people who bear them — nursing for FP, the quality team and clinicians for FN — and record them with the model, because the threshold is derived from them (Decision Before Model).
- Slice the matrix by ward and by admission source; an aggregate matrix can hide a ward where the model never fires (Evaluation Slices).
- Decide how to label flagged-and-treated patients before evaluation; either exclude them or treat the label as censored, and say so in the report (Label Construction).
- Walk the threshold in the Threshold Explorer at
/ml/thresholdwith the costs entered, so the trade-off is a curve everyone has seen rather than a constant one person set.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The four counts at the deployed threshold, per quarter and per ward, with the expected cost computed from the agreed prices. This is the number that maps to "is the model any good".
- Recall on confirmed sepsis cases — the quality team's number — and precision on alerts — the nurses' number — reported together, never one without the other.
- Accuracy looks relevant and is almost entirely the TN cell.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The costs attached to the FP and FN cells are still the clinical and operational costs; a scheduled review with the owners of each cell checks this.
- The label used to fill the actual-class rows is not systematically changed by the model's own alerts, or the contamination is accounted for; a comparison of code rates between flagged-treated and flagged-untreated patients would show it.
- The threshold that produced the matrix is the one deployed; the matrix is recomputed after every model or threshold change.
- Offline: the matrix at the deployed threshold on a later period, per ward, with expected cost; the same matrix for the pre-model protocol as a baseline (The Rule Baseline).
- Online: alert volume per ward (the predicted-positive column, available immediately) and, as diagnosis codes arrive, the FP and FN cells (Ground-Truth Delay).
- Over time: the expected cost per quarter, with the costs re-confirmed; a matrix whose FN cell grows while the FP cell is flat is a recall problem no accuracy figure would show.
What can go wrong
- The costs are agreed once and never revisited; a new rapid test halves the cost of a false alarm and the threshold, derived from the old cost, is now too conservative.
- The matrix is reported on all admissions but the alert only runs in some wards; the TN cell includes patients the model never saw.
- Feedback contamination grows as the protocol succeeds, the measured FP cell inflates, and someone raises the threshold to "fix precision", which costs real patients.
- Putting a price on a missed sepsis case is uncomfortable and someone has to do it; the alternative is a price set implicitly by whoever tuned the threshold.
- Per-ward matrices multiply the reporting and the small wards never have enough cases to say anything.
- Handling label contamination honestly means excluding or censoring the very cases where the model helped most, which makes the model look worse on paper.
- "Accuracy is high, so the alert is working." Accuracy is the TN cell divided by everything. The two complaints are about the other three cells, and accuracy cannot see them.
- "The nurses say too many alerts; the quality team says too few. They cancel out." They are the FP and FN cells, and the threshold moves them in opposite directions. Only the costs can say which way to move it.
- "Precision dropped after the protocol rolled out, so the model got worse." Treated patients stop meeting the diagnosis code and are counted as false positives. The model may have got better.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe four-cell decomposition and the cost reading apply to any binary decision derived from any model; multi-class problems generalise to a k×k table with a cost per off-diagonal cell.
- DOMAIN-SPECIFICIn clinical settings the FN cost is typically far above the FP cost and thresholds sit low; in content moderation or ad fraud the two are closer and the argument is about capacity. The same matrix, read with different prices, gives a different threshold.
- SIMULATEDThe counts in the matrix below are for the shape of the argument, not a measurement on any clinical dataset; the cost reading is the lesson.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Clinical decision analysis — the cost-per-cell reading is decision theory; this domain supplies the counts and links the discipline that decides what a missed diagnosis is worth.