Classification Metrics

The confusion matrix and everything derived from it — precision, recall, F1, ROC AUC, PR AUC, calibration — each with its business reading and the case where it misleads.

The Confusion Matrix
▶ lab

Four counts — TP, FN, FP, TN — and four business outcomes with four different prices. Every classification metric is a way of reading this table; read the table first.

Q · Before any metric, what did the classifier actually do to each of the four kinds of case, and what does each cell cost the business?
Precision, Recall & F1
▶ lab

Precision reads the flagged column: how many alarms were real. Recall reads the positive row: how many real cases were caught. F1 averages them as if the two mistakes cost the same, which they never do.

Q · Which of precision, recall and F1 corresponds to the complaint the business is making, and what does moving the threshold do to each?
Threshold Selection
▶ lab

Flag when P × cost_FN exceeds (1 − P) × cost_FP. The threshold falls out of the costs and the calibrated probability; 0.5 is what you get when the costs are equal and nobody checked.

Q · Why not always 0.5 — and given the costs, the prevalence and a calibrated probability, where should the threshold sit?
Accuracy Under Imbalance
▶ lab

When the positive class is one in a thousand, predicting "no" every time is 99.9% accurate. Accuracy measures the majority class; use the metrics that read the positive row and the flagged column.

Q · Why is a high accuracy on a rare-positive problem almost meaningless, and which numbers should replace it?
ROC AUC
▶ lab

The probability that a random positive scores above a random negative. A pure ranking metric — invariant to threshold, to prevalence, and therefore blind to the precision that prevalence destroys.

Q · What does ROC AUC actually measure, why does it look fine on rare-positive problems where the flagged set is mostly wrong, and when is it the right number?
PR AUC
▶ lab

Precision against recall across every threshold, and the area under it. It follows the positive class, so it falls when the flagged set fills with negatives — which is exactly what ROC AUC cannot see.

Q · What does the precision–recall curve show that the ROC curve hides, and why is its area the honest ranking summary when positives are rare?
Calibration
▶ lab

Does 0.8 mean 80%? The reliability curve answers it bin by bin. Calibration matters when the probability is multiplied by a value; it matters not at all for a pure ranking.

Q · When does it matter that the model's probabilities are true frequencies, how do you check, how do you fix it, and when can you ignore it?