Accuracy Under Imbalance
When the positive class is one in a thousand, predicting "no" every time is 99.9% accurate. Accuracy measures the majority class; use the metrics that read the positive row and the flagged column.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Why is a high accuracy on a rare-positive problem almost meaningless, and which numbers should replace it?
A fintech reports to its board that the new fraud model is "99.8% accurate". The board is pleased. The head of risk, who knows fraud is about a tenth of a percent of transactions, asks what the old rule-based system scored. Nobody has computed it.
Accuracy is the fraction of predictions that were right. 99.8% right is excellent by any everyday standard. Report it.
A model that predicts "not fraud" for every transaction is 99.9% accurate, because 99.9% of transactions are not fraud. The new model's 99.8% is *worse* than doing nothing, by accuracy — and accuracy cannot even see that, because it never asked about the positives.
- A model that predicts "not fraud" for every transaction is 99.9% accurate, because 99.9% of transactions are not fraud. The new model's 99.8% is *worse* than doing nothing, by accuracy — and accuracy cannot even see that, because it never asked about the positives.
- The number the board heard says nothing about the only cell that matters: how much fraud is caught. A model could catch none of it and beat the reported figure.
- The old rule-based system was never scored, so there is no baseline; the reported accuracy is a number floating free of any comparison (Majority Class and Mean Predictor).
- The team tunes for accuracy in the next iteration, which pushes the threshold up until almost nothing is flagged — that is where accuracy is maximised when positives are rare — and recall collapses while the headline number improves.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a transaction is fraudulent. Fraud is roughly one in a thousand transactions, so the negative class outnumbers the positive by a thousand to one.
- The decision is block-or-allow, and the point of the model is the positives — the thousand-to-one majority is the background the model must not drown in.
- One example is one transaction with cardholder aggregates, device and merchant features as of the transaction time.
- A month of data contains millions of transactions and a few thousand confirmed frauds; the confirmations arrive weeks later (Ground-Truth Delay).
- The validation split was stratified so the rare positives are represented, which is correct and does nothing about the metric problem (Stratified Split).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Accuracy = (TP + TN) / total. With a positive rate π, the TN cell alone can contribute up to (1 − π) of accuracy. At π = 0.001 the majority-class predictor scores 0.999 with TP = 0, and any model's accuracy lives in the third decimal place, where it is dominated by how many negatives it wrongly flags rather than by how many positives it catches (The Confusion Matrix).
- Accuracy weights every case equally, so it weights the two error types by their frequency: at a thousand to one, a false positive costs the metric as much as a false negative, but there are a thousand times more opportunities for the former. Maximising accuracy therefore means minimising false positives at almost any recall — the wrong direction for fraud.
- Metrics that read the positive row (recall) or the flagged column (precision) do not have a majority-class term and cannot be won by predicting the majority. Their baseline under the majority predictor is zero, which is honest (Precision, Recall & F1).
- Ranking metrics divide differently again: ROC AUC compares positives with negatives pairwise and is unaffected by the ratio, which makes it stable under imbalance and also blind to the precision the imbalance destroys; PR AUC follows the positive class and shows it (ROC AUC, PR AUC).
The number that measures the negatives
Set the prevalence at one in a thousand. A model that never fires has TP = 0, FN = every fraud, FP = 0, TN = everything else. Its accuracy is 99.9%. The reported model scored 99.8%, so by the board's metric it is worse than a model that does nothing, and the metric cannot even say so, because it never distinguishes the cells that matter from the one that does not.
This is not a subtle statistical point. Accuracy is a weighted average in which the majority class holds almost all the weight. When the class you care about is the minority, accuracy is a measurement of the class you do not care about.
The majority-class predictor. Compare any real model to this table before quoting accuracy: if the real model's accuracy is lower, it is because it flagged something, which is what it is for.
What to report instead
Recall and precision have no majority-class term. Under the do-nothing model both are zero, which is the correct description of a model that does nothing. Under a real model they say how much fraud was caught and how clean the blocked set was — the two things the head of risk asked about. With the prevalence stated and costs attached, expected cost puts the same information in currency, which is the form a board can act on.
Ranking metrics need the same care. ROC AUC is unmoved by the thousand-to-one ratio and can look comfortable while the flagged set is mostly legitimate; PR AUC tracks the positive class and drops when precision does. Under imbalance, PR AUC is the honest ranking summary.
"The fraud model is 99.8% accurate." No baseline, no prevalence, no statement about fraud caught.
"Fraud is 0.1% of transactions. Doing nothing is 99.9% accurate. The model catches most confirmed fraud at a precision the review team can work with, and its expected cost is below the rule-based system's on the same month."
The second slide is falsifiable and comparable; the first is a number that a do-nothing model would beat. The metrics that read the positive row and the flagged column are the only ones that move when fraud is caught.
1import numpy as np2 3def accuracy(y, y_hat):4 return np.mean(y == y_hat)5 6def majority_baseline(y):7 majority = int(np.mean(y) >= 0.5) # 0 when positives are rare8 return np.full_like(y, majority)9 10def report(y, p, threshold):11 y_hat = (p >= threshold).astype(int)12 base = majority_baseline(y)13 tp = np.sum((y_hat == 1) & (y == 1)); fp = np.sum((y_hat == 1) & (y == 0)); fn = np.sum((y_hat == 0) & (y == 1))14 return {15 "prevalence": float(np.mean(y)),16 "accuracy_majority": accuracy(y, base), # print this next to the model's, every time17 "accuracy_model": accuracy(y, y_hat),18 "recall": tp / (tp + fn) if tp + fn else 0.0,19 "precision": tp / (tp + fp) if tp + fp else 0.0,20 }The dictionary is the whole lesson: prevalence first, the baseline's accuracy before the model's, and then the two numbers that have no majority-class term.
What must hold for the report to stay honest
The prevalence is the parameter that makes every number in the report mean what it means. A precision quoted at one prevalence is a different number at another; a baseline accuracy is only the baseline for the period it was computed on. Fraud rates move — a new attack, a new merchant category — and every metric on the slide moves with them without the model changing.
So the prevalence is monitored, the baseline is recomputed on every period, and any report that quotes a metric without the prevalence next to it is incomplete by construction.
Every reported precision, recall or accuracy is accompanied by the positive rate on the period it was computed on, and that rate is still the served rate.
holds when Confirmed-fraud rate is monitored monthly and the report is regenerated per period with the baseline re-scored.
breaks when A fraud wave or a merchant-mix change moves the prevalence; the report template keeps a stale baseline; a resampled training set changes the score scale.
respond Re-score the baseline and re-derive the threshold from costs at the new prevalence; a change in fraud rate is a change in the problem, not evidence about the model.
How to build it
Most important first.
- Report the majority-class predictor's accuracy next to the model's, always. If the two are within rounding, the metric has said nothing (Baselines Are Mandatory).
- Report precision and recall at the deployed threshold, and PR AUC for the ranking, with the prevalence stated; these are the numbers that move when the positives are caught.
- Compute the confusion matrix and attach costs; expected cost is the metric the board should hear, in currency, against the rule-based baseline (Threshold Selection).
- Do not "fix" the imbalance by resampling for the sake of the metric; the imbalance is the problem's shape, and resampling distorts the probabilities (Sigmoid & Probability, Class Imbalance).
- Show the effect in the Threshold Explorer at
/ml/threshold: set prevalence to a tenth of a percent and watch accuracy sit near one while recall goes anywhere.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Recall at the deployed threshold — how much fraud is caught — and precision — how much of what is blocked was fraud — with the prevalence stated. These are the numbers that answer the head of risk's question.
- Expected cost against the rule-based baseline on the same period.
- Accuracy is the number that was reported and it measures the negatives.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The prevalence quoted next to every metric is the current prevalence; a monitor on confirmed-fraud rate per month checks it, and any metric reported without it is incomplete.
- The rule-based baseline is re-scored on every evaluation period the model is scored on, so the comparison stays on the same data.
- The metrics reported to decision-makers are the positive-row and flagged-column ones; a report that reverts to accuracy has silently changed what is being claimed.
- Offline: accuracy of the majority predictor and of the rule baseline on the validation period, next to the model's precision, recall and PR AUC; if accuracy is reported at all, it is reported with those.
- Online: blocked-transaction volume immediately; precision and recall as confirmations mature; prevalence per month.
- Over time: expected cost per month against the rule baseline, with prevalence, so a change in fraud rate is not mistaken for a change in the model.
What can go wrong
- The majority baseline is reported once, the prevalence changes, and the baseline is no longer the number quoted next to the model.
- Precision is reported on a tiny flagged set and swings wildly week to week; nobody puts an interval on it (Metric Uncertainty).
- The team switches to ROC AUC to escape accuracy and reports a comfortable figure that hides a precision the review team cannot work with (ROC AUC).
- Precision and recall on a rare class are noisy — a few thousand positives a month — and honest reporting includes an interval that will look uncomfortably wide.
- Reporting in expected cost needs a cost per miss and per block that finance must own, which takes longer than reporting accuracy.
- Explaining to a board why 99.8% is not good news costs credibility once; not explaining it costs more later.
- "99% accuracy means the model is good." At a one-percent positive rate, 99% accuracy is what you get from a model that never fires. Ask what the majority predictor scores and what the recall is.
- "Accuracy went up in the new release." When positives are rare, accuracy goes up when fewer things are flagged. Check whether recall went down.
- "We stratified the split, so imbalance is handled." Stratification puts positives in every fold; it does not change what accuracy measures.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe arithmetic holds for any classifier at any prevalence; the effect becomes serious somewhere below a positive rate of ten percent and dominant below one percent.
- DATA-SPECIFICOn a balanced problem — half positives — accuracy is a reasonable summary and the majority baseline is fifty percent; the lesson is about rare-positive problems, where the majority baseline sits next to one and swallows the metric.
- SIMULATEDThe 99.8%, 99.9% and 0.1% figures are for the shape of the argument, produced by the module's threshold model at a set prevalence rather than measured on a real transaction stream.
Where the depth lives
This domain teaches the model and hands the rest off by name.