Precision, Recall & F1
Precision reads the flagged column: how many alarms were real. Recall reads the positive row: how many real cases were caught. F1 averages them as if the two mistakes cost the same, which they never do.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Which of precision, recall and F1 corresponds to the complaint the business is making, and what does moving the threshold do to each?
A payments company runs a chargeback model that sends transactions to a manual review team. Finance says "we are losing too much to fraud we never reviewed"; the review team says "most of what you send us is fine". The model owner reports F1 and says it went up last release.
Report F1 as the one number that balances precision and recall. If F1 went up, the model is better for both finance and the review team. Tune the threshold to maximise F1.
F1 went up because precision rose while recall fell slightly. Finance's complaint is recall; their loss grew. The single number hid a move in the direction the business cares about least.
- F1 went up because precision rose while recall fell slightly. Finance's complaint is recall; their loss grew. The single number hid a move in the direction the business cares about least.
- The threshold that maximises F1 weights a false positive and a false negative equally. A missed fraud costs the full amount; a wasted review costs minutes. The F1-optimal threshold is far too conservative for finance and nobody chose that weighting on purpose (Threshold Selection).
- Precision was computed on the reviewed population — the transactions a previous threshold selected. At a new threshold the flagged population is different and the estimated precision is about the wrong set.
- The review team's capacity is a hard cap. The metric the team actually experiences is precision at the capacity cut, and the threshold that maximises F1 has no relationship to that cut.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a card transaction will be charged back as fraud within 90 days. The label arrives up to three months late and only for transactions that were approved (Ground-Truth Delay).
- The decision is review-or-approve, and each cell has a price finance can name: a reviewed-good transaction costs an analyst's minutes and a delayed customer; a missed fraud costs the transaction amount plus a fee.
- One example is one approved transaction with cardholder aggregates, merchant features and device signals as of the transaction time.
- Fraud is well under one percent of transactions; the review team can handle a fixed number of cases a day.
- Declined transactions have no label — nobody knows whether they would have been fraud — so the training population is what a previous policy approved (Selection Bias).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- Precision = TP / (TP + FP): of everything flagged, the fraction that was real. It is the review team's experience — the share of their queue that was worth the time. It goes up as the threshold rises, because the model flags only its most confident cases.
- Recall = TP / (TP + FN): of everything real, the fraction that was flagged. It is finance's experience — the share of fraud the process caught. It goes up as the threshold falls, because more of the positive row crosses into the flagged column.
- The two move in opposite directions as the threshold moves, because the threshold slides a line through two overlapping score distributions: lowering it captures more of the positive distribution and more of the negative one at the same time. There is no threshold that improves both unless the model itself changes (The Confusion Matrix).
- F1 = 2·P·R / (P + R), the harmonic mean, which punishes whichever is lower. It embeds the choice that a false positive and a false negative are equally bad; Fβ makes that weighting explicit with β² as the ratio of recall's importance to precision's. F1 is Fβ with β = 1, and β = 1 is not a fact about any business.
Two questions, two directions
Precision answers "of the alarms, how many were real" and reads down the flagged column. Recall answers "of the real cases, how many did we catch" and reads across the positive row. The review team lives in the column; finance lives in the row. When the two teams disagree, they are reading different lines of the same table, and both are right.
Move the threshold down and cases migrate from the not-flagged column into the flagged one. Some of them are fraud — recall rises — and more of them are not — precision falls. The two cannot both improve from a threshold change. Only a different model, with less overlap between its two score distributions, moves both.
Illustrative counts at a mid threshold. Lowering it would move some of the 190 into TP and a larger number of the 98,260 into FP: recall up, precision down, the same model.
F1 is a cost table in disguise
The harmonic mean punishes the smaller of precision and recall, which makes F1 a reasonable single number when a false positive and a false negative are roughly equally bad and nobody can say more. In the chargeback case someone *can* say more: finance has the loss per missed fraud and the review team has the cost per case. Once those exist, F1 is a worse version of expected cost with the prices overwritten by 1 and 1.
Fβ makes the hidden weighting visible — β² is how many times more recall matters than precision — and the code below shows how little separates "maximise F1" from "minimise cost with equal prices". The difference is only that one of them admits what it assumed.
The ratio of a missed fraud's cost to a wasted review's cost, as the threshold encodes it, is still the business's ratio.
holds when Average transaction values, scheme fees and analyst costs are stable, and the merchant's chargeback ratio is comfortably below the scheme's penalty threshold.
breaks when Ticket sizes shift, a scheme changes its fee schedule, the review team is halved, or the merchant nears the penalty threshold, where each additional miss costs far more than the transaction.
respond Re-derive the threshold from the new costs; the precision–recall curve is unchanged and the model does not need retraining.
1def precision(tp, fp):2 return tp / (tp + fp) if tp + fp else 0.03 4def recall(tp, fn):5 return tp / (tp + fn) if tp + fn else 0.06 7def f_beta(tp, fp, fn, beta=1.0):8 p, r = precision(tp, fp), recall(tp, fn)9 if p + r == 0:10 return 0.011 b2 = beta * beta # beta² = how much more recall matters than precision12 return (1 + b2) * p * r / (b2 * p + r)13 14def cost(fp, fn, cost_fp, cost_fn):15 return fp * cost_fp + fn * cost_fn # the number finance actually reads16 17# maximising f_beta(beta=1) and minimising cost(cost_fp == cost_fn) pick similar thresholds;18# with cost_fn = 40 * cost_fp they do not, and only one of them was a decisionβ is a business parameter pretending to be a statistical one. If nobody can say what β should be, they usually can say what a miss costs relative to a review, which is the same question with better units.
The number the release note hid
The release reported F1 up. Pulled apart, precision rose and recall fell, so the review team was happier and finance lost more money. A pair of numbers would have said this; the single number was constructed to average it away. This is the general pattern: any scalar that combines the two mistakes has a weighting inside it, and if the weighting was not chosen, it was inherited.
The offline/online gap here is not skew or drift. It is that the offline number answered a question nobody asked, and the online number — finance's loss — answered the one that mattered.
F1 improved on the validation period; the release note quoted it as the headline.
Fraud losses rose the following quarter as chargebacks matured; the review queue was shorter and cleaner.
- 1Precision rose and recall fell; the harmonic mean rose because precision was the smaller of the two and moved most. The business cost is dominated by recall.
- 2The validation recall was measured before the 90-day window matured, so the fall in recall was understated offline.
- 3The threshold was re-tuned to maximise F1 on the new model, which moved it up; part of the recall loss was the threshold, not the model.
How to build it
Most important first.
- Report precision and recall as a pair, at the deployed threshold and at the capacity cut, never F1 alone. Name which complaint each one answers.
- Replace F1 with expected cost when the costs are known — finance can name them here — and derive the threshold from that (Threshold Selection). Where they genuinely cannot be named, use Fβ with a β someone signed off.
- Report precision-at-capacity: sort by score, take the top N the team can review, and measure precision on those N. That is the review team's number.
- Evaluate on a later period, with the 90-day label window fully matured, on the approved population — and say that declined transactions are outside the measured population (Time-Based Split).
- Slide the threshold in the Threshold Explorer at
/ml/thresholdwith the costs entered; the curve makes the opposite motion of P and R obvious in a way a table does not.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- Recall on matured chargebacks at the deployed threshold — finance's number. Precision on reviewed transactions at the same threshold — the review team's number. Together, always.
- Expected cost at the threshold, using finance's cost per missed fraud and the team's cost per review, which is the number that decides.
- F1 looks relevant and is expected cost with the costs forced to be equal.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The relative cost of a missed fraud to a wasted review, as encoded in the threshold, is still finance's and the review team's cost; a periodic review with both records any change.
- The base rate of fraud and the score distribution stay close to the evaluation period's, so precision and recall at the fixed threshold stay where they were measured; a weekly served-score histogram checks this.
- The label window has matured before recall is quoted; a recall figure on transactions less than 90 days old is provisional and labelled as such.
- Offline: precision and recall at the deployed threshold and at the capacity cut, on a later period with matured labels; the same pair for the previous model on the same period.
- Online: queue size and precision on reviewed transactions immediately; recall as chargebacks mature, reported with the maturity of the window it was measured on.
- Over time: the pair per month on a fixed-maturity basis, so a fall in recall is visible as a trend rather than dismissed as "the labels are not in yet".
What can go wrong
- The label matures over 90 days, so recall measured on last month is optimistic — the late chargebacks have not arrived — and every fresh model looks better than the old one until its labels catch up.
- Precision-at-capacity is measured, capacity changes, and the number now describes a queue that no longer exists.
- The threshold is tuned to Fβ with a β nobody can justify, and the next person changes β to "improve the metric".
- Reporting two numbers instead of one means someone has to say which matters more when they disagree, which is the conversation F1 lets everyone avoid.
- Expected cost is the right objective and requires costs that finance may only estimate; a bad estimate produces a confidently wrong threshold.
- Precision-at-capacity depends on a capacity that moves with staffing, so the metric has a parameter that is not the model's.
- "F1 improved, so the model is better." F1 improved because precision rose more than recall fell. If the business cost is dominated by misses, the model got worse for the business.
- "We optimise F1 because it balances precision and recall." It balances them at a 1:1 cost ratio. Nobody at the company believes a wasted review costs as much as a missed fraud.
- "Recall is low, so we need a better model." Recall is a function of the threshold. Lower it first and see what precision the review team can live with; a better model moves the whole curve, a threshold moves along it.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe definitions and the opposite motion under a threshold hold for any scored binary classifier; the question of which one maps to which complaint is domain-specific and has to be asked each time.
- DOMAIN-SPECIFICIn fraud and medicine the FN cost usually dominates and recall is the number to protect; in spam filtering or content recommendation a false positive lands on a user and precision is protected. F1 treats both settings identically, which is why it is wrong in both.
- SIMULATEDThe counts and any precision or recall values below are for the shape of the argument, produced by the module's threshold model rather than measured on a payments dataset.
Where the depth lives
This domain teaches the model and hands the rest off by name.