SIMULATED

Threshold Explorer

A classifier emits a score; a threshold turns it into a decision. Slide it and watch precision, recall, F1, the confusion matrix and the business cost move — in opposite directions, at rates that depend on prevalence and on what each mistake costs.

Problem→Target→Data→Representation→Split→Model→Training→Evaluation→Validation→Deployment→Inference→Monitoring→Drift→Retraining

The threshold is a business decision wearing a number. It fixes how many items receive the action and how many of those deserved it, and the right value depends on things the model never sees: how much a false alarm costs, how much a miss costs, how many the team can handle. 0.5 is the default of a library, not of a problem. Set the two costs below to whatever your problem charges, then press the cost-optimal button and notice how far from 0.5 it lands — and switch prevalence to 1% to watch a working model lose to one that predicts nothing.

The data and the costs

Prevalence — how rare the positive class is
Separation — how well the model tells the classes apart · 1.00
Miscalibration · 0.00 (calibrated)
Cost of a false positive
Cost of a false negative

The threshold

An item is flagged when its score is at or above this value.

threshold · 0.50
precision
0.798
recall
0.437
F1
0.565
accuracy
0.933
flagged
109 / 2000
cost
1,142
Confusion matrix at threshold 0.50
True positive
87
caught positive
False negative
112
missed positive
False positive
22
negative flagged as positive
True negative
1,779
correctly left alone
n = 2,000precision = 0.798recall = 0.437accuracy = 0.933
a false positive costs 1 each × 22 = 22
a false negative costs 10 each × 112 = 1,120

Total cost 1,142. The cost-optimal threshold in a 101-point sweep is 0.13 at cost 597.

Precision and recall against the threshold

They move in opposite directions, and the crossing point is not where the cost is lowest.

00.5110thresholdrate
precisionrecallcurrent thresholdcost-optimal

ROC and precision–recall

Both are threshold-free summaries of the same scores. Only one of them notices prevalence.

ROC · AUC 0.915
011false positive ratetrue positive rate

The ROC AUC is the probability that a random positive outscores a random negative. It is the same at every prevalence, which is why it can look excellent on a problem where the model is useless.

Precision–recall · AP 0.687
011recallprecision

The dashed line is the prevalence (10.0%) — the precision of flagging at random. Under imbalance this curve is the honest one, because it is measured among the items you would actually act on.

Calibration

What the model claimed against what happened, in ten score bins. The dot at (0.7, 0.7) means seven in ten items scored 0.7 were positive.

bin 0: 1553 items, claimed 0.02, observed 0.02bin 1: 168 items, claimed 0.14, observed 0.15bin 2: 83 items, claimed 0.24, observed 0.20bin 3: 52 items, claimed 0.35, observed 0.38bin 4: 35 items, claimed 0.44, observed 0.46bin 5: 23 items, claimed 0.54, observed 0.61bin 6: 22 items, claimed 0.64, observed 0.68bin 7: 26 items, claimed 0.75, observed 0.77bin 8: 18 items, claimed 0.84, observed 1.00bin 9: 20 items, claimed 0.95, observed 1.00011mean predicted scoreobserved positive rate

Calibrated by construction: the score is the true posterior for this mixture. Move the miscalibration slider and watch the dots leave the diagonal while the AUCs above do not move at all.

Accuracy against the majority baseline

The classifier that predicts negative for everyone. Its accuracy is exactly 1 − prevalence.

model accuracy at this threshold
0.933
predict nothing, always
0.900

How to read this page honestly

What the model is, and what it deliberately refuses to be.

  • SIMULATED2,000 scored items drawn from two unit-variance Gaussians at ±separation, seeded. The score is the *exact posterior* for that mixture, so the calibration curve hugs the diagonal because the data really is calibrated, not because the curve was drawn there. Every metric on the page is computed from those items at the threshold you chose.
  • SIMULATEDThe miscalibration knob scales the logit. It is monotone, so ROC AUC, PR AUC and the whole precision/recall sweep are *exactly unchanged* while the reliability plot bends away from the diagonal. That is the point: discrimination and calibration are different properties, and a threshold chosen on one model's scale is wrong on another's.
  • SIMPLIFIEDPrecision is defined as 0 when nothing is flagged, so the sweep has no holes. The cost model is linear — one number per false positive and one per false negative — which is the simplest model that already makes 0.5 the wrong threshold; real costs vary by item and the ranking should reflect that.

Take it further

Leakage Simulator →

A threshold on a leaked score is a decision about a number that will not exist in production.

Threshold Selection →

Choosing it from the costs and the capacity, and revisiting it when they change.

Calibration →

Why a score is not a probability until somebody checked.

Accuracy Under Imbalance →

The 99% that means nothing, and what to report instead.