PR AUC
Precision against recall across every threshold, and the area under it. It follows the positive class, so it falls when the flagged set fills with negatives — which is exactly what ROC AUC cannot see.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
What does the precision–recall curve show that the ROC curve hides, and why is its area the honest ranking summary when positives are rare?
A security team's alert-triage model ranks incoming alerts so analysts work the likeliest true incidents first. Two candidate models have nearly identical ROC AUC. Analysts who trialled both say one is "obviously better" in the first hour of the shift. The team lead wants a number that agrees with the analysts.
ROC AUC is the standard; the two models tie on it, so they are equivalent and the analysts are reacting to noise. Pick the cheaper one.
The two models order the thousands of easy negatives identically, which is most of the pairs ROC AUC counts. They differ in how they order the hard negatives against the true incidents at the top — the only part of the ranking the analysts see — and ROC AUC gives that part almost no weight.
- The two models order the thousands of easy negatives identically, which is most of the pairs ROC AUC counts. They differ in how they order the hard negatives against the true incidents at the top — the only part of the ranking the analysts see — and ROC AUC gives that part almost no weight.
- On the PR curve the difference is large: one model holds high precision until a decent recall before falling; the other falls immediately. The analysts' "first hour" is the left end of that curve.
- The team lead's number was ROC AUC because it is the default; PR AUC was never computed, so the analysts' experience had no metric to agree with (ROC AUC).
- Once PR AUC is adopted, it is compared across weeks with different incident rates without stating the prevalence, and a rise in incidents reads as a model improvement.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether an alert corresponds to a real incident. The label is the analyst's disposition after investigation, which exists only for alerts that were worked (Selection Bias).
- Analysts work from the top of the ranking for a fixed shift, so the product is precision in the top slice of the list at whatever recall that slice achieves.
- One example is one alert: source, rule, asset criticality, recent alert count on the asset, and features from the alert payload.
- True incidents are a small fraction of alerts. The vast majority of negatives are noisy rules firing on routine activity and are easy to rank low.
- Dispositions are recorded inconsistently by shift, so the label has noise that varies by time of day (Label Quality).
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- The PR curve plots precision against recall as the threshold sweeps. Recall is TP over all positives; precision is TP over everything flagged. Precision's denominator includes the false positives, so as the threshold drops and negatives pour into the flagged set, precision falls in proportion to how many negatives there are — which is where prevalence enters and where the ROC curve's FP *rate* hides it (Precision, Recall & F1).
- The area under the PR curve summarises precision across all recall levels. Its floor is the prevalence: a random ranking gives precision equal to the base rate at every recall, so PR AUC ≈ π for a useless model. That makes PR AUC prevalence-dependent, which is a feature — the metric knows how hard the problem is — and a caveat: it cannot be compared across populations without stating π.
- Because it weights the high-precision, low-recall end as heavily as the rest, PR AUC is sensitive to precisely the region a top-of-list consumer uses. Two models that tie on ROC AUC by agreeing about easy negatives separate on PR AUC when they disagree about the hard ones.
- Average precision — the mean of precision at each positive's rank — is the usual computable form and is nearly identical to the area under the step-shaped curve; interpolated versions differ slightly and should not be mixed across reports.
Same sweep, different axes
Both curves are drawn from one threshold sweep. The ROC curve plots TP rate against FP rate — each normalised by its own class — and the PR curve plots precision against recall. Recall is the same number as the TP rate. The difference is entirely the other axis: FP rate divides by the number of negatives, precision divides by the number flagged. When negatives outnumber positives a hundred to one, a tiny FP rate is a large fraction of the flagged set, and precision says so while FP rate does not.
That is why the two models tie on one and separate on the other. They agree about the easy negatives, which is nearly all of the FP-rate axis, and disagree about the hard negatives near the top, which is where precision is decided.
1import numpy as np2 3def curves(y, p):4 order = np.argsort(-p) # descending score5 y_sorted = y[order]6 tp = np.cumsum(y_sorted) # positives flagged at each cut7 fp = np.cumsum(1 - y_sorted) # negatives flagged at each cut8 n_pos, n_neg = y.sum(), len(y) - y.sum()9 recall = tp / n_pos # = TP rate, shared by both curves10 fpr = fp / n_neg # ROC's x-axis: divides by ALL negatives11 precision = tp / (tp + fp) # PR's y-axis: divides by what was FLAGGED12 return recall, fpr, precision13 14def average_precision(y, p):15 recall, _, precision = curves(y, p)16 y_sorted = y[np.argsort(-p)]17 return precision[y_sorted == 1].mean() # mean precision at each positive's rank ≈ PR AUCThe only line that differs between the curves is the denominator. Add a thousand easy negatives at the bottom of the ranking: fpr barely moves at the top, precision at every cut that includes some of them collapses.
The floor is the prevalence
A random ranking flags positives at the base rate wherever it cuts, so its PR curve is a flat line at π and its PR AUC is π. A useless model on a 1%-prevalence problem has PR AUC near 0.01 and ROC AUC near 0.5 — the same model, two floors. That is what "prevalence-dependent" means: PR AUC starts from how hard the problem is, which is honest, and it is why a PR AUC must always be read next to π.
It is also why comparing PR AUC across weeks is dangerous. A week with twice the incidents lifts the floor and the whole curve with it, and a fixed model looks better. State π beside the number, every time.
Illustrative counts for the top slice of the ranking. The FP cell here is what the PR curve measures and the ROC curve barely registers: 38 against 4,900 negatives is a tiny FP rate and a precision the analysts feel every hour.
Both candidates tie; the report concludes they are equivalent and picks by cost. The analysts' trial is dismissed as anecdote.
Prevalence stated; random baseline drawn at π; both PR curves plotted on the same axes. One holds precision to a useful recall, the other does not. Precision at the shift's workload marked on each.
The consumer works the top of the list. The PR curve is the metric drawn from the consumer's side of the ranking, and its prevalence floor tells the reader how much of the number is the model and how much is the problem.
What must hold for PR AUC to keep meaning something
PR AUC is a function of the ranking and the prevalence, computed on whatever population was labelled. Three things can move it without the model changing: the incident rate, the labelling policy, and the population that gets labelled. The last is the quiet one — once the model's list drives which alerts get worked, the labels come only from the top of the model's own ranking, and the curve is computed on a population the model curated.
A random sample of unworked alerts, however small, is what keeps the curve honest. It costs analyst time on alerts the model ranked low, and it is the only source of evidence that ranking them low was right.
The labelled alerts used for the PR curve are a random sample of what the model ranks, not only the alerts the model or the previous process surfaced.
holds when A fixed fraction of unworked alerts is sampled and dispositioned each week, and evaluation uses that sample.
breaks when The sample is dropped under load; labels come only from worked alerts; the model becomes the only route to a disposition.
respond Reinstate the sample before trusting any curve; a PR AUC computed on model-selected alerts measures the model's agreement with itself (Feedback Loops).
How to build it
Most important first.
- On rare-positive problems with a ranked consumer, select and compare models on PR AUC or on precision at the consumer's k, and report ROC AUC only as a secondary, cross-period comparable number.
- State the prevalence next to every PR AUC, and report the random-ranking baseline (π) beside it so the number has a floor (Baselines Are Mandatory).
- Plot the curve, not just the area; two models with the same PR AUC can have different shapes, and the consumer lives at one end of it.
- Label a random sample of unworked alerts so the curve is computed on the population analysts will actually see, not on the alerts a previous ranking selected (Selection Bias).
- Walk the threshold in the Threshold Explorer at
/ml/thresholdwith prevalence set low; the PR curve and the ROC curve are drawn from the same sweep, and the difference in what they show is the lesson.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- PR AUC on a random-sample-labelled period, with prevalence, per model — the number that agrees with the analysts.
- Precision at the number of alerts a shift actually works, at the recall that implies — the analysts' hour, as a number.
- ROC AUC, for cross-period tracking; it is the number that said the models were equivalent.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- The prevalence on the evaluation period is stated with the PR AUC and is close to the served prevalence; an incident-rate monitor per week checks it.
- The labelled population includes a random sample of unworked alerts, so the curve describes the list analysts see; the fraction of labels from the random sample is tracked.
- The consumer still works from the top of the ranking for roughly the same number of alerts; a change in shift capacity changes which part of the curve matters.
- Offline: PR curve and PR AUC for both models on the same random-sample-labelled period, with prevalence and the random baseline; a bootstrap interval on the difference.
- Online: precision on the alerts worked per shift, by model, during the trial; analyst-reported time-to-first-true-incident as a sanity check.
- Over time: PR AUC monthly with prevalence beside it; ROC AUC alongside for the cross-period view.
What can go wrong
- PR AUC rises because incidents rose that week, and the report reads it as model improvement.
- With few positives the curve is jagged and PR AUC has a wide interval; two models are declared different on a gap smaller than the noise (Metric Uncertainty).
- The label noise varies by shift, so PR AUC computed on night-shift dispositions is not comparable with the day-shift figure.
- PR AUC's prevalence dependence makes it honest about difficulty and useless for comparing across periods or populations without extra bookkeeping.
- Its sensitivity to the top of the ranking is also sensitivity to a handful of positives, so it is noisier than ROC AUC on small evaluation sets.
- A random sample of unworked alerts costs analyst time on alerts the model ranked low — which is the only way to know whether it was right to.
- "ROC AUC is the same, so the models are equivalent." They are equivalent on the pairs ROC AUC counts, most of which are easy negatives. Look at the PR curve, where the top of the list lives.
- "PR AUC went up this month, so the model improved." Check the prevalence first; PR AUC rises with the base rate for a fixed model.
- "PR AUC is low, so the model is bad." Compare it with the prevalence. A PR AUC several times the base rate is a strong model on a hard problem.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- GENERALThe relationship between the two curves — same sweep, different axes, precision carrying the prevalence — is mathematical and holds for any scored classifier.
- DATA-SPECIFICOn balanced data the two curves tell nearly the same story and ROC AUC's comparability wins; the divergence grows as prevalence falls and is severe below a few percent, which is where PR AUC becomes the honest summary.
- SIMULATEDAny AUC, precision or prevalence values quoted here come from the module's threshold model and are for the shape of the argument, not measurements on a security dataset.
Where the depth lives
This domain teaches the model and hands the rest off by name.