The model ranks well. Why would you care whether it is calibrated?
Answer it out loud before you open anything. The value of the flags below is in comparing them to what you actually said — including whether you asked about the data before naming a model.
An insurer uses a claim-likelihood model to price quotes: expected cost equals probability of claim times expected claim size. The model has a high ROC AUC (illustrative). After launch, the book of business is unprofitable in the low-risk segment and quotes are uncompetitive in the high-risk one. The model was trained with heavy downsampling of non-claims.
React to this
Say what you would question, what you would trust, and what you would need to know first.
Reliability diagram — claim model v3, validation set (illustrative) predicted bin n mean predicted observed claim rate [0.00-0.05) 41,200 0.031 0.008 [0.05-0.10) 22,900 0.072 0.021 [0.10-0.20) 14,100 0.141 0.046 [0.20-0.40) 6,300 0.281 0.112 [0.40-1.00] 1,900 0.552 0.298 Training: non-claims downsampled 1:4. ROC AUC 0.84. Brier score: not reported.
What it is really testing
Whether the candidate understands the difference between ranking quality and probability quality, and can say exactly when it matters: whenever the score is *used as a number* — multiplied, thresholded against a cost, summed into an expectation — rather than merely sorted.