Curves + calibration

This is not folklore - it has a citation. Saito and Rehmsmeier (PLOS ONE, 2015) showed that ROC plots can deceive on imbalanced datasets while PR plots are more informative, and the reason is structural: precision is prevalence-sensitive and the ROC axes are not. TPR and FPR are each computed within one actual class, so the class ratio never enters the plot - a model looks identical on ROC whether positives are half the data or 3.2% of it. Precision mixes the classes in its denominator, so the PR curve feels the base rate directly. On rare-event problems - which b1 argued are MOST valuable classification problems - the curve that feels the base rate is the curve that matches production experience.

One more honesty clause, straight from sklearn's own docs: linear interpolation between points on a PR curve is overly optimistic. Precision does not move linearly between measured thresholds the way ROC's axes do, so computing PR-AUC with the trapezoidal rule (as auc(recall, precision) happily will) credits the model for area it never earned. This is why average_precision_score exists and deliberately avoids trapezoids - it is a step-wise weighted mean of precisions, not a geometric area.

from sklearn.metrics import roc_auc_score, average_precision_score

scores = model.predict_proba(X_test)[:, 1]
roc_auc_score(y_test, scores)           # ranking quality - flattering on imbalance
average_precision_score(y_test, scores) # PR-AUC done right: step-wise,
                                        # no optimistic trapezoids

Two-line takeaway for your next review: PR-AUC computed by trapezoid is a small silent inflation on top of a metric you chose FOR its honesty. Use average_precision_score and the inflation never happens.

Neither curve is retired. ROC-AUC remains the right single number when classes are roughly balanced, when you care about ranking across the full score range, or when comparing models across datasets with different base rates (its prevalence-blindness becomes a feature). PR-AUC wins whenever the positive class is rare and the flags are what you act on - fraud queues, outreach lists, alert triage. Google's crash course teaches ROC/AUC as the default lens; this session adds the imbalance asterisk that Lumen's 3.2% base rate makes mandatory.

Diagnosis and repair are three calls apart:

from sklearn.calibration import (CalibratedClassifierCV,
                                 CalibrationDisplay, calibration_curve)

# diagnose: binned predicted vs actual - the reliability diagram
prob_true, prob_pred = calibration_curve(y_test, scores, n_bins=10)
CalibrationDisplay.from_predictions(y_test, scores, n_bins=10)

# repair: wrap the model, refit the mapping on held-out folds
cal = CalibratedClassifierCV(model, method="isotonic", cv=5)
cal.fit(X_train, y_train)   # cal.predict_proba is now the trusted one

Two repair methods, one selection rule:

• Platt scaling (method="sigmoid") fits a logistic curve to the scores - two parameters, hard to overfit, the safe pick on small data or when the distortion looks sigmoid-shaped.
• Isotonic regression (method="isotonic") fits a free-form monotonic step function - more expressive, and hungry: sklearn's guidance is to reach for it with more than ~1,000 calibration samples, or it will happily overfit the wiggles.

Lumen's slice clears the isotonic bar on raw count - but remember the minority class: 2,000 sessions hold only ~64 converters, and calibration in the high-score bins is only as good as the positives available there. When in doubt, sigmoid first, isotonic when the reliability diagram shows a distortion the sigmoid cannot bend to.

The Brier score - mean squared error between predicted probabilities and 0/1 outcomes - looks like the obvious "calibration metric." Here is the caveat, and it is sklearn's calibration guide speaking, not a blog: a lower Brier score does NOT necessarily mean better calibration. Brier mixes two ingredients - calibration (do the probabilities match reality?) and discrimination (does the model separate the classes?) - into one number. A sharper, better-discriminating model can post a lower Brier than a more honest one; you cannot tell from the score alone which ingredient improved, or whether one improved at the other's expense.

Both are proper scoring rules over predicted probabilities; they differ in temperament. Brier is squared error on probabilities: bounded between 0 and 1, forgiving at the extremes - a confident wrong answer costs at most 1. Log loss charges the negative log of the probability assigned to the truth: unbounded, and brutal at the extremes - predict 0.99 for the wrong class and the penalty explodes toward infinity. Track log loss when confident wrongness is the failure mode you most need to punish (it is also what logistic-family models optimize); track Brier when you want a stable, bounded probability-quality number. And carry tonight's caveat to both: neither one, alone, certifies calibration - the reliability curve does.