When accuracy is the wrong score
You met the accuracy paradox in b3: with a 3.2% base rate, "always predict no" scores 96.8% and catches zero buyers. High accuracy, zero value. The fix is not a better model first - it is a better ruler. Once you measure precision and recall instead of accuracy, the picture snaps into focus, and once you realise the 0.5 decision threshold is a choice rather than a law, you can tune the model to the business cost you actually face. This session is where scoring gets honest.
classification_report, compute roc_auc_score (~0.71 on Lumen), and slide the threshold to trade precision for recall by business cost.
Beyond accuracy 8 min live
Every classification score starts from one table: the confusion matrix. It cross-tabs what actually happened against what the model predicted, giving four cells - true positives (TP), false positives (FP), false negatives (FN), true negatives (TN). Two ratios pulled from those cells tell the real story. Precision reads down the predicted-positive column: of everyone you flagged as a buyer, how many truly were? Recall reads across the actual-positive row: of all the real buyers, how many did you catch? F1 is their harmonic mean, and ROC-AUC summarises performance across every threshold at once.
LivePrecision vs recall3 min▶
They answer different questions, and the difference is the whole game:
- Precision = TP / (TP + FP). Of everyone you predicted is a buyer, what share truly are? High precision means few false alarms - when you flag someone, you are usually right.
- Recall = TP / (TP + FN). Of all the actual buyers, what share did you catch? High recall means you miss few real buyers, even if you raise some false alarms doing it.
You almost never get both at once. Chase recall and precision falls; chase precision and recall falls. F1, their harmonic mean, is a single number for when you want to balance the two - but the right balance depends on cost, which is where the threshold comes in.
Self-studyThe threshold is yours to move2 min▶
A classifier does not output a class - it outputs a probability. The 0.5 cutoff that turns that probability into a yes/no is a default, not a law. Lower it and you flag more sessions as buyers: recall climbs (you catch more real buyers) while precision falls (more false alarms). Raise it and the reverse happens.
So the threshold is a business dial, not a modelling detail. You set it by the relative cost of a false positive versus a false negative - and no algorithm can pick that for you, because it depends on your economics, not your data.
Feel it for yourself. The tool below runs Lumen's logistic model and lets you drag the decision threshold - watch the confusion matrix and the precision/recall numbers move in real time, and see why 96.8% accuracy hides a model that catches no one.
Compute the honest metrics in Python 18 min · everyone builds
We refit the b3 logistic model on Lumen's numeric features - scaled on the training set only - then score it the honest way: classification report, confusion matrix, ROC-AUC, and finally a moved threshold.
Reload, split, and refit the b3 model. Scale on train only (fit the scaler on train, apply to test) to avoid leakage - the pipeline version lands in b9.
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
df = pd.read_parquet("lumen_sessions.parquet")
num = ["prior_30d_spend", "pages_viewed", "session_hour"]
X, y = df[num], df["converted"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
scaler = StandardScaler().fit(X_train) # fit on TRAIN only
clf = LogisticRegression(max_iter=1000).fit(scaler.transform(X_train), y_train)
proba = clf.predict_proba(scaler.transform(X_test))[:, 1] # P(buyer)
Read the classification report and confusion matrix. At the default 0.5 threshold, precision is low and recall is moderate - accuracy alone would have hidden both.
from sklearn.metrics import classification_report, confusion_matrix pred = (proba >= 0.5).astype(int) print(confusion_matrix(y_test, pred)) # [[TN, FP], [FN, TP]] print(classification_report(y_test, pred, digits=3)) # accuracy looks ~96-97% (the paradox) while precision/recall for the buyer class are modest
Notice the buyer-class precision and recall in the report - both far below the headline accuracy. That gap is the accuracy paradox, now measured instead of just described.
Score across all thresholds with ROC-AUC. One number that does not depend on where you set the cutoff.
from sklearn.metrics import roc_auc_score, roc_curve
import matplotlib.pyplot as plt
print("ROC-AUC = %.2f" % roc_auc_score(y_test, proba)) # -> ~ 0.71
fpr, tpr, _ = roc_curve(y_test, proba)
plt.plot(fpr, tpr, label="Lumen logistic")
plt.plot([0, 1], [0, 1], "--", color="grey", label="random")
plt.xlabel("false positive rate"); plt.ylabel("true positive rate"); plt.legend()
plt.show()
ROC-AUC lands at ~0.71 - a real but modest signal, comfortably above the 0.5 of random guessing but nowhere near perfect. Because AUC averages over all thresholds, it is the fairest single summary of this model on imbalanced data.
Move the threshold and re-score. Drop it from 0.5 to 0.2 and watch recall climb while precision falls; plot the full precision-recall curve to see the whole trade.
from sklearn.metrics import precision_score, recall_score, precision_recall_curve
for t in [0.5, 0.2]:
p = (proba >= t).astype(int)
print("t=%.1f precision=%.2f recall=%.2f" % (
t, precision_score(y_test, p), recall_score(y_test, p)))
# lower t -> recall up, precision down
prec, rec, _ = precision_recall_curve(y_test, proba)
plt.plot(rec, prec); plt.xlabel("recall"); plt.ylabel("precision"); plt.show()
Lumen set the threshold by weighing two costs: a false positive is a wasted free-shipping offer sent to someone who was never going to buy, while a false negative is a genuine high-value buyer they failed to nudge. Because a missed big spender hurt more than a wasted coupon, they lowered the threshold to buy recall - accepting more false alarms to catch more real buyers. The maths gave them the curve; the business chose the point on it.
This week ◐ 40 min total
- Build the confusion matrix by hand. Take
probaat two thresholds (say 0.5 and 0.2) and count TP/FP/FN/TN yourself with boolean masks. Confirm your counts matchconfusion_matrix. - Find the F1-maximising threshold. Sweep the threshold from 0.05 to 0.95 and compute F1 at each. Where does F1 peak, and how does that point compare to 0.5?
- Explain the cost call. In one paragraph, describe a situation where you would deliberately prefer high recall over high precision - and one where you would prefer the reverse.
- Optional: compare the ROC curve to the precision-recall curve. On data this imbalanced, which one tells the more honest story, and why?
Three questions before you go 🎯 ◐ 90 seconds
1 · What does precision measure?
Precision = TP / (TP + FP) - it reads down the predicted-positive column: of everyone you flagged as a buyer, how many really were. Catching actual buyers is recall; overall correctness is accuracy.
2 · What does recall measure?
Recall = TP / (TP + FN) - it reads across the actual-positive row: of all the real buyers, how many did you catch. The predicted-column version is precision.
3 · You lower the decision threshold from 0.5 to 0.2. What happens?
A lower threshold flags more sessions as buyers, so you catch more real buyers (recall up) but raise more false alarms (precision down). The threshold is a business dial, set by the cost of a false positive vs a false negative.
What this session covers
This builder session covers classification metrics in depth on the Lumen dataset - the confusion matrix and everything derived from it - building on b3's accuracy paradox. Regression metrics come next in b8.
Builder Session 7 cheat sheet · pin this
confusion_matrix(y, pred).