Picking a model is a protocol, not a vibe
Beginners pick the model that scored best on the test set - which quietly turns the test set into a tuning set and inflates every number they report. Professionals separate the two jobs: use cross-validation to choose, and reserve the held-out test set for one final, honest estimate of the winner. This capstone walks that protocol end to end on Lumen, then writes the model card that makes the result shippable.
Compare, pick, then score once 6 min live
The order of operations is the whole discipline. You compare candidates on cross-validation, because a single split can be lucky; you pick the winner from those CV estimates; and only then do you open the test set - once - to report how the chosen model will generalise. Touch the test set earlier and it stops being an honest estimate.
LiveCompare fairly - same CV, same metric3 min▶
A fair comparison changes only the model. Everything else is held constant:
- Same splits. Every candidate is cross-validated on the identical folds, so a lucky split cannot flatter one model over another.
- Same metric. Pick the metric that matches the business question first (ROC-AUC for imbalanced conversion), then judge every model by it - not accuracy for one and F1 for the next.
- Same prep. Each candidate sits inside the same b9 Pipeline, so no model gets an unfair prep advantage or a leakage shortcut.
Self-studyScore the test set exactly once3 min▶
The held-out test set is a one-time resource. Its whole value is that the chosen model has never touched it, so the score is an unbiased estimate of real-world performance. The moment you score, look, adjust, and score again, you have used the test set to tune - and the estimate inflates.
- Do all choosing on cross-validation or a separate validation split.
- Refit the single winner on all the training data.
- Score on the test set once, report that number, and stop.
If the test result disappoints, the honest move is to gather more data or rethink features - not to keep poking the same test set until it smiles back.
Run it end to end, then peek at unsupervised 22 min · everyone builds
We classify converted using the b9 Pipeline, compare three models on cross-validation, score the winner once, write the model card, and finish with a taste of clustering. Your exact numbers will be near the canon ROC-AUC ~0.71 - teach the relationships, not a false fourth decimal.
Reload and rebuild the prep. Same ColumnTransformer as b9 - the model is the only thing we will swap.
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
df = pd.read_parquet("lumen_sessions.parquet")
y = df["converted"]
X = df.drop(columns=["converted", "order_value"])
num = ["prior_30d_spend", "pages_viewed", "session_hour"]
cat = ["channel", "device", "new_vs_returning", "product_category"]
pre = ColumnTransformer([
("num", StandardScaler(), num),
("cat", OneHotEncoder(handle_unknown="ignore"), cat),
])
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y)
Cross-validate three candidates. Same splits, same metric (ROC-AUC), each inside the same Pipeline. Print mean and spread per model.
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
models = {
"logistic": LogisticRegression(max_iter=1000, class_weight="balanced"),
"tree_d3": DecisionTreeClassifier(max_depth=3, class_weight="balanced"),
"forest": RandomForestClassifier(n_estimators=200, class_weight="balanced"),
}
for name, m in models.items():
pipe = Pipeline([("pre", pre), ("model", m)])
s = cross_val_score(pipe, X_tr, y_tr, cv=5, scoring="roc_auc")
print("%-9s ROC-AUC %.2f +/- %.2f" % (name, s.mean(), s.std()))
# logistic and forest around ROC-AUC ~0.71; the shallow tree a touch lower
Logistic and random forest land around the ~0.71 AUC region; the shallow tree sits a touch lower. Keep the language qualitative - your numbers will be near the canon, and the ranking is the point, not the fourth decimal.
Pick the winner, refit, score once. Choose by the CV estimate, refit on all training data, then open the held-out test set a single time.
from sklearn.metrics import roc_auc_score
best = Pipeline([("pre", pre),
("model", LogisticRegression(max_iter=1000, class_weight="balanced"))])
best.fit(X_tr, y_tr) # refit on ALL train
proba = best.predict_proba(X_te)[:, 1]
print("held-out test ROC-AUC %.2f" % roc_auc_score(y_te, proba)) # ~ 0.71, scored ONCE
One number, reported once. It sits near the CV estimate, which is exactly what a clean protocol should produce - no nasty surprise, because we never tuned on the test set.
Write the model card. A model nobody can maintain is not shipped. Capture task, data, metric, limits, and monitoring in one small table.
card = pd.DataFrame([
("Task", "Predict session conversion (binary)"),
("Data", "40k Lumen checkout sessions, ~3.2% convert"),
("Model", "LogisticRegression in a leak-proof Pipeline"),
("Metric", "ROC-AUC ~0.71 (held-out test, scored once)"),
("Limits", "Modest signal; threshold is a business choice"),
("Monitoring", "Watch AUC + base rate drift monthly"),
], columns=["field", "value"])
print(card.to_string(index=False))
| Field | Value |
|---|---|
| Task | Predict session conversion (binary) |
| Data | 40k Lumen checkout sessions, ~3.2% convert |
| Model | LogisticRegression in a leak-proof Pipeline |
| Metric | ROC-AUC ~0.71 (held-out test, scored once) |
| Limits | Modest signal; classification threshold is a business choice |
| Monitoring | Watch AUC and the base rate for drift, monthly |
A taste of unsupervised. Everything so far had a label. KMeans has none - it groups similar customers on its own, from spend and order value alone.
from sklearn.cluster import KMeans
buyers = df[df.converted == 1][["prior_30d_spend", "order_value"]]
Xs = StandardScaler().fit_transform(buyers) # scale before distance-based clustering
km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(Xs)
buyers = buyers.assign(segment=km.labels_)
print(buyers.groupby("segment").mean().round(0))
# three natural segments emerge, e.g. budget / core / high-value
On the final comparison, random forest edged logistic regression on AUC by a hair - and Lumen shipped logistic regression anyway. The reason was interpretability: the marketing team needed to explain to finance why a session scored the way it did, and coefficients they could read beat a small AUC gain they could not. They added monthly monitoring on AUC and the base rate, and moved on. A slightly better score is not always the better business decision.
This week ◐ 40 min total
- Write a model card for one of your own use cases - task, data, metric, limits, monitoring. If you cannot fill in "limits" and "monitoring", the model is not ready to ship.
- Add a fourth candidate to the comparison (for example a gradient-boosting or K-nearest-neighbours classifier), on the same CV splits and metric. Does the ranking change?
- Re-run KMeans with a different k (try 2 and 4) and eyeball the segments. Which k tells the clearest customer story? There is no label to grade you - that is the unsupervised world.
- Optional: write the one sentence you would say to a stakeholder to justify shipping your chosen model - the way Lumen justified logistic over the forest.
Three questions before you go 🎯 ◐ 90 seconds
1 · What is the right order for choosing and reporting a model?
Choosing on the test set turns it into a tuning set and inflates the reported number. The honest protocol chooses on cross-validation, refits the single winner, and opens the held-out test set exactly once for the final estimate.
2 · What does KMeans do?
KMeans is unsupervised: it finds groups of similar rows from the features alone, with no target column to learn from. That is a different job from the supervised classification and regression the rest of this course covered.
3 · Why score the held-out test set only once?
The test set is an honest estimate only while the model has never seen it. Score, adjust, and re-score, and you are tuning on the test set - the number climbs but no longer reflects real-world performance. Choose on CV, spend the test set once.
What this session covers
This capstone puts the whole workflow together - compare, pick, score once, document - and opens the door to unsupervised learning and the courses that come next.