The moment the course has been building toward
You have met the whole supervised zoo and watched each animal face the Lumen data one at a time. Today they compete side by side, under one fair set of rules, and you deliver a verdict the way a working analyst does - not "logistic scored 0.69" but "we can rank buyers well enough to prioritise the top decile, and here is the dollar case." The bake-off is a capstone in the real sense: nothing new to learn, everything to put together.
How to run a fair bake-off 6 min live
A comparison is only worth as much as its fairness. Three rules carry over from session 9 and become law today: candidates enter with the settings cross-validation chose (no last-minute fiddling), every candidate is scored on the same held-out test set, and that test set is touched exactly once. Then the reporting rule from intro-ml: translate the metric into money before you say it out loud.
LiveThe leakage checklist - one last pass3 min▶
Before any number counts, confirm the pipeline is honest. This is the checklist that separates a real result from a lie you will discover in production:
- Every transformer (scaler, one-hot encoder) lives inside the
Pipeline, so it is fit on train folds only - never on the full data. - The train/test split happened once, up front, with
random_state=42andstratify=yon the classification task. - No feature was computed from the target or from information unavailable at prediction time.
- The test set has not been used to choose anything - not a model, not a hyperparameter, not a threshold.
LiveWhat "winning" actually means3 min▶
The top of the leaderboard is necessary, not sufficient. A model wins the bake-off when it clears three bars at once:
- Best (or tied-best) test score - it generalises.
- Explainable - you can tell a stakeholder why it predicts what it predicts. A readable coefficient beats a marginally higher score you cannot narrate.
- Shippable - it trains, scores, and monitors within the constraints you actually have (latency, memory, retraining cadence).
Two leaderboards, one honest surprise 14 min live
Here is the whole course on two tables. Each candidate ran once on the held-out test set with its session-9 settings. Read them slowly - the ranking is the lesson, and the lesson is that sophistication did not win.
LiveReading the classification podium5 min▶
Full features, test ROC-AUC:
| Model | Test ROC-AUC | Why it landed here |
|---|---|---|
| Logistic regression | ~0.69 | a straight boundary is the right shape for this data |
| Linear SVM (LinearSVC) | ~0.69 | same straight-boundary family - it ties, and that is the tell |
| Decision tree (depth 5) | ~0.66 | axis-parallel cuts approximate the line, a little worse |
| Gaussian naive Bayes | ~0.66 | a fast, wrong-but-useful probability model |
| kNN (k=101, scaled) | ~0.64 | local voting in a 20-dimensional one-hot space, blurred |
| SVC RBF (default) | ~0.56 | an untuned curved boundary on data that wants a line |
The pattern behind the ranking is not "fancier is better" - it is boundary-shape match beats algorithm sophistication. Logistic and LinearSVC tie because they draw the same object: a straight cut through feature space. When the true signal is (log-)linear, that is exactly the right hypothesis, and nothing more elaborate can do better by being more elaborate.
LiveReading the regression podium4 min▶
On the 1,306 buyers, test R² and RMSE:
| Model | Test R² | RMSE | Read |
|---|---|---|---|
| Linear regression | ~0.59 | ~$25 | the workhorse wins |
| Decision tree (depth 4) | ~0.54 | - | rectangles approximating a ramp |
| kNN regressor (k=25) | ~0.46 | - | neighbor-averaging, over-smoothed |
| Mean baseline | 0.00 | ~$40 | the floor every model must beat |
The regression story rhymes with the classification one. Order value rises smoothly with prior spend (the r ~ 0.6 the statistics course fit), so a straight line captures it best. The tree carves that smooth ramp into steps and loses ground; kNN averages across neighbours and blurs it. Same lesson, second task: match the model to the shape of the truth.
Self-studyWhat would flip the ranking3 min▶
The linear family did not win because linear models are generally best - they won because the Lumen generator is log-linear by construction. Change the data and the podium reshuffles:
- Interactions and nonlinearity - if converting depended on "returning customers on mobile after 9pm," trees and gradient-boosted ensembles would pull ahead. That is the terrain of learn-ensemble-methods.
- Better features - a ratio, a rolling average, or a well-chosen interaction can lift every model at once. That is learn-feature-engineering.
- A curved boundary - on the two-moons data in the explorer, the RBF kernel that lost here wins comfortably. No free lunch.
Tell it like an analyst 10 min live
A number without a card is a rumour. The last skill of the course is packaging: a model card that a colleague can read in two minutes and act on, and a headline that leads with dollars, not decimals. This is what gets a model shipped.
LiveDollars before scores3 min▶
The single habit that makes a data person sound senior: translate the metric into money before you say the metric. Two examples straight from these leaderboards:
- Regression: not "R² is 0.59" but "our order-value model is off by about $25 on a typical order, against a $40 miss if we just guessed the average."
- Classification: not "ROC-AUC is 0.69" but "we rank buyers well enough that the top-scoring tenth of sessions is roughly two to three times richer in real buyers than a random tenth - so target them first."
LiveThe recommendation4 min▶
Given the two leaderboards and the three bars of winning, the call is clear:
- Ship logistic regression for conversion. Tied-best AUC (~0.69), coefficients that read as odds ratios a marketer understands (email x4.1, returning x2.8), and millisecond scoring. LinearSVC ties it but gives no probabilities - logistic wins on explainability.
- Ship linear regression for order value. Best R² (~0.59), coefficients denominated in dollars, and an honest "$25 vs $40 baseline" headline.
- The fancy models lost with dignity. The RBF kernel and deep trees were not broken - they were the wrong hypothesis for log-linear data. Keep them in your kit for the day the data curves.
Self-studyBecause models drift2 min▶
Shipping is the start of the model's life, not the end. Conversion rates shift with seasons and campaigns, and a model trained on last quarter can quietly decay. The monitoring-and-drift lifecycle - what to watch, when to retrain - is covered from the leader's angle in intro-ml session a6. The model card's "review date" field exists precisely so that decay has an owner.
Where you go from here 4 min live
You now speak algorithm - you can name the shape each model bets on, tune it, compare it fairly, and present the result. Four sibling courses turn that fluency into an edge.
learn-feature-engineering - the highest-leverage skill in tabular ML. Better inputs lift every model on both leaderboards at once; this is usually where real gains come from, not fancier algorithms.
learn-ensemble-methods (XGBoost & LightGBM) - the tabular champions. When the truth has interactions and nonlinearity that beat a straight line, gradient-boosted trees are what win in practice.
learn-model-evaluation - deeper judging: calibration, cost-sensitive thresholds, and the metrics beyond AUC that a high-stakes decision needs.
learn-experimentation (live now) - the leap from predicting to causing. Every model here answers "what will happen"; experimentation answers "what happens if we change something." The natural next rung.
.fit() and hopes - you are someone who chooses, and can say why.Run the bake-off, write the card ★ 20 min · everyone builds
Load both tasks, run every session-9-tuned pipeline exactly once on the held-out test set, print both leaderboards, and emit one model card. This is the script you keep and re-run for every future project.
Assemble the finalists. Each candidate is a full Pipeline carrying the settings CV chose in session 9 - no retuning here.
Fit on train, score once on test. One loop, one roc_auc_score (or r2_score) per model. The test set is spent now, for the first and only time.
Print both leaderboards sorted. The ranking should match the tables above within library jitter.
Emit a model card. One dict per finalist, dumped as readable text - the artifact you hand a stakeholder.
Finish the course strong ◐ 40 min
- Write a full model card for every finalist, not just logistic - a short paragraph each, with the business read in dollars or decile lift.
- Pull out the model-recommendation memo you wrote in session 9 and grade it against these actual test results. Where were you right, where surprised, and why?
- Add the SVC-RBF finalist (on a 6,000-row subsample) to the classification bake-off and confirm it lands near ~0.56 with defaults - the honest last-place finish.
- Optional, audience: post your two leaderboards and one insight ("the linear models won, and here is why") on LinkedIn and link the course. One cheerful line, no pressure - teaching what you learned is how it sticks.
Three questions before you go 🎯 ◐ 90 seconds
1 · Why does every candidate get exactly one pass over the test set?
The moment you use the test set to choose anything, it stops being an honest estimate of new-data performance. Tuning happened on cross-validation in session 9; the test set is touched once, at the end, and never again.
2 · Logistic regression and LinearSVC tied at ~0.69 AUC. What does the tie tell you?
Both are linear classifiers: they fit a straight boundary in feature space. When that shape matches the truth, both hit the same wall, and no amount of algorithmic difference between them changes it. Boundary shape, not brand name, sets the ceiling.
3 · How should you present the regression winner to a stakeholder?
Dollars before scores. A decision-maker acts on "$25 versus $40," not on 0.59. Leading with the business translation is the habit that separates a report that gets shipped from one that gets filed.
What this session covers
The capstone introduces no new source content - it synthesises the nine sessions into a fair comparison and an analyst's write-up. The threads pulled together: