The judgement session
You have met all seven animals: lines (s2), sigmoids (s3), the engine underneath (s4), neighbors (s5), probability multipliers (s6), rectangles (s7), and margins (s8). Each session ended with a number on the Lumen leaderboard. Today we stop collecting animals and start choosing between them - which is a different skill entirely. Choosing well needs two things: a mental map that says where each algorithm lives, and a protocol that compares them without cheating. Both fit in one session, because you already did the hard part.
compare.py, a harness that tunes every candidate by 5-fold CV on train only and leaves the test set sealed for session 10. Fair comparison is not a vibe; it is a protocol.
The zoo on one axis 8 min live
Every algorithm you met sits somewhere on a single axis running from stiff (high bias: a strong opinion about the shape of the truth, hard to bend, hard to fool) to bendy (high variance: few opinions, fits anything, memorizes noise if you let it). And every knob you learned - polynomial degree, k, tree depth, gamma, C - does the same job: it slides its animal along this axis. Eight sessions of knobs were secretly one knob.
LiveThe knobs are one knob - complexity4 min▶
Slide too far left and the model is too stiff to see the pattern (underfitting: train AND test scores both poor). Slide too far right and it bends around every noisy point (overfitting: train score great, test score collapses). You have already watched every animal fall off the right edge - here is the reunion tour, with the numbers you produced yourself:
| Knob pushed to the edge | What happened on test | Session |
|---|---|---|
| Polynomial degree 12 (50 training points) | test R² -798,595 - six orders of magnitude worse than the mean | s2 |
| kNN k=1 | AUC 0.50 - memorization scores a coin flip | s5 |
| Tree depth None | train 1.00, test 0.51 - a perfect memory of the past, useless on the future | s7 |
| RBF gamma cranked up | boundary wraps single points (you watched it in the explorer) | s8 |
Four failures, one disease. Diagnosis is now easy: big train-test gap = too far right, slide left. Both scores low = too far left, slide right (or get better features). That is Ng's bias/variance diagnosis in one sentence.
LiveNo free lunch - why OUR leaderboard loves linear4 min▶
All course long, the linear family has topped the Lumen leaderboard (logistic ~0.69, LinearSVC ~0.69) while the bendier animals trailed. Say the honest reason out loud: the Lumen generator is log-linear. The machine that makes the data IS a linear machine, so models that assume linearity start with the right bet. That is not a law of nature - it is a property of this dataset.
On interaction-heavy tabular data (where the effect of one feature flips depending on another), trees and their ensemble descendants routinely win. There is no universally best algorithm - that is the "no free lunch" theorem in practice. The skill is matching the model's assumed shape to the data's true shape, and being suspicious of anyone who has one favorite algorithm for everything.
The decision guide 9 min live
The axis says how bendy each animal is; this table says what it costs to keep one. Every cell below is something you verified yourself in sessions 2-8 - this is not a blog-post cheat table, it is your own lab notebook, compressed.
| Algorithm | Needs scaling? | Trains fast? | Interpretable? | Handles nonlinearity? | Watch out for |
|---|---|---|---|---|---|
| Linear regression | Not for plain OLS | Yes - closed form | Yes - coefficients in real units ($) | Only via built features (poly) | Multicollinearity; degree explosion (s2's -798,595) |
| Logistic regression | Helps solvers + any penalty | Yes | Yes - readable odds ratios | No - one straight seam | Regularized by default; C is INVERSE strength |
| kNN | Mandatory (0.64 vs 0.58 AUC) | "Trains" instantly - but predicts slowly | No - the model is the data | Yes - follows the crowd | Ships the training set as the model: memory + slow predict |
| Naive Bayes | No | Fastest in the zoo | Somewhat - per-class stats | Mildly - per-feature only | Lying probabilities: never trust its predict_proba |
| Decision tree | No | Yes | Yes - you can read the rules aloud | Yes - rectangles | Instability (small data change, new tree) + must cap depth |
| SVM linear | Mandatory | Yes - LinearSVC scales fine | Partly - weights, but no probabilities | No | No predict_proba; use decision_function |
| SVM RBF | Mandatory | No - roughly O(n²), subsample big data | No | Yes - smooth curves | Tune gamma and C or lose to linear (0.56 vs 0.69 here) |
LiveStart simple - the baseline is the floor3 min▶
The most useful row of that table is the first two. A logistic or linear baseline trains in seconds, explains itself, and sets the floor every fancier model must beat. On Lumen, nothing beat it - and you only know that because you fit the simple thing first.
The professional habit: fit the simple model on day one, ship the number to the team, THEN explore. If a deep tree grid buys you +0.002 AUC over logistic, you have learned something valuable: the signal is linear, stop spending compute. If it buys +0.05, you have also learned something: go nonlinear, and say why.
Self-studyInterpretability is a requirement, not a preference3 min▶
In regulated settings - credit, insurance, healthcare, hiring - "why did the model say no?" is a question you must answer to a regulator or a customer, sometimes by law. There, the Interpretable column is not a nice-to-have; it eliminates candidates before accuracy is even discussed. A readable odds ratio ("returning customers have ~2.8x the odds") beats hand-waving at a black box every single time.
Treat interpretability as a requirement you gather up front, like latency or memory. If the answer is "we must explain every decision", your shortlist is logistic, a shallow tree, maybe linear SVM - and the bake-off runs among those.
Self-studyLearning curves and error analysis - the diagnosis toolkit3 min▶
Two habits from Ng's C2 W3 that complete the picture. Learning curves: plot train and CV score against training-set size. If the two curves have converged and both sit low, you have a bias problem - more data will NOT help; better features or a bendier model will. If a big gap remains between them, you have a variance problem - more data (or regularization, or sliding left on the axis) helps. Teams burn months collecting data for bias problems; the curve tells you in minutes.
Error analysis: read a sample of the actual mistakes. On Lumen you might find the model misses converters arriving from a channel it rarely sees - a feature fix, not an algorithm swap. The best model-choice tool is sometimes 30 minutes of looking at what the current one gets wrong.
The fair-comparison protocol 8 min live
A comparison is only worth what its protocol guarantees. Ours: every candidate gets its knobs tuned by 5-fold cross-validation on the training set only, each is judged by its CV mean and spread, and the held-out test set is spent exactly once - in session 10, on the finalists. Run our logistic model through it and you get CV AUC 0.706 +/- 0.010 (folds ranging 0.688-0.715, reconciling with intro-ml's "~0.71"). That +/- is not decoration: the spread across folds tells you how much a single lucky or unlucky split could move the number.
LiveGridSearchCV per candidate, one harness3 min▶
The mechanics are intro-ml's GridSearchCV, applied at zoo scale: one dict of (pipeline, param grid) pairs, one loop, cv=5 and scoring="roc_auc" for all. Because every pipeline carries its own preprocessing, scaling happens inside each CV fold - no leakage, no favoritism. Candidates with nothing to tune (GaussianNB) just run through the same loop with an empty grid, so every number on the board was produced by the identical procedure.
One nice detail: LinearSVC has no predict_proba, but scoring="roc_auc" falls back to decision_function automatically - the s8 lesson, paying rent.
LiveHARD RULE: never tune on test3 min▶
The test set answers one question - "how will the chosen model do on data it has never seen?" - and it can only answer it once. Every time you peek at test, adjust a knob, and peek again, the test set stops being unseen: you are selecting for whatever flatters those 10,000 particular rows. Repeated peeking turns your test set into a validation set, and the final score you report becomes an optimistic lie - one that production traffic will correct, publicly.
This is why compare.py below computes nothing on X_te. Not one number. The cliffhanger is deliberate: session 10 opens the vault.
Self-studyWhy report the +/-2 min▶
Our logistic folds ranged 0.688 to 0.715. If you had judged it on one unlucky split you would call it a 0.69 model; on one lucky split, a 0.72 model. The fold spread is the honest error bar on your estimate - a single split can be lucky, and the +/- is how you confess that.
It also breaks ties like an adult: a candidate at 0.70 +/- 0.01 is a better bet than one at 0.71 +/- 0.04, because the second one might be a 0.67 model wearing a lucky split. Stability is information. Report mean AND spread, always.
Build the fair harness ★ 18 min · everyone builds
One script, five candidates, identical treatment: each pipeline gets GridSearchCV with cv=5 and scoring="roc_auc" on the training set. We print each winner's params and CV mean +/- std - and we do not touch the test set. That restraint is the whole point of the session.
Load and split as always. Seed 42, stratified, 25% test - the same vault every session has used. The split happens; the test half then goes untouched.
Declare the candidates. A dict of (estimator, grid) pairs. Grids are small and honest: k in [25, 51, 101], depth in [3, 5, 8], C in [0.1, 1, 10]. GaussianNB and logistic run with empty grids - same loop, same judge.
Run the loop. Every candidate is tuned by the same 5-fold CV on the same folds of the same training data. Print best params, CV mean, CV std.
Stop. Rank by CV mean, note the spreads, write your recommendation - and leave X_te sealed. Session 10 opens it, once.
Before session 10 ◐ 30 min
- Add
SVC(kernel="rbf")as a sixth candidate - on a 6,000-row subsample of train (the s8 speed lesson), with a small grid overgammain [0.01, 0.1, 1] andCin [0.1, 1, 10]. Does tuning rescue it from the default's ~0.56? - Rank the candidates twice: once by CV mean, once by CV std (most stable first). Do the two rankings agree? Where they disagree, which would you trust for a production pick, and why?
- Write your model-recommendation memo NOW, before seeing any test number: which model do you ship for Lumen conversion, and your top three reasons. Session 10 grades it against reality - that is the fun of writing it first.
Three questions before you go 🎯 ◐ 90 seconds
1 · Why do we tune hyperparameters with cross-validation on train, instead of just picking whatever scores best on the test set?
Every peek-and-adjust cycle leaks information about the test rows into your choices. CV simulates unseen data using only the training set, so the test set stays a genuine stranger - and its one verdict, in session 10, stays honest.
2 · Your candidate's five CV folds score 0.63, 0.75, 0.66, 0.74, 0.64. What does that large spread tell you?
The fold spread is the error bar on your estimate. A wide spread means one split could easily mislead you - which is exactly why we report CV mean +/- std, and why 0.70 +/- 0.01 can be a better bet than 0.71 +/- 0.04.
3 · The Lumen leaderboard has favored linear models all course. The honest explanation is...
There is no universally best algorithm. Our data's true machine is linear, so linear bets win here - say that out loud in your readout. On nonlinear, interaction-heavy data the ranking flips, which is the ensembles course's opening story.
What this session covers
Session 9 covers the model-selection layer of the anchor sources: Ng's bias/variance diagnosis week and the practical-choice wisdom scattered across the scikit-learn module pages. Certificates, videos, and graded labs stay with the official providers.