From prop to subject
A decision tree is the zoo's most legible animal: a stack of yes/no questions anyone can follow with a finger. That legibility is earned by a brutally simple training procedure - greedily audition every possible question, keep the one that un-mixes the classes most, repeat inside each branch. No loss surface, no gradient, no scaling requirements. The catch is the same one intro-ml drew as a U-curve: a tree flexible enough to answer anything is flexible enough to memorize everything, so the craft of trees is the craft of stopping them.
How a split is chosen 9 min live
Training a tree is an audition, not an optimization: at each node, try every feature at every threshold, score each candidate split by how pure its two children would be, and keep the winner. Then recurse into each child and audition again. "Pure" means one-class: a node that is all converters (or all non-converters) needs no further questions. The score sklearn uses by default is gini impurity - for a binary node, 2p(1-p), where p is the converter share. It is 0 for a pure node, and peaks at 0.5 for a 50/50 coin-flip node.
LiveGini vs entropy - the debate that is not one3 min▶
Entropy is the other purity score, and with it comes information gain - Ng's C2 framing: a split's value is the reduction in entropy from parent to (weighted) children. Gini and entropy are shaped almost identically: both are 0 at purity, both peak at 50/50, and they usually pick the same splits.
On Lumen the whole debate collapses into one line: depth-5 tree with gini scores 0.660 test AUC, with entropy 0.653. Near-identical. Gini is sklearn's default because it is cheaper to compute (no logarithms), not because it is more accurate - never teach or believe the "entropy is more precise" folklore. Pick gini, move on with your life, and spend the attention on depth and leaf size, which actually move the number.
LiveNo scaling needed - and what to do with categoricals3 min▶
Every split compares values within one feature: is spend above 120, is the channel email. Features never get added together, so their units never compete - the zoo's low-maintenance animal needs no StandardScaler, ever. (Compare session 5, where skipping the scaler cost kNN six points of AUC.)
Categoricals still need encoding, though: sklearn's CART implementation has no native categorical support, so we one-hot them - exactly as Ng's tree module teaches. One-hot turns each category into its own yes/no split candidate, which trees digest happily.
Self-studyWhere do thresholds for continuous features come from?2 min▶
For a continuous feature like prior_30d_spend, the audition does not try infinitely many thresholds - it sorts the values seen at that node and tries the midpoints between consecutive sorted values. With n distinct values, that is at most n-1 candidate thresholds, each scored by the purity of the split it produces.
This is why tree training scales with data size times feature count, and why a tree can place a threshold anywhere the data suggests - it is not binning, it is auditioning every boundary the training data can distinguish.
Growth, and how to stop it 9 min live
Left unchecked, the audition never ends: the tree keeps splitting until every leaf is pure - one leaf per stray point if that is what purity takes. That is memorization wearing a flowchart costume. Here is intro-ml's U-curve again, now measured in AUC on Lumen (full features, test set):
| max_depth | Test ROC-AUC | Reading |
|---|---|---|
| 2 | 0.632 | too shallow - good questions left unasked |
| 5 | 0.660 | the peak |
| 8 | 0.650 | starting to fit noise |
| None | 0.509 (train 1.000) | pure memorization - a coin flip on new data |
LivePlay: depth on the boundary explorer3 min▶
Crank depth on the moons and watch the tree carve one rectangle per stray point - memorization you can see.
LivePre-pruning - the knobs that stop growth early3 min▶
Three knobs cover most real work. max_depth caps how many questions any path may ask. min_samples_split refuses to audition splits at nodes with too few rows. min_samples_leaf refuses any split that would leave a child below the floor - the sklearn practical-tips list suggests starting at 5, because a leaf holding one or two rows is almost always memorized noise.
Depth has a hidden cost worth internalizing: since each level can double the number of leaves, a tree needs roughly twice the samples for every extra level of depth it is going to use honestly. Depth 10 wants around a thousand times the data of depth 1 - which is why "just grow deeper" stops working long before you run out of RAM. The same tips list says to visualize the tree as you train (plot_tree, Part 3) rather than tuning blind.
LivePost-pruning - grow big, then cut with ccp_alpha3 min▶
The other philosophy, from StatQuest's cost-complexity pruning episode: grow the full tree first, then prune it back. Weakest-link pruning scores every subtree by error plus a complexity penalty - alpha times the number of leaves - and repeatedly removes the branch that buys the least purity per leaf it costs. Bigger ccp_alpha = heavier penalty = smaller tree.
On Lumen, ccp_alpha=1e-4 with no depth cap at all lands at 0.635 test AUC - most of the way from memorization (0.509) back toward the depth-5 peak (0.660), with zero manual depth tuning. In practice you sweep alpha with cross-validation (sklearn's cost_complexity_pruning_path hands you the candidate alphas). Whether any alpha can beat 0.660 here is your homework.
Self-studyRegression trees - variance instead of gini3 min▶
Swap the purity score and trees predict numbers: a regression tree audition picks the split that most reduces the variance of the target within the children, and each leaf predicts the mean of its training rows. Same greedy recursion, same overfitting physics - here is the sweep on the 1,306 Lumen buyers:
| max_depth | Test R² | Reading |
|---|---|---|
| 3 | 0.53 | nearly there |
| 4 | 0.54 | the peak |
| 6 | 0.36 | fitting noise fast |
| 10 | 0.12 | mostly noise now |
| None | -0.08 (train 1.00) | memorization goes NEGATIVE |
Note how much sharper the cliff is than in classification - and note what a leaf-predicts-its-mean model can never do: a regression tree's output is piecewise constant, a staircase. Ask it about a spend value beyond anything in training and it answers with the edge leaf's mean - trees cannot extrapolate, not even a little.
Reading the tree 7 min live
The tree's superpower is that you can print it. plot_tree draws the boxes; export_text gives you the same thing as indented if/else lines you can paste into a design doc. Every prediction is a walkable path - which makes trees the white-box animal you reach for when a stakeholder asks "but WHY did it flag this session?". The second readout is feature_importances_: each feature's share of the total impurity reduction, summing to 1. It is genuinely useful and genuinely treacherous - both below.
LiveFeature importance - useful, with caveats3 min▶
Impurity-based importance credits each feature with the purity gain of the splits it powered, normalized to sum to 1. On Lumen it delivers this course's most satisfying receipt: session_hour - the noise feature the generator planted with a true coefficient of zero - lands at the bottom with importance ~0. The tree auditioned it at every node and almost never found it worth a split.
Two caveats before you put importances in a slide. First, they are biased toward high-cardinality features: a feature with many distinct values offers many more candidate thresholds, so it gets more chances to look good by luck. Second, importance is not causation - it says "splitting here purified the training data", not "changing this changes the outcome". Session hour being useless is a modeling fact; email being important does not mean sending more email causes conversions. (Your experimentation course owns that question.)
Self-studyInstability - and the forest-shaped teaser2 min▶
Greedy auditions have a fragile side: change a handful of training rows and a different feature may win the root split, sending the entire tree down a different path - same data story, differently shaped flowchart. sklearn's own docs list this instability as a core limitation of single trees.
The classic fix is to embrace it: train many trees on resampled data and average them, so the reshuffling cancels out. That is a random forest - and it belongs to learn-ensemble-methods, named here so you know where the story continues, not taught here.
LiveWhere trees lose - the linear truth problem2 min▶
Honest leaderboard time. On the Lumen buyers, the best tree regressor (depth 4) scores R² 0.54 - and plain linear regression from session 2 scores 0.59. The tree loses, and it loses for a structural reason: the Lumen generator's spend-to-order relationship is genuinely linear, and a piecewise-constant model can only approximate a ramp with stairs. Every stair tread wastes resolution a straight line gets for free.
When the truth is linear, rectangles approximate a ramp with stairs - remember that sentence in session 9, when choosing between families becomes the whole game. Trees earn their keep on interaction-heavy, threshold-y data; on smooth linear signal, the humble line wins.
Grow it, prune it, read it ★ 18 min · everyone builds
Five moves: sweep depth and reproduce the U-curve, prune with ccp_alpha instead of a depth cap, plot the top of the winning tree, sort the feature importances to catch session_hour, and run the regressor head-to-head against session 2's linear model.
Sweep depth. One-hot only - no scaler, trees do not care. Loop depth over 2, 5, 8, None and print train and test AUC. The gap between them at depth None IS the lesson.
Prune instead. Drop the depth cap, set ccp_alpha=1e-4, and see how close weakest-link pruning gets to the hand-tuned peak.
Plot and read. plot_tree with max_depth=2 keeps the picture human-sized. Walk one path out loud - root to leaf.
Sort the importances. The planted noise feature should sit at the bottom, importance ~0. Enjoy the receipt.
Regressor vs linear. Depth-4 tree on the buyers vs plain LinearRegression - the stairs vs the ramp.
random_state=42 on the tree itself, not just the split - ties inside the audition are broken randomly, and an unseeded tree can pick a different (equally good) split and grow a different shape from identical data.Before session 8 ◐ 25 min
- Sweep
ccp_alpha(trycost_complexity_pruning_pathfor candidates) and either find an alpha that beats depth-5's 0.660 test AUC or conclude, with numbers, that none does. - Sweep
min_samples_leafover 1, 5, 25 at depth None. How much of the memorization does the leaf floor alone prevent? - In 3 sentences: why is the depth-None tree's train AUC of 1.000 a warning, not a win? Write it as if to a stakeholder who just said "but it scored perfectly".
Three questions before you go 🎯 ◐ 90 seconds
1 · Gini vs entropy in practice: what should you actually do?
The two scores are shaped almost identically and usually choose the same splits. Gini is the default because it skips the logarithms, not because it is better - and entropy is not "more accurate", on Lumen or in general. Spend the tuning budget on depth and leaf size instead.
2 · The depth-None tree scored train AUC 1.000 and test AUC 0.509. What happened?
Unchecked, the audition continues until every leaf is pure, and purity on noisy data means carving a private rectangle around each point. Perfect train score, useless test score - the U-curve's right-hand cliff, now measured in AUC.
3 · The depth-4 tree regressor (R² 0.54) lost to plain linear regression (0.59) on the buyers. Why?
Trees never need scaling (each split lives inside one feature), so A is out. The real reason is shape: leaves predict constants, so the tree builds a staircase over a genuinely linear ramp. Match the model to the data's shape - the theme session 9 turns into a method.
What this session covers
Session 7 covers the tree material across all three anchor sources - the StatQuest tree quartet, Ng's C2 W4 module, and sklearn's decision-tree guide. Videos, graded labs, and certificates stay official.