Where session 2 left you
The forest reached 68 percent holdout by averaging sixty independent trees - variance canceled, and the score stopped improving. Averaging can only cancel noise; it cannot add signal the trees never found. Adding missing signal is a different job with a different tool, and the tool is boosting.
★ What you walk out with tonight. Boosting understood as gradient descent in function space - not as a slogan but as a loop you can narrate. A live booster you drive yourself, whose curves expose both the power and the trap. And the tuning law from the standard reference, verified by your own clicks.
LiveThe game, in four sentences3 min▶
Start with the dumbest possible model: predict the base churn rate for everyone. Compute each customer's residual - the gap between what actually happened and what you currently predict. Fit a small tree to those residuals: not to churn, to the mistakes. Add that tree's output to the model, scaled down by a learning rate, and repeat.
Every round, the model asks one question: where am I still wrong, and in which direction? Then it takes a small step that way. If that sounds like gradient descent, that is because it is - which is the part Friedman made precise.
The intuition fits a newsroom: the first draft states the base facts, each editing pass fixes only what the previous pass still got wrong, and no single pass is brilliant. The article converges because every pass targets the current residual error, not because any editor is a genius.
Gradient descent, in function space
8 min liveFriedman's 2001 paper opens with the reframe that organizes everything: "Function estimation/approximation is viewed from the perspective of numerical optimization in function space, rather than parameter space." One picture makes it concrete.
LiveWhy the trees must be weak4 min▶
Session 2's forest wanted deep, unstable trees - averaging needed their disagreement. Boosting wants the opposite: shallow trees, deliberately too weak to solve the problem alone. Depth 3 is the course default; depth 1 stumps genuinely work too.
The reason is the division of labor. In a forest, each tree attempts the whole problem and the vote cleans up. In boosting, no tree attempts the whole problem - each contributes one small correction, and the SUM is the model. A deep tree in a boosting loop grabs too much at once, including the noise, and the loop then spends rounds polishing that noise. Tonight you will watch exactly that happen at depth 6.
The stump surprise, measured. Depth-1 stumps - single splits, the weakest tree there is - reach 68 percent holdout on the course data after 200 rounds. Sixty deep bagged trees also reached 68. Two hundred one-question trees tie a forest of experts, because the chain lets each question build on every answer before it.
LiveThe learning rate: the one law worth memorizing4 min▶
The learning rate v scales every tree's contribution down before it joins the model. Small v means each round fixes only a fraction of what it could - which sounds wasteful and is in fact the single best-established tuning law in the field. The Elements of Statistical Learning, section 10.12.1, states it plainly:
Small steps, many rounds, stop when the validation curve says stop. ESL is equally clear about the bill: "smaller values of v give rise to larger values of M, and computation is proportional to the latter." You pay in trees for what you gain in generalization - tonight's simulator lets you feel both sides of that trade in about ninety seconds.
Self-studyWhat the Newton detail buys, for the curious3 min▶
The engine on this page does one refinement beyond plain residual fitting: each leaf's value is the residual sum divided by the sum of p(1-p) - a Newton step using the second derivative of the loss, not just the first. Customers the model is already confident about (p near 0 or 1) contribute tiny p(1-p), so uncertain customers dominate the leaf value.
Keep that detail warm: it is the direct ancestor of XGBoost's second-order objective in session 4, and knowing it came from somewhere - the LogitBoost line of work in 2000 - is what separates reading the XGBoost paper from being impressed by it.
Watch it learn - and watch it memorize
★ 22 min · everyone buildsThe widget below runs the loop from Part 1 for real: every press of the button fits genuine trees to the 300-customer Mango Lane sample in your browser, and the two curves - train in olive, holdout in gold - are measured after every single tree.
Defaults first: depth 3, learning rate 0.1. Press +10 trees a few times. Both curves climb together out of the low 60s. This is the healthy phase: every correction the trees find in the training data is real signal that also holds on the holdout.
Press Run 200 rounds and read the two endings. Train marches to 100 percent. Holdout tops out at 73 - and if you watch the gold curve closely, it PEAKS at 77 near round 48 and drifts back to 73 by round 200. Nothing is scripted there: that little hump is real early-stopping evidence, generated by real trees, in front of you. ESL said choose M by early stopping; your own screen just showed you why.
Reset. Now the careless recipe: depth 6, learning rate 0.5. Run 200. Train hits 100 almost immediately. Holdout stalls at 68 - a 32-point gap, the widest in the course. Deep trees plus big steps grabbed the noise in round after round, and the holdout curve is flat because there was no signal left to find, only noise to polish.
Compare the two configurations you just ran. Same data, same loop, same number of trees. Shallow-and-slow: 73. Deep-and-fast: 68. The tuned booster beats the careless one by five points using nothing but restraint - the ESL law, verified by your own two runs.
Try depth 1. Two hundred stumps land at 68 holdout with a train score of only 72 - the smallest gap on the board. Underfitting is honest: what stumps learn is real, there is just not enough of it. Depth is a real dial with real ends, and 3 sits between them for a reason.
The sentence to keep from tonight. Train accuracy is what the model can memorize; holdout accuracy is what it actually learned; and every tuning decision in boosting is about refusing memorization politely enough that learning continues.
Every boosting curve tells the same three-act story
5 min liveYou will read hundreds of these curves in a working life. They all have the same anatomy, and the three acts are worth naming once, precisely.
LiveWhat each act means operationally3 min▶
- Act 1, rising together: the trees are extracting shared signal. More rounds are free money. If your budget forces a stop here, lower v was too low or the trees too weak.
- Act 2, the crest: M*, the round count you actually want. Production practice is to hold out a validation slice and let the library stop automatically when the curve flattens - the standard reference's advice, mechanized.
- Act 3, the divergence: every further round transfers noise from the training sample into the model. The train curve rewards you for it, which is exactly why the train curve must never be the one you watch.
Mango Lane's contractor-era model was retrained monthly "with more rounds each time, since the accuracy kept improving". The accuracy that kept improving was training accuracy. The voucher list quietly got worse for three months before anyone checked a holdout - the exact failure this session's second run performs in ninety seconds.
Before session 4
30 min- Run the widget's four corners: depth 1 and 6, learning rate 0.1 and 0.5, 200 rounds each. Write the four holdout scores in a 2x2 grid. You now have a personally-measured map of the two most important dials in gradient boosting.
- Find M* for the default config. Reset, then +10 trees repeatedly, noting the holdout after each press. Write down where the peak came for you and what the score was.
- Narrate the loop to someone - the four-arrow napkin version. If you stall at "fit a tree to the residuals", re-open Part 1's diagram; that arrow is the whole idea.
- Read ESL section 10.12.1 (free PDF at the authors' site) - two pages. Underline the v < 0.1 sentence you already verified by clicking.
- Optional: explain to a colleague why "training accuracy 100 percent" made this session's audience wince, not applaud.
Three questions
Clear all three and session 3 gets stamped in your passport.
1 · In the boosting loop, what does each new tree actually get fitted to?
That substitution is the entire Friedman insight: fitting to residuals IS gradient descent, taken in function space, with a tree as the step. Bootstrap-and-vote is the other recipe - session 2's - and mixing up which loop you are in is the classic ensemble mistake.
2 · Depth 6 with learning rate 0.5 reached train accuracy 100 but holdout 68, while depth 3 with 0.1 reached the same train 100 but holdout 73. Why does the careless config lose?
Both configs memorized the training set - the same 100 up top. The five-point holdout difference is entirely about WHAT ELSE got learned on the way: deep trees and big steps hoover up noise; shallow trees and small steps mostly cannot. The ESL law, measured live.
3 · The holdout curve at default settings peaked at 77 around round 48 and drifted to 73 by round 200. What is the correct operational response?
The crest is M*. ESL's exact strategy: set v small, then choose M by early stopping, monitoring prediction risk on a validation sample. Act 3 rounds cost compute and return damage - and shipping a model other than the one you measured is its own, worse mistake.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 3 cheat sheet · pin this
Tonight's engine was a teaching implementation - honest but minimal. Session 4 is the industrial version: XGBoost's regularized objective, the second-order trick and where it actually came from, the four knobs that matter, and the famous 17-of-29 statistic quoted the way the paper wrote it rather than the way conference talks repeat it.