Random forests

A random forest is four decisions, and you already understand three of them from session 1:

• Bootstrap sample per tree - each tree trains on 200 customers drawn WITH replacement from the 200. Different tree, different world.
• Random feature subset per split - at every node, a tree may only consider a random handful of the six features. This is the part that is actually new, and the part that earns the name.
• Deep trees, on purpose - the instability that ruined the single tree is the fuel here. Do not prune the very thing you are about to average away.
• Vote - sixty churn calls per customer, majority wins.

Everything in the next forty minutes is one of these four decisions examined properly.

Before class, try saying the recipe out loud without the page. The order matters: bootstrap gives each tree its own sample, feature subsetting gives each tree its own perspective, depth keeps each tree opinionated, and the vote turns sixty opinionated witnesses into one calm verdict. If you can say why each step exists, session 3's boosting - which deletes the vote and replaces it with a chain - will land in minutes.

To bootstrap the Mango Lane training set, draw a customer at random from the 200, write it down, put it back, and repeat 200 times. Some customers appear twice or three times; others never appear at all. Do this sixty times and you have sixty overlapping-but-different training sets - sixty perturbed worlds.

Now recall session 1: a deep tree is spectacularly sensitive to its sample - move ten customers and the upper tree reshapes. Bagging turns that sensitivity into diversity. Sixty worlds produce sixty genuinely different trees, and that is why the recipe says deep trees, unpruned: instability is the fuel. Tame the trees and you starve the average.

Each deep tree's error has two parts: the signal it caught, and private mistakes tied to its particular bootstrap sample - which duplicates it saw, which noise-labeled customers it happened to memorize. The signal part is shared across trees; the private part is independent-ish, pointing in different directions tree by tree. Average sixty of them and the shared signal survives while the disagreements cancel toward zero. That is variance reduction, in one sentence.

The same logic says when bagging is useless. Bag sixty logistic regressions and each fit lands on nearly the same coefficients - it is a stable method, exactly the case Breiman's sentence excludes. Sixty copies of the SAME mistake do not cancel; averaging identical errors changes nothing. The cure only works on the disease.

Notice the train number went UP too, 75 to 88.5 - the forest is not weaker on the training data, it is stronger everywhere, because sixty perspectives recover signal any single perturbed tree missed. And the train-holdout gap shrank from 17 points to about 20 on a much higher base - the noise-chasing got diluted. Session 3's engine fits this exact forest live so you can watch both numbers move.

On this data, daysSinceOrder is the loudest feature - you found that yourself in session 1's hand-split. Left unconstrained, every one of the sixty trees grabs daysSinceOrder at the root, because it wins the split search in nearly every bootstrap world. Sixty trees, one skeleton: the forest degenerates into sixty near-copies, their votes redundant, and the averaging math from part 1 quietly dies - correlated errors do not cancel.

Force each split to choose from a random handful of features and some trees must open with tickets, or sessions30d, or discountShare. Those trees carve the customers along genuinely different boundaries, make genuinely different private mistakes - and the vote becomes what it pretends to be: independent witnesses, not one witness with sixty microphones.

Breiman measured what this buys: random feature selection "yields error rates that compare favorably to Adaboost... but are more robust with respect to noise." Hold that phrase - our lab data has 10 percent label noise by construction, and robustness to noise is exactly why the forest handles it with grace.

The lineage matters, and Breiman himself is careful about it: his 2001 paper credits Ho 1998 (the random subspace method) and Amit & Geman 1997 as precursors of the random-feature idea. So when someone tells you "random forest is just bagged trees", you now hold two corrections: it is bagging PLUS per-node feature subsetting, and the subsetting idea has its own family tree. Precision about credit is the same muscle as precision about scope - session 1 taught you the second, this is the first.

Breiman's claim is bold and verbatim: "using the out-of-bag error estimate removes the need for a set aside test set", and he reports the OOB estimates are "unbiased". For every customer, some trees never saw her - their votes are honest tests, not recall. Aggregate those honest votes across all 200 training customers and you have a generalization estimate that cost you nothing.

The caveat you must carry: OOB overestimates error early in training. With only a handful of trees grown, each customer's OOB committee is tiny - two or three trees instead of twenty - and a tiny committee votes badly. The unbiasedness arrives once enough trees are grown, not at tree five. Watch a forest's OOB curve and you will see it start pessimistic and settle as the committee fills out.

OOB is forest-internal - it validates a random forest against resamples of its own training data. Two situations where that is not enough:

• Time-based splits. Churn is a future-prediction problem: in production you train on last quarter and predict next quarter. Bootstrap resampling shuffles time away, so OOB cannot tell you whether the model survives the drift between quarters. A holdout cut by date can.
• Comparing across model families. Next session's booster has no bootstrap and therefore no OOB. To put a forest, a booster and a logistic regression on one honest leaderboard, they must all face the SAME unseen customers - which is exactly why this course keeps its 100-customer holdout even though tonight's model comes with free validation built in.

So the working rule: OOB to monitor and tune the forest, the true holdout to judge it against everything else.