Why trees win tables

The claim is not "trees always win". The claim, from the paper that tested it properly: "tree-based models remain state-of-the-art on medium-sized data (~10K samples) even without accounting for their superior speed." That is Grinsztajn, Oyallon and Varoquaux, NeurIPS 2022, across 45 datasets - and the word doing quiet work is medium-sized. Most tables you will ever model - churn lists, credit files, ops logs - live exactly in that range.

Careful scope is not hedging, it is the skill. A claim you can state with its boundary attached survives every meeting; the boosterish version dies at the first counterexample.

• Ensemble - many weak-ish models combined into one strong one. The two great recipes: average independent models (bagging, session 2) or chain corrective models (boosting, session 3).
• Variance - how much your model changes if the training sample changes. Deep single trees are high-variance: perturb the data, get a different tree. Averaging attacks variance.
• Bias - error from the model being too simple to express the truth. Boosting attacks bias, one small correction at a time.
• Holdout - the 100 customers the model never sees while learning. Every honest number in this course is a holdout number; every seductive number is a train number.

The course dataset has six features: days since last order, orders in 90 days, average order value, support tickets, discount share, sessions in 30 days. Watch the three reasons land on it:

• Uninformative features. Average order value turns out to carry almost no churn signal here. A tree simply never splits on it. A neural network spends capacity learning to ignore it, and on 200 training rows there is no capacity to spare.
• Orientation. "Days since last order over 45" is a meaningful statement about a real column. A rotated mixture of days-and-ticket-counts is meaningless - and tabular data is full of such axis-aligned meaning. Trees cut along axes by construction; the paper shows rotating the data destroys the tree advantage, which proves the advantage was the orientation.
• Irregular functions. Churn risk does not glide upward with inactivity - it steps: fine at 30 days, gone at 50. A tree IS a step function. A smooth network has to contort to make a cliff.

A course built on evidence has to show you the newest evidence even when it complicates the title. Two dated facts, both real:

• Benchmarks, 2025: TabPFN, a tabular foundation model published in Nature, "outperforms all previous methods on datasets with up to 10,000 samples by a wide margin" - in 2.8 seconds, against baselines tuned for four hours. Its successor claims wins up to 50,000 rows, though those numbers are vendor-authored and not yet peer-reviewed.
• Competitions, through 2024: among the year's winning solutions, 16 used LightGBM, 13 CatBoost, 8 XGBoost - and not one tabular foundation model appears. In the wild, where data is messy and stakes are real, the trees still cash the checks.

Both facts, both dated, no side picked before the evidence settles. Session 6 puts TabPFN on the same leaderboard as everything else and lets the measured numbers argue.

Leo Breiman named the mechanism in 1996, and his sentence is still the cleanest statement in the field: "The vital element is the instability of the prediction method. If perturbing the learning set can cause significant changes in the predictor constructed, then bagging can improve accuracy."

Deep trees are spectacularly unstable - move ten customers and the whole upper tree can reshape. That is exactly why averaging many of them works: each tree's private, sample-specific mistakes point in different directions, and directions that disagree cancel. Instability, which made one tree untrustworthy, is the raw material that makes a forest strong.

Every named tool from here - random forest, XGBoost, LightGBM, CatBoost - is one of these two recipes plus engineering. Keep the two-row map and nothing in the field will surprise you.

Three things in that chart carry the whole course:

• The forest's plus-ten. Same trees, same data - just sixty of them, each on a bootstrap sample with random feature subsets, voting. Averaging is worth ten points here. That is session 2's entire argument, measured.
• The tie at 68. Careless boosting - deep trees, fast learning rate - reaches training accuracy of 100 percent and STILL only ties the forest on customers it never saw. The 32-point train-holdout gap is the most instructive number in the course.
• The falling bar. Six hundred rounds of the same careless recipe lands at 70 - below the tuned 73. More trees is not a strategy. Session 6 makes you feel that with a button.