Low training error is not the goal - low test error is
A beginner's instinct is to push training error as low as possible. But a model that fits the training data perfectly has usually just memorised its noise, and it falls apart on new data. The honest question is never "how well does it fit what it has seen?" - it is "how well does it do on what it has not?" The gap between those two answers is overfitting, and today you will watch it grow, live, one tree-depth at a time.
The U-curve, by hand 7 min live
We fit a DecisionTreeRegressor on Lumen order_value at rising max_depth and record two scores each time: train R² and test R². Train error keeps falling as depth rises - a deeper tree can always carve the training data finer. Test error is the honest one: it falls while the model is learning real signal, then rises once the model starts memorising noise. Where test error bottoms out is the sweet spot. This is the exact experiment in the live tool below - turn the dial and watch it happen.
LiveReading the gap3 min▶
Diagnose a model from just two numbers - train score and test score:
- Both low (train 0.31, test 0.22 at depth 1) - underfitting. The model is too simple to capture the signal; add complexity.
- Both decent and close (train 0.46, test 0.36 at depth 3) - the sweet spot. This generalises.
- Train high, test low or negative (train 0.84, test negative at depth 15) - overfitting. The model memorised the training rows; the gap is the tell.
Self-studyBias vs variance3 min▶
The U-curve has a name for each side. A too-simple model has high bias: it makes systematic errors because it cannot represent the real pattern (the underfit, left side). A too-complex model has high variance: it swings wildly to fit whatever noise happens to be in the training sample (the overfit, right side).
You cannot drive both to zero - lowering one tends to raise the other. The goal is not perfection on either side but the minimum of their sum, which is exactly where the test-error U bottoms out. The formal treatment and how regularization lets you tune this trade-off deliberately is the b5 story; today you just need to feel the shape.
Fit rising depths and print the U-curve 22 min · everyone builds
We reload Lumen, take the converters for an order_value regression, split, then loop tree depth and record train and test R². Your table must match the course canon - it was fit with a real DecisionTreeRegressor, seed 42.
Reload and set up the regression. Order value only exists for buyers, so we regress on the converter subset - one feature, so the depth effect is clean and visible.
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_parquet("lumen_sessions.parquet")
buyers = df[df.converted == 1] # order_value is meaningful for buyers
X = buyers[["prior_30d_spend"]]
y = buyers["order_value"]
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.25, random_state=42)
Loop the depth, record both scores. Fit a fresh tree at each depth and keep the train and test R².
from sklearn.tree import DecisionTreeRegressor
rows = []
for depth in [1, 2, 3, 5, 8, 15]:
tree = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(X_tr, y_tr)
rows.append((depth,
round(tree.score(X_tr, y_tr), 2), # train R^2
round(tree.score(X_te, y_te), 2))) # test R^2
for d, tr, te in rows:
print("depth %2d | train R2 %5.2f | test R2 %5.2f" % (d, tr, te))
Read the table - it must match canon. This is the U in numbers.
| max_depth | Train R² | Test R² | Diagnosis |
|---|---|---|---|
| 1 | 0.31 | 0.22 | underfit - too simple, both low |
| 2 | 0.40 | 0.31 | improving |
| 3 | 0.46 | 0.36 | sweet spot - test R² peaks |
| 5 | 0.51 | 0.29 | starting to overfit |
| 8 | 0.62 | 0.15 | overfit - gap widening |
| 15 | 0.84 | negative | badly overfit - worse than the mean |
Train R² climbs the whole way to 0.84. Test R² peaks at depth 3, then falls, and by depth 15 it is negative - the deep tree has memorised the training rows so thoroughly that on new data it does worse than just predicting the average order value.
Plot both curves and pick depth 3. Seeing the U beats reading it. This draws the exact picture from Part 1 on your own numbers.
import matplotlib.pyplot as plt
depths = [r[0] for r in rows]
plt.plot(depths, [r[1] for r in rows], marker="o", label="train R2")
plt.plot(depths, [r[2] for r in rows], marker="o", label="test R2")
plt.axvline(3, linestyle="--") # the sweet spot
plt.xlabel("max_depth"); plt.ylabel("R2"); plt.legend(); plt.show()
A pricing team crowned a deep gradient-boosted model because it "won" on last month's data - it explained almost every historical order. The next month it lost badly to a far simpler model. It had not learned how customers behave; it had memorised last month's specific rows, right down to the noise. The depth-15 negative test R² above is that story in miniature - and the reason you always pick complexity by the held-out score, never the training one.
This week ◐ 40 min total
- Rerun on classification. Swap in a
DecisionTreeClassifieronconvertedand record ROC-AUC (b3's honest metric) per depth instead of R². Does the same U appear? - Find the turn. Identify the depth where your test score first starts falling - that is your model's edge of overfitting. Note it; it is the depth you would ship.
- Explain the negative. Write one sentence explaining how a test R² can be negative at depth 15 - what does R² < 0 mean about the model versus simply predicting the mean?
Three questions before you go 🎯 ◐ 90 seconds
1 · What is the signature of overfitting?
Overfitting means the model has learned the training rows (including noise) but fails to generalise: train R² keeps rising while test R² falls. The gap between the two is the tell - at depth 15, train 0.84 versus a negative test R².
2 · At depth 1 the tree scores train R² 0.31 and test R² 0.22 - both low. What is that?
When both scores are low, the model is too simple to represent the real pattern - high bias, underfitting. The fix is more complexity (more depth), which moves you toward the sweet spot at depth 3.
3 · How do you honestly pick the tree depth?
Training R² always favours the deepest tree, so it is useless for choosing. You pick by performance on data the model did not train on - the test set, or better still cross-validation (b6). Here the test R² peaks at depth 3.
What this session covers
This builder session draws the U-shaped test-error curve on Lumen order value and teaches the train/test gap as the overfitting diagnosis. The formal bias-variance decomposition and regularization as the tuning lever come in b5.