Cross-validation, and how to tune without cheating

In b4 you picked the sweet-spot tree depth by watching a single test curve. That works, but it rests on one random split - shuffle the rows differently and the peak can wobble. Cross-validation gives you a steadier grade by averaging over several splits, and it is the tool that lets you tune hyperparameters honestly, because the test set stays sealed until the end. Think of it as taking the same exam five times with different question sets and reporting your average plus how much you varied.

Split the training data into k equal folds. Each fold takes a turn as the validation set while the other k-1 folds train the model; you do this k times, so every row is validated exactly once. Average the k scores for a steadier estimate, and look at the spread - a wide spread means the score depends heavily on which rows you happened to hold out. Crucially, all of this happens on the training data. You tune on these CV scores and never on the test set, which you spend exactly once, at the very end.

We stay on the b4 regression - predict order_value from prior_30d_spend with a decision tree - so the numbers line up. First a plain 5-fold CV, then GridSearchCV to pick max_depth, then one final look at the sealed test set.

import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split

df = pd.read_parquet("lumen_sessions.parquet")
buyers = df[df.converted == 1].copy()

X = buyers[["prior_30d_spend"]]
y = buyers["order_value"]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42)     # test set: do not touch until the end

from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import cross_val_score

tree = DecisionTreeRegressor(max_depth=4, random_state=0)
scores = cross_val_score(tree, X_train, y_train, cv=5, scoring="r2")

print("fold scores:", scores.round(2))          # e.g. ~0.33 .. ~0.50
print("mean R2 = %.2f  +/- %.2f" % (scores.mean(), scores.std()))
# -> mean ~ 0.40  +/- ~ 0.06

from sklearn.model_selection import GridSearchCV

grid = GridSearchCV(
    DecisionTreeRegressor(random_state=0),
    param_grid={"max_depth": range(1, 13)},
    cv=5, scoring="r2")
grid.fit(X_train, y_train)                       # searches on TRAIN only, via CV

print("best params:", grid.best_params_)         # -> max_depth around 3-4
print("best CV R2 : %.2f" % grid.best_score_)    # -> ~ 0.40
# matches b4's U-curve peak at depth 3 - CV found it without touching the test set

best = grid.best_estimator_
print("final held-out test R2 = %.2f" % best.score(X_test, y_test))
# report THIS number - it is the one you never tuned against

• Run cv=10. Repeat the depth-4 CV with cv=10 and compare the mean and spread to cv=5. Does the mean move much? Does the spread tighten or widen, and why might more folds change it?
• Grid a second hyperparameter. Add min_samples_leaf to the param_grid alongside max_depth and let GridSearchCV search the combined space. What does it pick?
• Confirm the sweet spot. Check that the CV-chosen depth matches b4's U-curve peak (depth ~3). If it differs, explain what the difference tells you.
• Optional: print grid.cv_results_["mean_test_score"] and plot it against depth - you have just drawn b4's validation curve, this time built from cross-validation.

This builder session covers k-fold cross-validation and hyperparameter tuning on the Lumen dataset, building directly on b4's overfitting curve and b5's alpha dial. Classification metrics in depth come next in b7.