The tradeoff from b4, now with a knob
Last session you controlled overfitting by choosing tree depth - a blunt lever. Regularization is the smooth version: you keep the model expressive but add a penalty term that punishes large, wild coefficients. Turn the penalty up and the fit gets simpler and steadier; turn it down and it is free to chase every wiggle. It is the same bias-variance dial you already understand, now a continuous number you can tune, and it works on almost every model you will meet after this course.
Ridge (L2, shrinks every coefficient) and Lasso (L1, can zero coefficients out entirely). You will sweep the alpha strength dial, plot coefficient paths, and see Lasso automatically discard the useless session_hour feature - the same planted noise from b1.
Bias-variance, and the penalty idea 7 min live
A model that is too simple has high bias - it underfits, missing the real pattern. A model that is too complex has high variance - it overfits, memorising noise. Regularization adds a penalty on complexity to the thing the model minimises, so a coefficient only survives if it earns its keep against that penalty. The fit can no longer chase noise for free. Two flavours: Ridge (L2) shrinks all coefficients smoothly toward zero; Lasso (L1) can push some coefficients to exactly zero, which drops those features entirely.
LiveRidge (L2) vs Lasso (L1)3 min▶
Both add a penalty to the least-squares loss, but they measure "coefficient size" differently, and that one difference changes everything:
- Ridge (L2) penalises the sum of squared coefficients. It shrinks every coefficient smoothly toward zero but never quite to zero - all features stay in the model, just quieter. Great when many features each carry a little signal.
- Lasso (L1) penalises the sum of absolute coefficients. Its geometry lets it push weak coefficients to exactly zero, so it performs feature selection - it decides which columns to keep. Great when you suspect some features are useless (like
session_hour).
Self-studyalpha is the strength dial2 min▶
Both models take one key hyperparameter, alpha - the weight on the penalty:
alpha = 0- no penalty; you are back to plainLinearRegression, free to overfit.- small alpha - a light touch; coefficients barely shrink.
- large alpha - a heavy hand; coefficients shrink hard (Ridge) or get zeroed (Lasso), pushing toward a simpler, higher-bias model.
So alpha is literally the bias-variance dial in numeric form. You pick it the same honest way you picked tree depth in b4 - by held-out performance, which b6 formalises with cross-validation. Never eyeball it on the test set.
Sweep the dial in Python 18 min · everyone builds
We predict order_value among the buyers with numeric plus one-hot features, then compare plain linear, Ridge, and Lasso. On clean data expect all three to land near the full-model R² of ~0.52 from the canon - regularization here mainly stabilises the fit, it does not dramatically improve it. The payoff is watching Lasso zero out the planted noise.
Reload and set up the regression. One-hot the categoricals with pd.get_dummies for now (proper leak-free pipelines come in b9), then split.
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_parquet("lumen_sessions.parquet")
buyers = df[df.converted == 1].copy() # regression lives on the ~1,300 buyers
num = ["prior_30d_spend", "pages_viewed", "session_hour"] # session_hour = planted noise
cat = ["channel", "device", "new_vs_returning", "product_category"]
X = pd.get_dummies(buyers[num + cat], columns=cat, drop_first=True)
y = buyers["order_value"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42)
print(X_train.shape) # rows x features
Fit three models and compare test R². Same fit/predict API; only the penalty differs.
from sklearn.linear_model import LinearRegression, Ridge, Lasso
for name, model in [
("Linear (alpha=0)", LinearRegression()),
("Ridge (alpha=10)", Ridge(alpha=10.0)),
("Lasso (alpha=1)", Lasso(alpha=1.0, max_iter=10000)),
]:
model.fit(X_train, y_train)
print("%-18s test R2 = %.2f" % (name, model.score(X_test, y_test)))
# all three land near ~0.52 - regularization stabilises here, it does not transform
All three sit near the canonical full-model R² ~ 0.52. That is the honest result: on clean, well-behaved data the penalty buys robustness, not a headline jump. The difference shows up in the coefficients, not the score.
Sweep alpha and plot coefficient paths. Watch how each coefficient shrinks as the penalty grows - and watch Lasso drive session_hour to exactly zero.
import matplotlib.pyplot as plt
alphas = np.logspace(-2, 2, 30) # 0.01 -> 100
paths = []
for a in alphas:
m = Lasso(alpha=a, max_iter=10000).fit(X_train, y_train)
paths.append(m.coef_)
paths = np.array(paths)
hour_idx = list(X.columns).index("session_hour")
plt.plot(alphas, paths[:, hour_idx], label="session_hour (noise)")
plt.xscale("log"); plt.axhline(0, color="grey", lw=0.8)
plt.xlabel("alpha"); plt.ylabel("Lasso coefficient"); plt.legend()
plt.show()
# confirm it hits exactly zero
z = Lasso(alpha=1.0, max_iter=10000).fit(X_train, y_train)
print("session_hour coef:", round(z.coef_[hour_idx], 4)) # -> 0.0
Lasso automatically dropped session_hour to zero - the useless noise feature we planted in b1 - so the Lumen team did not have to argue about it in a review. The model made the call: this column carries no signal, keep it out. That is feature selection happening as a side effect of the penalty, and it is why analysts reach for Lasso when a dataset has dozens of maybe-useful columns.
Take the win. Regularization buys robustness and simpler, more defensible models - not magic accuracy.
lin = LinearRegression().fit(X_train, y_train)
lasso = Lasso(alpha=1.0, max_iter=10000).fit(X_train, y_train)
print("features used by Linear:", int((lin.coef_ != 0).sum()))
print("features used by Lasso :", int((lasso.coef_ != 0).sum())) # fewer
# same ~0.52 R2, but Lasso got there with fewer moving parts -> easier to trust + explain
This week ◐ 40 min total
- Sweep Ridge alpha. Fit
Ridgeacrossnp.logspace(-2, 3, 40)and plot test R² vs alpha. Where does the score peak, and where does too much penalty start to underfit? - Find Lasso's zero point. Increase Lasso's alpha until it zeros the most features. Which columns survive longest? (Hint:
prior_30d_spendandproduct_categoryshould be the last standing.) - Explain L1 vs L2 in one sentence - in your own words, why can Lasso zero a coefficient out while Ridge only shrinks it?
- Optional: standardise the features first (Ridge and Lasso are scale-sensitive) and confirm the coefficient paths change. This previews the pipeline discipline of b9.
Three questions before you go 🎯 ◐ 90 seconds
1 · A model with high variance is best described as -
High variance = too complex = overfitting: the model bends to the noise in the training data and fails on new data. High bias is the too-simple, underfitting case.
2 · What does regularization actually do?
Regularization penalises large/complex coefficients so the fit cannot chase noise. On clean data it mainly buys robustness and simplicity, not a guaranteed score jump.
3 · How do Lasso and Ridge differ?
Lasso's L1 penalty can push weak coefficients to exactly zero (feature selection); Ridge's L2 penalty shrinks every coefficient toward zero but keeps them all in the model.
What this session covers
This builder session covers the bias-variance and regularization content of the sources below - on the Lumen dataset - and connects straight to b4's overfitting curve. Cross-validation, the honest way to pick alpha, is b6.
Builder Session 5 cheat sheet · pin this
Ridge(alpha=...). Penalises squared coefficients; shrinks all toward zero, keeps every feature. Good for many small signals.Lasso(alpha=...). Penalises absolute coefficients; can zero features out entirely - feature selection for free.session_hour (the planted noise) - robustness and simplicity, not magic accuracy.