A prediction is only useful if you can say how wrong it is
In b7 you judged a classifier - a yes/no machine - with precision, recall, and ROC-AUC. Regression is different: the model predicts order_value, a dollar amount, and "right or wrong" is no longer a clean split. Instead you measure the size of the miss. This session turns that miss into four numbers, then translates the best one back into the language a CFO actually cares about: dollars off, on average.
MAE, MSE, RMSE, and R-squared 6 min live
Every regression score starts from one quantity: the residual, the gap between what the model predicted and what actually happened. Each metric summarises those residuals a different way, and the differences are the whole point - they are not interchangeable.
LiveRMSE vs MAE - two honest answers to "how far off?"3 min▶
Both measure the typical miss in dollars, but they weight the misses differently:
- MAE (mean absolute error) - the average size of the miss, in dollars, treating a $10 miss and a $100 miss proportionally. It is the number a stakeholder reads as "we are off by about this much".
- MSE (mean squared error) - the average of the squared misses. Squaring blows up big errors, so MSE reacts strongly to a few large misses. Its units are dollars-squared, which nobody can picture.
- RMSE (root mean squared error) - the square root of MSE, which pulls the units back to dollars. Because the squaring happened first, RMSE still punishes big misses harder than MAE does.
Self-studyR-squared and the predict-the-mean baseline3 min▶
R-squared answers a different question: not "how many dollars off" but "how much better than doing nothing?" The do-nothing model is predicting the mean order value for every row - it ignores all features. R-squared is the share of the variance in order value that your model explains on top of that baseline.
- R-squared = 0 means you are no better than predicting the mean.
- R-squared = 1 would mean perfect prediction (never happens on real, noisy money).
- The predict-the-mean baseline has RMSE equal to the standard deviation of the target - on Lumen converters, about $41.
So R-squared and RMSE are two views of the same fit: R-squared is unit-free and comparable across problems, RMSE is in dollars and lands in a room. You will report both, but lead with dollars.
From predict-the-mean to a real model 22 min · everyone builds
We predict order_value for the sessions that converted, and we build the score up in stages so every number has a reference point. Your figures will land near the course canon (a real sklearn fit on seed 42); teach the relationships, not a false fourth decimal.
Reload and set up the regression. Order value only makes sense for buyers, so we model the converter subset. Same train/test discipline as every session.
import numpy as np, pandas as pd
from sklearn.model_selection import train_test_split
df = pd.read_parquet("lumen_sessions.parquet")
buyers = df[df.converted == 1].copy() # regression is on converters only
y = buyers["order_value"]
X = buyers[["prior_30d_spend", "product_category",
"channel", "new_vs_returning"]]
X_tr, X_te, y_tr, y_te = train_test_split(
X, y, test_size=0.25, random_state=42)
print(X_tr.shape, X_te.shape, "target SD $%.0f" % y.std()) # SD ~ $41
Set the baseline: predict the mean. Before any model earns praise, know what doing nothing scores. The mean is the honest floor.
from sklearn.dummy import DummyRegressor
from sklearn.metrics import mean_squared_error, r2_score
dummy = DummyRegressor(strategy="mean")
dummy.fit(X_tr[["prior_30d_spend"]], y_tr)
pred0 = dummy.predict(X_te[["prior_30d_spend"]])
rmse0 = mean_squared_error(y_te, pred0) ** 0.5
print("baseline RMSE $%.0f" % rmse0) # ~ $41 (= the SD)
print("baseline R2 %.2f" % r2_score(y_te, pred0)) # 0.00
RMSE ~$41 and R-squared 0. That $41 is the standard deviation of order value - the baseline error is literally how much order value bounces around on its own. Every model from here has to beat it.
One feature: prior spend. The statistics course found prior spend correlates with order value at r ~ 0.6. A single-feature linear model should turn that into real explained variance.
from sklearn.linear_model import LinearRegression
reg1 = LinearRegression()
reg1.fit(X_tr[["prior_30d_spend"]], y_tr)
p1 = reg1.predict(X_te[["prior_30d_spend"]])
print("R2 %.2f" % r2_score(y_te, p1)) # ~ 0.40
print("RMSE $%.0f" % (mean_squared_error(y_te, p1) ** 0.5)) # ~ $32
R-squared ~0.40, RMSE ~$32 - already well below the $41 baseline. Note r ~ 0.6 means r-squared ~0.36 on the full data, and a touch higher on the fitted converter subset. One feature, one clear win.
The full model. Add product category, channel, and new-vs-returning via one-hot encoding. More honest signal should push the error down further.
from sklearn.metrics import mean_absolute_error
Xtr = pd.get_dummies(X_tr, drop_first=True)
Xte = pd.get_dummies(X_te, drop_first=True).reindex(
columns=Xtr.columns, fill_value=0) # align test columns to train
reg = LinearRegression().fit(Xtr, y_tr)
p = reg.predict(Xte)
print("R2 %.2f" % r2_score(y_te, p)) # ~ 0.52
print("RMSE $%.0f" % (mean_squared_error(y_te, p) ** 0.5)) # ~ $28
print("MAE $%.0f" % mean_absolute_error(y_te, p)) # ~ $20
R-squared ~0.52, RMSE ~$28, MAE ~$20. Notice MAE < RMSE - always true, and the gap is wide here because the right-skewed big orders inflate RMSE. The everyday miss is about $20; the big-order risk is what pushes RMSE to $28.
Look at the residuals. A single number hides where the model struggles. Plot residuals against the prediction to see the skew with your own eyes.
import matplotlib.pyplot as plt
resid = y_te - p
plt.scatter(p, resid, s=6, alpha=0.3)
plt.axhline(0, color="black")
plt.xlabel("predicted order_value ($)")
plt.ylabel("residual (actual - predicted)")
plt.title("Residuals fan out on the big orders")
plt.show()
When the Lumen analyst took this to the CFO, she did not say "R-squared 0.52". She said "our order-value model is off by about $28 on average, versus $41 if we just guessed the average order" - and the room understood it instantly. Dollars land; R-squared does not. Keep R-squared for the model card and lead every conversation with the RMSE-versus-baseline dollar figure.
This week ◐ 40 min total
- Compute the metrics by hand. Take the first 20 test predictions, form the residuals, and calculate MAE, RMSE, and R-squared yourself with numpy - then confirm they match scikit-learn. Owning the formula beats trusting the import.
- Drop the biggest orders. Refit and re-score with the top 1% of order values removed. Watch RMSE fall more than MAE does, and explain why in one sentence.
- Explain the gap. Write why MAE < RMSE for this target, referring to the right skew and the squaring step. Two sentences, plain English.
- Optional: report your final model as one sentence a CFO would accept - dollars first, baseline named, no jargon.
Three questions before you go 🎯 ◐ 90 seconds
1 · How does RMSE differ from MAE?
RMSE squares the residuals before averaging, so big misses count disproportionately, then the square root returns it to dollars. MAE weights every miss proportionally. On a right-skewed target RMSE is larger than MAE, never smaller.
2 · What does the predict-the-mean baseline score on Lumen converters?
Predicting the mean explains none of the variance, so R-squared is 0 by definition, and its RMSE equals the standard deviation of the target - about $41 here. That is the floor every real model has to beat.
3 · Your full model reaches R-squared ~0.52. What does that mean?
R-squared is the share of variance explained, not an accuracy or an exact-hit rate. ~0.52 means roughly half the variation in order value is captured - a solid, honest lift over the predict-the-mean baseline, matched by the RMSE dropping from $41 to $28.
What this session covers
This builder session covers regression evaluation - the metrics you use to judge any number-predictor - on the Lumen order-value target, and pairs with the classification metrics of b7.