Linear regression, properly

A residual is the vertical gap between a point and the line: actual minus predicted. Sum the raw residuals and positives cancel negatives - a terrible line can score zero. So we square first. Squaring does two jobs at once: it makes every miss count against you, and it makes big misses count disproportionately - a $20 miss costs 4x a $10 miss, not 2x. The line that wins is the one whose worst embarrassments are smallest.

There is a quieter reason too: the squared loss is smooth, so calculus can find the minimum exactly, in one closed-form step. That is why LinearRegression().fit() returns instantly with no learning rate, no iterations, no randomness. Session 4 shows the iterative alternative (gradient descent) that logistic regression needs - linear regression is the lucky one that does not.

Fit order value on prior 30-day spend alone (Lumen buyers, n=1,306) and the model learns two numbers: slope ~0.44 and intercept ~$42. Say them like a human: "every extra dollar a customer spent with us last month is worth about 44 cents on today's order, and a customer with no history lands around $42." That sentence is the entire model - and it is a sentence a CMO can act on.

Quality check: this one-feature model scores R² ~0.43 with RMSE ~$30 on the held-out test buyers. The floor from session 1 - just predict the mean order value - has RMSE ~$40. One feature bought you $10 of typical error back. Keep both numbers in the same breath; a metric without its baseline is decoration.

R² is the share of variance explained: how much of the spread in order values the model accounts for, compared to just predicting the mean. R² 0.43 means the model explains ~43% of why orders differ - and 57% remains unexplained noise or missing features. It is NOT accuracy, it is not "the model is 43% right", and on its own it does not tell you the dollar size of your errors - that is RMSE's job.

The trio you already met in intro-ml, one line each: R² for "how much of the pattern did we catch" (unitless, baseline 0), RMSE for "typical miss in dollars, big misses amplified", MAE for "typical miss in dollars, everyone counted equally". Report R² to modelers, RMSE or MAE to the business.

sklearn's OLS guide (section 1.1.1) carries one standing warning: least squares assumes your features are reasonably independent. When two features are strongly correlated - say you added both "prior 30-day spend" and "prior 90-day spend" - the design matrix becomes nearly singular and the fit can split credit between the twins almost arbitrarily. Predictions stay fine; the coefficients become unstable, swinging wildly with tiny changes in the data.

The practical rule: the moment you care about reading coefficients (and today you do), check correlations among your features first. If two features tell the same story, keep one, or accept that their individual coefficients are not individually meaningful. Ridge regression (end of Part 3) is the other standard fix - it calms the swinging at the price of a little bias.

A line cannot multiply "skincare" by a coefficient. pd.get_dummies(..., drop_first=True) turns each categorical into 0/1 columns, dropping one level per feature. The dropped level becomes the reference category baked into the intercept, and every dummy coefficient reads as "dollars relative to that reference". If affiliate is the dropped channel, the email coefficient means "email buyers order this many dollars more (or less) than affiliate buyers, all else equal".

Why drop one? With all levels present plus an intercept, the columns sum to a constant - perfect multicollinearity, the pathological case of the warning above. StatQuest calls this whole encoded table the design matrix; sklearn just calls it X. Inside pipelines we use OneHotEncoder(handle_unknown="ignore") instead, which keeps all levels but pairs with regularized models - today's get_dummies path is the interpretability convention this course uses whenever we read coefficients.

Full model: RMSE ~$25, MAE ~$20. That gap is not an accident - RMSE squares before averaging, so a handful of badly-missed orders dominate it, while MAE treats a $50 miss as exactly five $10 misses. Lumen order values are right-skewed (mean $82, SD $41 on buyers - a long tail of big baskets), and the model misses biggest on exactly those tail orders. Skewed target, so RMSE above MAE, every time.

The gap itself is diagnostic: if RMSE towers over MAE, your errors have a heavy tail - go look at which rows the model misses worst before reaching for a fancier algorithm.

Sharp-eyed alumni will notice: intro-ml's b8 page printed ~0.52 / $28 for this same full model. Re-validated in July 2026, the same code in the current environment produces 0.59 / $25 - the numpy random stream behind the generator drifted across library versions, so the "same" dataset is no longer bit-identical. The numbers on this page are the ones that reproduce today, and a fix for the intro-ml page is flagged.

A degree-12 polynomial has enough freedom to pass near every one of 50 training points - and it uses that freedom by swinging violently between them. Between two nearby points it can dip or spike hundreds of dollars; the training loss never sees those in-between regions, so nothing stops it. Worse, past the edge of the training data a high-degree polynomial's leading term takes over and the curve rockets toward plus or minus infinity - and the test set has points near those edges.

That is why test R² is not merely "bad" but -798,595: R² of 0 means "as good as predicting the mean", negative means worse, and this is six orders of magnitude worse. One test buyer sitting near an edge gets a prediction thousands of dollars wrong, and squared errors do the rest. The overfitting tell from session 1 - train up, test down - at maximum volume.

Regularization adds a penalty on the weights to the loss, so the fit pays for flexibility it does not earn. Ridge (L2, sklearn 1.1.2) penalizes squared weights: everything shrinks toward zero, nothing reaches it - the wild polynomial coefficients get pulled back to sane sizes. Lasso (L1, sklearn 1.1.3) penalizes absolute weights: it drives some coefficients exactly to zero, so it doubles as automatic feature selection. Elastic-net (1.1.5) mixes both. One dial rules them all: alpha - higher means a shorter leash, alpha 0 means plain least squares.

You met the concept in intro-ml b5; the API lives in the sklearn sections above; the intuition (why L1 gives corners that zero things out while L2 gives a circle that only shrinks) is beautifully told in StatQuest's Regularization Parts 1, 2, 2.5 and 3. Tonight's homework has you point Ridge at the degree-6 explosion and watch it get tamed.