From "what we minimize" to "how"
In session 3 you learned that logistic regression picks the weights that minimize log-loss - and we waved a hand at how it finds them. Today the hand-waving stops. Gradient descent is the how, and it is almost embarrassingly simple: check which way is downhill, take a step, repeat. Every solver in scikit-learn, and every optimizer that trains every neural network on Earth, is a variation on this one loop. Learn it once and a whole shelf of documentation stops being intimidating.
The loss surface 7 min live
Pick any pair of weights and the loss function hands back one number: how wrong the model is with those weights. Do that for every possible pair and you get a surface - a landscape where altitude is wrongness. Training is finding the lowest point in that landscape. For log-loss (and for MSE in linear regression) the landscape is a convex bowl: one valley, one global minimum, no fake bottoms to get stuck in. That single geometric fact is why training a logistic model is reliable and reproducible.
LiveWhat a gradient is - the slope in every direction at once4 min▶
In one dimension the slope is a single number: uphill or downhill, and how steeply. With many weights, the gradient is just the slope measured along every weight at once - a vector that points in the steepest uphill direction. Walk against it and you are heading downhill as fast as locally possible. That is the whole trick: the model never sees the landscape, it only ever feels the tilt under its feet.
How do we compute the tilt of a loss that runs through a sigmoid? The chain rule: a chain of effects multiplies. How loss changes with the prediction, times how the prediction changes with the score, times how the score changes with the weight. For log-loss through a sigmoid the chain collapses to something almost rude in its simplicity: the gradient is driven by (p - y) - prediction minus truth. Wrong by a lot, pushed by a lot.
LiveWhy convexity matters - one valley vs a bumpy range3 min▶
Session 3's callback: why log-loss instead of just squaring the errors on the sigmoid's output? Because MSE pushed through a sigmoid is not convex - the surface ripples, with local dents where a descending ball can park and honestly believe it is done. Log-loss on a linear score is provably one smooth bowl: wherever you start, walking downhill ends at the same global minimum.
This is why two runs of LogisticRegression on the same data give the same coefficients, and why the solver can tell you it "converged" with a straight face. Convexity is not a math nicety - it is the difference between "training works" and "training sometimes works".
The step rule and its one knob 9 min live
The entire algorithm is one line: w := w - lr * gradient. Current weights, minus the uphill direction scaled by the learning rate. Repeat until the loss stops falling. Everything that can go wrong with gradient descent goes wrong in that one multiplication - which makes the learning rate the single most consequential knob you will meet today.
LiveThe learning rate - the only knob that can ruin everything3 min▶
Too small: the ball crawls, and you wait thousands of iterations for what should take hundreds. Too big: each stride flies past the valley floor and lands higher on the opposite wall - the loss climbs instead of falling and eventually overflows to nan. There is no partial credit; a diverging run is garbage.
The convergence check is beautifully low-tech: plot the loss every few iterations. Falling steadily? Converging. Flat? Done - stop. Rising or jagged? Cut the learning rate and restart. When you tune, try values several times apart (0.005, 0.05, 0.5) rather than inching - the failure modes are orders-of-magnitude apart, so the search should be too.
LiveFeature scaling makes the bowl round3 min▶
On Lumen, prior_30d_spend runs 0-400 while pages_viewed runs 1-15. Feed those in raw and the bowl is not round - it is a stretched canyon, thousands of times longer in one direction than the other. The steepest-downhill direction in a canyon points at the near wall, not along the valley - so descent zigzags wall to wall, making tiny real progress per step. One learning rate cannot fit both axes: small enough for the steep wall means glacial along the floor.
Standardize the features and the canyon becomes a bowl: every direction has a comparable slope, downhill points at the minimum, and one learning rate works everywhere. This is also the payoff of session 3's solver card: this is WHY sag and saga demand scaled inputs - they are descent methods, and unscaled features hand them a canyon.
Self-studyBatch vs stochastic vs mini-batch4 min▶
Batch gradient descent (today's lab) computes the gradient over ALL training rows before each step: smooth, exact, but one step costs a full pass over the data. Stochastic (SGD) computes it from ONE row: each step is noisy and often slightly wrong, but costs almost nothing - so you take thousands of cheap wobbly steps and still reach the valley, even when the data no longer fits in memory. Mini-batch - a few dozen rows per step - is the compromise that deep learning standardized on.
In scikit-learn this lives in SGDClassifier(loss="log_loss"): same logistic model, descent done one row at a time. When Lumen outgrows lbfgs - say 400 million sessions instead of 40,000 - this is the escape hatch. The loss curve wobbles instead of gliding; squint and it still goes down.
What sklearn's solvers actually are 5 min live
Now re-read session 3's solver table with today's eyes. lbfgs, newton-cholesky, sag, saga - every one is an industrial-strength variant of the loop you are about to write. lbfgs approximates the bowl's curvature to pick smarter strides; newton methods use the curvature exactly; sag and saga are stochastic descent with variance tricks so they converge like batch. Different shoes, same walk. You now know the family, so the docs read like a menu instead of a spellbook.
Self-studyWhen descent fails - a postcard from deep learning3 min▶
Everything today leaned on one luxury: convexity. Neural-network loss surfaces are NOT convex - they are mountain ranges with saddle points (flat in one direction, curved in another) and countless local dips. Descent still works there, but it needs help: momentum to coast through flats, Adam to adapt the stride per weight, and backpropagation to compute gradients through many layers - which is just the chain rule, chained longer.
None of that is this course's job. It belongs to the deep-learning course - and when you get there, the optimizer pages will read as "gradient descent plus tricks", because that is what they are.
max_iter) and the warning evaporates. Ten minutes, no mystery, no rebuilt model.Descend from scratch - and reproduce sklearn ★ 18 min · everyone builds
The lab this whole session exists for: batch gradient descent on the real Lumen data, two scaled features, learning rate 0.5, 400 iterations, about 15 lines of numpy. Then we fit LogisticRegression on the exact same features and compare coefficients. If the two match, you did not learn an analogy of how models train - you learned the thing itself.
Build the pieces. Standardize prior_30d_spend and pages_viewed on train only, then write the sigmoid and the log-loss function - the surface we are descending, straight from session 3.
Take ONE step by hand. Compute (p - y), the gradient, and one update of w and b. Print the loss before and after - it fell. That falling number IS learning.
Loop it 400 times. lr = 0.5. Watch the printed loss fall fast, then level off - the flattening is your convergence check.
Compare with sklearn. LogisticRegression(C=1e6) (near-unregularized, so the bowls match) on the same two features. Read the two sets of numbers side by side, slowly.
Before session 5 ◐ 25 min
- Rerun
descend.pywith lr = 0.05 and lr = 5.0, recording the loss every 10 iterations. Plot the three loss curves on one chart and name each shape: crawl, glide, or bounce? - Convert the loop to SGD: each iteration, update from ONE random training row instead of all of them. Plot the loss - it wobbles on the way down. Say in one sentence why that wobble is a price worth paying at 400 million rows.
- Delete the two scaling lines and rerun batch GD with lr = 0.5. Does it converge in 400 iterations at all? How many does it need - or what learning rate does it force you down to? Feel the canyon.
Three questions before you go 🎯 ◐ 90 seconds
1 · What does the learning rate control, and what happens when it is too big?
w := w - lr * gradient. The gradient gives direction, lr gives stride length. Too small merely crawls; too big lands higher on the opposite wall every step - divergence, the one failure with no partial credit.
2 · Why do we scale features before gradient descent?
A stretched valley means no single learning rate suits both axes: safe for the steep wall = glacial along the floor. Standardizing rounds the bowl - which is why descent-based solvers (sag, saga, SGD) demand scaled inputs.
3 · Stochastic gradient descent vs batch - what is the trade, and who wins on huge data?
One row per step means each step is slightly wrong but costs almost nothing - thousands of wobbly steps still find the valley. That is SGDClassifier(loss="log_loss"): the same logistic model, built for data that outgrows lbfgs.
What this session covers
Session 4 covers the gradient-descent core of the anchor sources - StatQuest's GD episodes and Andrew Ng's C1 optimization arc, with the classic numpy GD lab pattern mirrored on Lumen data. Videos, graded labs, and certificates stay with the official providers.