Where session 2 left you
You can now compute, by hand, how much each weight contributed to one wrong answer. Training is that computation run over all examples, converted into a small correction, and repeated until the corrections stop helping. Everything else - optimizers, learning rates, epochs - is bookkeeping around that sentence.
★ What you walk out with tonight. The loop as five moves you can name in PyTorch and point to in the browser engine. A boundary you watched a real network carve. And the spiral's lesson - capacity and optimization are different problems - which is worth more than most courses' entire treatment of "hyperparameters".
LiveThe loop, in the words you will actually type4 min▶
PyTorch's own tutorial reduces training to five moves, and they map one-to-one onto what session 2 built on paper:
- pred = model(X) - the forward pass. Session 1, vectorized.
- loss = loss_fn(pred, y) - one number scoring the wrongness.
- loss.backward() - the chain rule walked in reverse. Session 2, automated.
- optimizer.step() - every weight moves a little, against its blame.
- optimizer.zero_grad() - reset the ledger. The tutorial's own reason, verbatim: "Gradients by default add up; to prevent double-counting, we explicitly zero them at each iteration."
The order of zero_grad is flexible; the invariant is once per iteration, before the next backward. As of September 2026 the stable release is PyTorch 2.14, and torch.compile is the standard opt-in speedup - but these five moves have not changed since the beginning, which is why they are the thing to learn.
Learning rate and epochs: the step and the patience
7 min liveTwo numbers govern the loop. The learning rate scales every correction; the epoch count decides how long you keep correcting. Both have failure modes at both ends, and tonight you will produce all four failures on purpose.
LiveThe Karpathy recipe: why loops fail silently5 min▶
Andrej Karpathy's "A Recipe for Training Neural Networks" opens with the premise that explains most wasted GPU-months: "neural net training fails silently." A broken loop does not crash - it trains a slightly wrong thing and reports plausible numbers. The recipe's defenses, in his own words:
- "Become one with the data" - hours of looking before any model exists.
- "Verify loss @ init" - an untrained classifier on balanced two-class data should start near log(2) ≈ 0.69. If it does not, the bug is already present.
- "Overfit one batch... as little as two" examples - if the loop cannot drive two examples to zero loss, moves 3-5 are broken, and no amount of data will hide it.
- "Don't be a hero" - standard architecture first, cleverness after the baseline works.
The habit version. Fix the seed, verify the init loss, overfit one batch, then scale. Four checks, five minutes, and the silent failure class mostly dies. The browser engine below fixes its seed for exactly the first reason: when you run it twice, you get the same numbers, so a change you see is a change you made.
Self-studyFrom this loop to Adam, honestly3 min▶
The engine on this page uses plain full-batch gradient descent - every example votes, one aggregate step. Production loops differ in two upgrades:
- Minibatches: step on a random few hundred examples at a time. Noisier steps, far more of them per pass - almost always a net win at scale.
- Adaptive steps: Adam (ICLR 2015) keeps running estimates of each weight's gradient moments and scales each weight's step individually - "well suited for problems that are large in terms of data and/or parameters", per the paper. The modern default is AdamW (2019), which fixed a subtle flaw: L2 regularization and weight decay "are equivalent for standard stochastic gradient descent... but as we demonstrate this is not the case for adaptive gradient algorithms, such as Adam". Decoupling them, in torch.optim.AdamW, measurably generalizes better.
Neither upgrade changes the five moves. They change how big move 4 is, per weight - the loop is the loop.
Carve boundaries, live
★ 22 min · everyone buildsThe playground below is the course engine running for real: every press of train executes genuine forward and backward passes, and the heatmap is the live network queried on a pixel grid. Cyan region: the net votes class one. Amber: class zero. The dots are the data it is learning from.
Start on moons, defaults (2 layers of 8, rate 0.3). Train 200 epochs. Watch the boundary bend around the two crescents and the numbers settle near 93 train / 88 test. Easy data: everything works, which is exactly why it is first - you now know what healthy looks like.
Switch to XOR and set hidden layers to 0. Train as long as you like. The boundary stays a straight line - it can only ever be a straight line - and the test score sits at 48.3 percent, a coin flip. This is session 1's algebra rendered in pixels: no bend, no XOR.
Give it one hidden layer of 8 and retrain. The line bends into the diagonal regions and the test score reaches 95. Nine-ish parameters of bend, exactly the network you built by hand in session 1, found by the loop instead of by you.
Now the main event: spiral, 1 layer of 8, rate 0.3. Train 600 epochs (three presses). Stuck at roughly 55 percent, boundary a vague smear. The tempting diagnosis is "not enough capacity". Hold that thought.
Same architecture, rate 1.5. Train 1200 epochs (six presses). The SAME network unwinds the spiral - about 98.9 train / 100 test. Nothing about capacity changed; the optimizer got there. The universal approximation theorem promised the network existed; it never promised rate 0.3 and 600 epochs would find it. You have now seen the difference between existence and findability, measured.
Finish with 2 layers of 16 at rate 1.5. The spiral falls in 600 epochs flat: 100 / 100. More capacity did not make the solution better - it made the solution EASIER TO FIND. That reframe, capacity as optimization aid, is the honest modern reading of why depth wins, and it returns in session 4 as the ResNet story.
The sentence to keep. "The network cannot learn it" is two different diagnoses wearing one phrase: not enough capacity, or not enough optimization. The spiral just showed you they respond to different medicine - and that the second is the one people misdiagnose.
Diagnosing the loop: a field guide
5 min liveEvery stuck training run you will ever debug is one of a handful of shapes. You produced most of them tonight on purpose; here is the field guide version, in the order to check.
LiveWhy the engine fixes its seed, and why you should too3 min▶
The recipe's second step includes an unglamorous instruction that pays for itself forever: "fix random seed... guarantee that when you run the code twice you will get the same outcome." Every number this course quotes - the 48.3, the 95, the 55-then-solved spiral - is reproducible on your machine because the engine's initialization and datasets are seeded.
Without the seed, a run that improves might be your change or might be luck, and you cannot tell. With it, every difference you see is a difference you caused. Determinism is not pedantry; it is the precondition for learning anything from an experiment.
Mango Lane's contractor "improved" a model four times in one week. Three of the four improvements were reruns with different random initializations of the same code. Nobody could tell, because nothing was seeded - a week of work indistinguishable from a random number generator.
Before session 4
30 min- Produce all four stuck shapes on the playground - flat (rate 0.03 on spiral), thrash (find a config where rate 1.5 hurts), memorize (you will need session 6's tabular mode for the full version - note which shape you could NOT produce on 2D toys and why), crawl (rate 0.03 on moons). Write one line per shape.
- Find every dataset's minimum architecture: for moons, xor, circle, spiral - the smallest layers-and-width that clears 90 percent test. Note that the ranking of datasets by difficulty IS a statement about their geometry.
- Write the five moves from memory as five lines of pseudo-PyTorch. Check against the official tutorial. Ten minutes, permanent.
- Read the Karpathy recipe post in full - it is short, and you have now personally verified three of its rules.
- Optional: install PyTorch and run the tutorial loop on its FashionMNIST example. You will recognize every line.
Three questions
Clear all three and session 3 gets stamped in your passport.
1 · The spiral sat at 55 percent with one layer of 8 at rate 0.3, then the SAME architecture solved it at rate 1.5 with more epochs. What does this demonstrate?
The theorem's existence promise was kept - the fix required zero new parameters. What changed was findability: a hotter rate and more epochs let gradient descent escape the flat region it was crawling through. "The network cannot learn it" is two diagnoses in one phrase, and this is the measured proof they differ. (And no - on easier surfaces a hot rate can thrash; the field guide's shape two.)
2 · Why must gradients be zeroed once per iteration in the PyTorch loop?
Accumulation is a feature (it enables gradient accumulation tricks), which is exactly why resetting is YOUR job. Forget it and the loop still runs, loss still falls for a while, and the model is quietly wrong - a textbook member of the fails-silently class the Karpathy recipe exists to catch.
3 · Your new classifier's loss at initialization is 2.9 on a balanced two-class problem. Before touching the architecture, what does the recipe say this means?
An untrained net on balanced two-class data should be maximally unsure, and maximal unsureness has a computable loss. 2.9 at init means the output layer, the loss wiring, or the labels are already wrong - and training would "work" anyway, silently, on top of the bug. Cheapest check in the whole discipline.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 3 cheat sheet · pin this
The loop can now train anything you can wire up. Session 4 is about what to wire: convolution as a promise about images (LeNet's 0.95 percent on digits in 1998), the two-GPU week that changed everything in 2012 (their entry: 15.3 percent top-5 against the runner-up's 26.2), and the residual trick that made 152 layers trainable - plus embeddings, where the hidden layers' learned representations become the product.