Where session 1 left you
Customer A - 62 idle days, 1 order, 4 tickets - went through your paper neuron and came out at 0.83: a strong churn vote. Then the truth arrived: A stayed. The network was wrong, and "the network" is nothing but weights, so some weights are more at fault than others. Tonight computes exactly how much, for every weight, in one backward sweep. That sweep has a fearsome reputation and a five-letter name; by minute 40 it will be the same arithmetic you did last week, read right to left.
★ What you walk out with tonight. Every gradient of a real neuron computed by hand - and reconciled, number by number, with what the numbers should be. The visceral reason big raw inputs hijack learning, which is the reason for standardization. And the honest history: who actually invented backprop, decades before the paper everyone cites.
LiveThe question backprop answers3 min▶
For one wrong prediction, backprop computes d(loss)/d(weight) for every weight in the network - all in one backward sweep. Not by wiggling each weight and re-running the forward pass (that would cost one full pass per weight); by walking the chain rule from the loss back to the inputs, once. The chain rule is the whole trick. There is no other idea in it.
Each gradient is a blame number: "if this weight rose a little, the loss would rise this much." Training (session 3) is just: compute the blame, nudge every weight against it, repeat. Tonight is the compute-the-blame half, done slowly enough to own it.
This is also the skill that survives the tooling. When a production model trains badly, the symptoms are gradient-shaped: exploding steps, dead units, one feature dominating. Everyone can call .backward(); the person who has computed a gradient by hand is the one who can read what it is saying.
Self-studyFour words tonight leans on2 min▶
- Gradient - d(loss)/d(weight): how much the loss moves per tiny move of that weight. Sign is direction, magnitude is urgency.
- Local derivative - what one node knows about itself: how its output moves per tiny move of one input. A multiply node, an add node, a tanh node each have one, and none knows anything about the rest of the graph.
- Chain rule - derivatives along a path multiply. The global gradient of a weight is the product of every local derivative between it and the loss.
- Delta - the error signal at the output, p - y for our loss setup. Every weight's gradient tonight is delta times something.
A graph of tiny operations, micrograd-style
7 min liveRedraw the neuron not as a diagram of a "unit" but as what the arithmetic actually was: multiplies feeding an add feeding a tanh feeding a loss. Every node is trivial. That triviality is the design: a node only ever needs to know its own local derivative, and the backward pass multiplies those local derivatives along the path from the loss to each weight.
LiveEach node's local derivative - the complete toolbox4 min▶
Tonight's graph has exactly three node types, and their local derivatives are the entire toolbox:
| Node | Local derivative | In words |
|---|---|---|
| add | 1, toward every input | The add node is a gradient distributor: whatever blame arrives, it hands on unchanged to each thing it summed - including the bias. |
| multiply (w · x) | d/dw = x, d/dx = w | Blame toward the weight is the OTHER factor. This one line is why input scale will dominate Part 2. |
| tanh | 1 - tanh(z)² | At z = 1.18: 1 - 0.83² ≈ 0.32. Near z = 0 it is almost 1 (learns fast); beyond |z| = 3 it is under 0.01 (Part 3's saturation story). |
And one gift at the output: the course engine uses a sigmoid output with binary cross-entropy loss, and that pairing simplifies famously - the combined derivative of loss-through-sigmoid collapses to delta = p - y, the prediction minus the truth. We state that as the standard simplification rather than deriving it; the derivation is a satisfying ten minutes of algebra you can do once and never need again.
Self-studyWhy this is called reverse-mode, and why 100 lines suffice3 min▶
Karpathy's micrograd is the cleanest statement that this is ALL there is. Its README, verbatim:
The engine is about 100 lines and scalar-valued - every value in the graph is a single number, exactly like your pencil work tonight. "Reverse-mode" names the direction: one sweep from the loss backwards touches every node once and yields the gradient for ALL weights simultaneously. Forward-mode would need one sweep per weight - fine for 4 parameters, absurd for 70 billion. Reverse-mode's one-sweep-for-everything is the reason training large networks is possible at all.
PyTorch's autograd is this same idea, industrialized: tensors instead of scalars, thousands of node types instead of three. Session 3 meets it as one line - loss.backward() - and you will know precisely what that line does, because tonight you were that line.
One complete backward pass on customer A
6 min liveSame numbers as session 1: inputs (62, 1, 4), weights (0.04, -0.5, 0.3), bias -2, so z = 1.18 and a = tanh(1.18) ≈ 0.83. One honest simplification before we compute: the single-neuron demo wires the tanh output straight into a p-like churn score, so we can use the clean output rule delta = p - y directly. With the true label y = 0 (A stayed), delta ≈ 0.83. In the full engine the hidden layers are tanh and only the output is a true sigmoid; the arithmetic pattern you are about to do is identical there.
LiveLesson one: blame scales with the input3 min▶
Look at the three weight gradients: 51.5, 0.83, 3.3. The ratios are exactly the ratios of the raw inputs (62 : 1 : 4), because grad(w) = delta x input and delta is shared. The days weight did not earn 62x the blame by being 62x more wrong - it grabbed it because "days" happens to be measured in a unit that produces big numbers. Measure days in years and its gradient shrinks 365-fold with zero change in meaning.
This is WHY standardization matters, stated as calculus rather than folklore: features on wild scales get wildly unequal gradients, so gradient descent races along the big-scale weights and crawls along the rest. The course engine standardizes every feature before training - session 6 shows the machinery - and after tonight you know exactly what failure it is preventing.
LiveLesson two: one step at lr 0.01, and the crash3 min▶
The update rule is w -= lr x grad. Apply it once with lr = 0.01:
| Parameter | Old | Gradient | New (old - 0.01 x grad) |
|---|---|---|---|
| w_days | 0.04 | 51.5 | -0.475 - the weight sails straight past zero and flips sign, 13x its own former size |
| w_orders | -0.5 | 0.83 | -0.508 - a sane nudge |
| w_tickets | 0.3 | 3.3 | 0.267 - a sane nudge |
| bias | -2 | 0.83 | -2.008 - a sane nudge |
Re-run the forward pass with the new weights and z lands near -31, so the output slams from +0.83 to essentially -1: the neuron overshot from "confidently churn" to "maximally confident stay", pinned deep in tanh's saturated zone. One feature's raw scale turned a modest learning rate into a wrecking ball. Small learning rates and standardization are not two separate hygiene tips - they are one lesson about keeping lr x grad small, seen from both sides.
Session 3 will hand you a learning-rate lever (0.03 / 0.3 / 1.5) on the live engine. When the hot rate destroys training on one dataset and rescues it on the spiral, tonight's crash is the mechanism to remember: the step size that matters is lr x grad, and grad depends on your data's scales.
1986: what the famous paper actually claimed
4 min liveRumelhart, Hinton and Williams, Nature, 1986 - the paper that made neural networks respectable again. It is worth reading what it actually says, because both the pitch and the credit are routinely misremembered.
LiveThe real pitch: learned representations2 min▶
The mechanics, in the paper's own words: backprop "repeatedly adjusts the weights of connections in the network so as to minimize" the output error - which is exactly the loop you just ran once by hand. But the headline the authors chose was not the mechanics. It was this:
Learned representations. Nobody designs what the hidden units detect; blame assignment alone sculpts them into feature detectors. That is the claim that ripened, over decades, into AlexNet's learned visual features (session 4) and embeddings (session 4's second act). The 1986 pitch was never "we can fit curves" - it was "the features design themselves".
The correction, stated as course content and not fine print: backprop was NOT invented in 1986. Seppo Linnainmaa published reverse-mode automatic differentiation in 1970. Paul Werbos applied it to network-like models in 1974 and again in 1982. What 1986 contributed was the demonstration that it works and matters - trained hidden representations on real tasks, in Nature, at the right cultural moment. The lineage is documented in Schmidhuber's 2015 survey (section 5.5), and getting it right is a cheap credibility test: anyone who says "Hinton invented backprop in 1986" is repeating the legend, not the record.
LiveSaturation: where gradients go to die2 min▶
One more thing your homework from session 1 already brushed against. tanh beyond |z| = 3 outputs within half a percent of ±1 - and its local derivative 1 - tanh(z)² drops below 0.01. A saturated unit passes less than 1 percent of the arriving blame through to everything behind it. The gradient vanishes.
One saturated neuron is a detail. A DEEP network of squashing activations is a chain of these factors multiplied together - and a chain of numbers below 1 shrinks exponentially with depth. This is a large part of why the sigmoid-era deep nets of the 1990s stalled, and it is the thread session 4 picks up when ResNet's authors open with "deeper neural networks are more difficult to train". Remember tonight's 0.01: the whole architecture story of session 4 is downstream of that little number.
Assign the blame, by hand
★ 20 min · everyone buildsPencil, paper, and customer A. Every number below reconciles - if yours does not, the error is findable, which is the entire luxury of a four-parameter network. Do this once slowly and loss.backward() stops being an incantation forever.
Redraw customer A's forward pass as a graph. Inputs (62, 1, 4) into three multiply nodes (x 0.04, x -0.5, x 0.3), their outputs (2.48, -0.5, 1.2) plus bias -2 into the add node: z = 1.18. Through tanh: a ≈ 0.83. Label EVERY intermediate value on the drawing - the backward pass is about to consume each one.
Compute the local derivatives you will need. At the output, with the demo's tanh-as-score simplification and y = 0: delta = p - y ≈ 0.83. At the tanh node, for practice (the multi-layer case needs it): 1 - tanh(1.18)² = 1 - 0.83² ≈ 0.32. At the add node: 1 toward every input. At each multiply node: the other factor.
Chain them - fill in all four gradients. grad(w_days) = 0.83 x 62 ≈ 51.5. grad(w_orders) = 0.83 x 1 = 0.83. grad(w_tickets) = 0.83 x 4 ≈ 3.3. grad(bias) = delta ≈ 0.83 (the add node hands blame through unchanged). Sanity-check the signs: every gradient is positive, meaning "raising this weight raises the loss" - correct, since the network over-predicted churn and all inputs are positive.
Take one step at lr 0.01 and re-run the forward pass. w_days: 0.04 - 0.515 = -0.475. w_orders: -0.508. w_tickets: 0.267. bias: -2.008. New z ≈ -31, new output ≈ -1: the network vaulted from "confident churn" to "maximally confident stay" in one step. Write one sentence on why the days weight moved 62x further than the orders weight - if your sentence does not contain the words "input scale", look at step 3 again.
Standardize and redo the gradients. Rough z-scores for customer A against the customer base: days 62 becomes about 1.2, orders 1 becomes about -1.1, tickets 4 becomes about 1.5. Now grad(w_days) ≈ 0.83 x 1.2 ≈ 1.0, grad(w_orders) ≈ 0.83 x -1.1 ≈ -0.91, grad(w_tickets) ≈ 0.83 x 1.5 ≈ 1.25. Comparable magnitudes; a 0.01 step now nudges every weight by about a hundredth. The standardization lesson, earned by hand rather than recited.
What to bring to session 3. Your graph with all four gradients on it. Session 3 closes the loop: compute blame, step, repeat, hundreds of times, live in your browser - then the same five moves spelled out in PyTorch (forward, loss, backward, step, zero the gradients). You have now done the first four by hand; the fifth exists for a reason the live session demonstrates.
Before session 3
30 min- Run the backward pass on customer B (days 8, orders 6, tickets 0 - from session 1: z ≈ -4.7, output ≈ -1). Compute tanh's local derivative there: 1 - (-1)² ≈ 0.0004. Almost no blame can flow through a saturated unit, whatever the loss says. You have just met a vanished gradient in person.
- Flip the label: redo customer A with y = 1 (pretend A truly churned). delta = 0.83 - 1 = -0.17: every gradient flips sign and shrinks to a fifth of its size. Blame is proportional to how wrong you were - a nearly-right prediction generates nearly no learning.
- Rescue the raw-input crash with the OTHER lever: keep the unstandardized inputs but drop lr to 0.001. w_days now moves 0.04 to -0.012 - still sign-flipping. Write one sentence on why shrinking the learning rate alone is a poor substitute for standardizing.
- Read micrograd's engine for real this time - about 100 lines. Find the tanh backward rule and confirm it is your 1 - tanh² from step 2, written as code. Every gradient you computed tonight, that file computes the same way.
- Optional: skim section 5.5 of Schmidhuber's 2015 survey for the full backprop lineage. Ten minutes, and the 1986 legend never fools you again.
Three questions
Clear all three and session 2 gets stamped in your passport.
1 · In the hand-computed backward pass, the days weight received a gradient of 51.5 while the orders weight received 0.83 - about 62x more. Why?
The multiply node's local derivative toward its weight is the OTHER factor - the input. Delta is shared, so gradient ratios equal raw input ratios: 62 : 1 : 4. Nothing about the data's meaning says days matter 62x more; measure days in years and the gradient shrinks 365-fold. This is the calculus behind standardization, and step 5 of the build-along showed z-scored inputs producing comparable gradients (1.0 / -0.91 / 1.25).
2 · What did the 1986 Rumelhart, Hinton & Williams paper actually contribute to backpropagation?
Reverse-mode automatic differentiation was published by Linnainmaa in 1970 and applied to network-like models by Werbos in 1974 and 1982 (lineage per Schmidhuber 2015, section 5.5). The 1986 paper's contribution was the demonstration - and its chosen headline was learned internal representations, not curve-fitting. "Invented in 1986" is the legend; "shown to work and matter in 1986" is the record.
3 · A tanh unit sits at z = 5, outputting 0.9999. Why has it effectively stopped learning?
The backward pass multiplies local derivatives along the path, and a saturated tanh contributes a factor below 0.01 (at |z| = 3; far smaller at 5). Whatever blame arrives, under 1 percent continues through. Stack many squashing units in a deep net and these sub-1 factors multiply into exponential vanishing - a core reason sigmoid-era deep nets stalled, and the thread session 4's ResNet story picks up.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 2 cheat sheet · pin this
Tonight you computed one backward pass. Session 3 closes the loop: blame, step, repeat - gradient descent running live in your browser on a real network, where you will watch XOR go from a 48.3 percent coin flip with no hidden layer to 95 with one layer of 8, and a spiral sit stuck at 55 until the optimizer (not the theorem) arrives. Then the same loop, spelled as PyTorch's five moves.