How this course works
Six sessions, 45 minutes each, one network followed all the way: built as scalars on paper (sessions 1-2), trained live in your browser where every forward and backward pass is real (session 3), grown into the architectures that ate vision (session 4), tamed when it memorizes (session 5), and finally benchmarked - honestly, on identical data - against the gradient booster from the ensemble methods course (session 6). The build-it-yourself ethos is deliberate: the whole modern stack is this session's neuron, repeated.
★ What you walk out with tonight. The neuron computed by hand until it is boring. The one-line proof-sketch of why depth without nonlinearity is a scam. And the exact scope of the universal approximation theorem, which will let you catch its misuse in roughly every third AI meeting.
LiveThe whole unit, in one line3 min▶
A neuron takes inputs x1..xn and computes a = f(w1·x1 + ... + wn·xn + b). The weights w say how much each input matters and in which direction. The bias b shifts the threshold. The activation f bends the line - tanh squashes to (-1, 1), ReLU clips negatives to zero, sigmoid squashes to (0, 1).
That is the entire unit. A layer is neurons sharing the same inputs; a network is layers feeding layers. When the press says a model "has 70 billion parameters", it is counting the w's and b's in this line.
Mango Lane's churn score in the tree course started from the same raw material: days since last order, orders, tickets. A single neuron with weights (+ on days, - on orders, + on tickets) IS a tiny churn scorer - it just cannot express "inactive AND low sessions" as a distinct condition yet. That gap is what tonight's Part 2 is about.
Self-studyThe vocabulary for the whole arc3 min▶
- Forward pass - push inputs through the layers to an output. Session 1's whole job.
- Loss - one number scoring how wrong the output is. Sessions 2-3 chase it downhill.
- Backward pass - the chain rule walked in reverse, assigning each weight its share of the blame. Session 2, by hand.
- Hidden layer - any layer between input and output. What its neurons compute is not designed - it is learned, and that is the 1986 result session 2 quotes.
- Activation / nonlinearity - the bend. Without it, no depth is real depth (Part 2 proves it in three lines).
Three numbers and a squash
6 min liveOne neuron, drawn honestly, with real numbers flowing through it. Anyone who can multiply can run this by hand - which is exactly what the build-along makes you do until it stops feeling clever.
LiveWhat each part is FOR4 min▶
- Weights are learned opinions. 0.04 per idle day means idleness matters, gently, toward churn. The sign is the direction; the magnitude is the volume. Nothing else in the network carries knowledge.
- The bias is the threshold. With b = -2, this neuron stays quiet unless the weighted evidence clears 2. Without a bias, every neuron would be forced to fire at the origin, which is almost never where the interesting boundary lives.
- The activation is the commitment device. tanh(1.18) = 0.83 turns unbounded evidence into a bounded vote. And its S-shape is what makes the calculus of session 2 work: near the extremes the neuron is confident and barely budges; near zero it is undecided and learns fast.
One habit from tonight forward: count parameters. This neuron has 4 (three weights, one bias). A layer of 8 such neurons: 32. Session 3's spiral-solving network of two hidden layers of 16: 353. Parameters are the budget the data has to pay for - a number you will use against overfitting in session 5 and in every architecture review after.
Self-studyChoosing the bend: tanh, ReLU, sigmoid3 min▶
| Activation | Shape | Where it shines | Its known trouble |
|---|---|---|---|
| tanh | S-curve, (-1, 1), zero-centered | Small nets, this course's engine | Saturates: gradients vanish at the extremes |
| ReLU | max(0, z) - a hinge | Deep nets; cheap; the modern default | Dead units: a neuron stuck below zero learns nothing |
| sigmoid | S-curve, (0, 1) | Output layers needing a probability | As a hidden activation, saturates worse than tanh |
The course engine uses tanh in hidden layers and sigmoid at the output - the classic small-network recipe, and the one whose derivatives you can verify by hand in session 2.
Stacking linear things buys nothing
7 min liveRemove the activation and a deep network collapses. Not degrades - collapses, mathematically, into a single linear model wearing a costume. Three lines show it, and one tiny dataset from 1969 shows why it matters.
LiveXOR: the tiny dataset that makes it concrete4 min▶
Four points. (0,0) and (1,1) are class A; (0,1) and (1,0) are class B. Try to separate them with one straight line - you cannot, and no weighted-sum-plus-threshold ever will, because the classes sit on the diagonals. This is XOR, the example that famously stalled neural network research after Minsky and Papert highlighted it in 1969.
One hidden layer fixes it: neuron one learns roughly "x OR y", neuron two learns roughly "x AND y", and the output learns "OR but not AND" - a bent region no single line can draw. In session 3 you will watch the live network do exactly this, and the measured numbers are already on the course map: a no-hidden-layer model scores 48.3 percent on XOR test data - a coin flip - while one hidden layer of 8 reaches 95.
Mango Lane's churn signal contains a real XOR-shaped clause: customers who are BOTH long-idle AND low-session churn at a much higher rate than either condition alone predicts. A linear scorer adds the two evidences; only a bent model can price the combination differently from the sum. Interactions are XOR wearing business clothes.
What the universal approximation theorem promises - and refuses to
6 min liveThe most-cited justification for neural networks is also the most misquoted. Cybenko in 1989, and Hornik, Stinchcombe and White the same year, proved something real and beautiful - and narrower than almost everyone who cites it believes.
LiveThe exact words, and the misuse to catch4 min▶
Cybenko's 1989 result, in the paper's own framing: "arbitrary decision regions can be arbitrarily well approximated by continuous feedforward neural networks with only a single internal, hidden layer and any continuous sigmoidal nonlinearity." Hornik, Stinchcombe and White proved the same year that one hidden layer with any squashing function approximates any reasonable function "provided sufficiently many hidden units are available."
The misuse arrives in meetings as "neural networks can learn anything - it is proven." The correction is one sentence: the theorems prove a good network exists; they say nothing about whether your optimizer can find it, how much data pins it down, or whether the required width is affordable. Existence, findability, and affordability are three different claims, and only the first one was proven in 1989.
Why depth, then, if one layer suffices in theory? Because "sufficiently many hidden units" can be brutally many. Depth often buys the same expressiveness with far fewer parameters, and - as session 4's residual story shows - the modern craft is largely about making that depth trainable. The theorem permits shallow; practice votes deep; neither contradicts the other.
Neurons on paper, until they are boring
★ 18 min · everyone buildsNo library tonight. You are the forward pass. The point of doing it by hand once is that session 2's backprop - the part with the fearsome reputation - becomes just this, walked backwards.
Run the Part 1 neuron on three customers. Weights (0.04, -0.5, 0.3), bias -2, tanh. Customer A: days 62, orders 1, tickets 4. Customer B: days 8, orders 6, tickets 0. Customer C: days 40, orders 2, tickets 2. Compute the weighted sum first, then look up tanh roughly: tanh(1.2) is about 0.83, tanh(-4.7) is about -1, tanh(-0.8) is about -0.66. Two clear votes and one lukewarm one.
Change ONE weight and re-run customer C. Raise the tickets weight from 0.3 to 0.8. C's sum goes from -0.8 to +0.2 - the vote flips sides. You have just felt the entire mechanism of learning: nothing changes but weights, and everything changes with them.
Try to draw XOR's separating line. Plot the four points from Part 2 on paper and attempt one straight line with all A's on one side. Spend ninety honest seconds. The failure you feel is the same 48.3 percent the live network scores with zero hidden layers.
Now build XOR's hidden layer by hand. Neuron h1: weights (1, 1), bias -0.5, output "fires if x + y > 0.5" - roughly OR. Neuron h2: weights (1, 1), bias -1.5 - roughly AND. Output neuron: weight +1 on h1, -1 on h2, bias -0.5 - fires for OR-but-not-AND. Check all four points. You just hand-built the network that ended the 1969 objection.
Count your parameters. The XOR net: 2+1 per hidden neuron, twice, plus 2+1 at the output = 9. Write that number down: in session 5 you will meet a net with 100 times the parameters of its dataset and watch what it does with the surplus.
What to bring to session 2. Your three computed neurons. Session 2 asks the reverse question - given that customer A's true label was "stayed", how much was each of your three weights to blame for the wrong vote? - and answers it with nothing but the chain rule you already know.
Before session 2
30 min- Run five more customers through your neuron - invent the numbers. Note when tanh saturates (sums beyond about 3 all output nearly 1) and keep that observation: it becomes "vanishing gradients" in session 2.
- Rebuild the XOR hidden layer with different numbers. Any weights that make h1 an OR-ish gate and h2 an AND-ish gate work. The lesson: many weight settings express the same function - remember this when session 5 discusses why nets have so many good solutions.
- Find the XOR in your own data: one pair of conditions whose combination means something different from the sum of the parts. Write it as a sentence.
- Read the Cybenko abstract (one paragraph) and underline what is promised and what is absent. Ten minutes, permanent immunity to the meeting misquote.
- Optional: skim the README of Karpathy's micrograd - about 100 lines of code implementing everything session 2 does on paper. Do not study it yet; just see how small it is.
Three questions
Clear all three and session 1 gets stamped in your passport.
1 · A colleague proposes a 10-layer network with no activation functions "for maximum depth". What does the network actually compute?
Substitute layer by layer: W2(W1x + b1) + b2 = (W2W1)x + (W2b1 + b2) - one matrix, one bias, however many layers you stack. The nonlinearity between layers is the only thing that makes depth real. Ten linear layers cost ten layers of compute and buy one layer of model.
2 · The universal approximation theorem is cited to claim "neural networks can learn anything". What did Cybenko and Hornik actually prove?
Both 1989 results are existence theorems, with Hornik's own condition attached: "provided sufficiently many hidden units are available" - and that width can be exponential. Existence, findability, and affordability are three separate claims. The live spiral demo in session 3 is the gap made visible: capacity present, optimizer not yet arrived.
3 · Why does XOR defeat any single neuron, no matter the weights?
One neuron draws one line. XOR needs a bent region: roughly "OR but not AND", which two hidden neurons plus an output express with 9 parameters, as you built by hand. Measured on the live engine: zero hidden layers scores 48.3 percent on XOR test data; one layer of 8 scores 95.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 1 cheat sheet · pin this
Tonight you ran the forward pass by hand. Session 2 runs it backwards: the chain rule walked through a scalar graph, micrograd-style, assigning every weight its exact share of the blame for a wrong answer - plus the honest history of who actually invented backprop, which is not who the textbooks imply.