From loops to architectures
You now own the five moves: forward, loss, backward, step, zero the gradients. The loop does not care what network you hand it - it trained the spiral net and it would train a thousand-layer monster with equal obedience. So the question moves up a level: what should you wire, and why does the shape of the wiring matter more than almost any knob you tuned last week? Tonight covers the three architectures that answered that question for vision, and the one output everyone quietly ships to production: the embedding.
★ What you walk out with tonight. A one-line definition of inductive bias you can defend in a review. The two most misquoted numbers in deep learning history (AlexNet's 15.3 and ResNet's 3.57), quoted correctly. And a by-hand parameter count that shows convolution saving roughly 545x - and what that saving costs.
LiveAn architecture is a promise about the data4 min▶
Every architecture bakes in an assumption before a single gradient flows. Convolution promises that nearby pixels relate - that a pattern worth detecting in one corner is worth detecting in every corner. Sequence models promise that order matters - that yesterday constrains today. A plain MLP promises nothing at all - every input may relate to every other, equally, in any direction.
Open-mindedness sounds like a virtue, but the MLP pays for it in parameters and data: with no assumption to lean on, every relationship must be learned from scratch, and the data must fund every one of those weights. When the promise matches the data, the architecture learns more from less. When it does not - and session 6's tabular showdown is exactly that mismatch - the promise is dead weight.
You already run on promises. The tree course's booster promises that axis-aligned splits on individual features capture the signal - a promise tabular data usually keeps. Tonight's question, every time you meet a new dataset: what structure does this data actually have, and which architecture has already promised to exploit it?
Self-studyTonight's vocabulary3 min▶
- Inductive bias - the assumptions an architecture makes before seeing data. The stronger and truer the bias, the less data pays.
- Kernel / filter - a small grid of weights (3x3 tonight) slid across the image, computing the same detection everywhere.
- Receptive field - the patch of input a given unit can actually see. Grows as layers stack.
- Residual connection - a skip that adds a layer's input to its output: out = x + F(x). Session 4's second act.
- Embedding - the activation vector of a late hidden layer, used as learned coordinates. Session 4's third act, and the one you can ship.
LeNet's three moves
7 min liveLeCun et al.'s 1998 LeNet-5 paper is the clearest statement of the promise ever written down. Three moves, each one a bet on how images work: local receptive fields (a unit looks at a neighborhood, not the whole image), shared weights (the same detector runs at every position), and spatial subsampling (nearby detections get pooled, buying tolerance to small shifts). Together: shift invariance, and a massive cut in parameters - all still trained end-to-end with the same gradient descent you ran last session.
LiveThe numbers, said correctly4 min▶
LeNet-5 reached 0.95 percent test error on MNIST digits, and 0.8 percent when the training set was augmented with distorted copies. Trained end-to-end with gradient descent - the promise did not need a special optimizer, just the loop you already run.
- Precision trap: the 0.7 percent sometimes quoted from that paper belongs to boosted LeNet-4 - an ensemble, not LeNet-5. Same lesson as every headline number tonight: check whether it is one model or a committee.
- Lineage trap: convolution did not appear in 1998. LeCun's zip-code nets used it in 1989, and the architectural idea traces to Fukushima's Neocognitron in 1980. 1998 is when the recipe was written down cleanly, not when it was invented.
What the 3x3 kernel cannot do: see anything wider than 3 pixels. One conv layer is nearsighted by design. The fix is stacking - each layer's units look at 3x3 patches of the layer below, so the effective receptive field grows with depth. Depth is how convolution keeps its local promise and still ends up seeing the whole image.
Self-studyThe three moves, one table3 min▶
| Move | The promise it encodes | What it buys |
|---|---|---|
| Local receptive fields | Meaning is local: edges, corners, strokes live in small neighborhoods | Each unit needs 9 weights, not 784 |
| Shared weights | A pattern worth detecting here is worth detecting everywhere | One detector serves all positions - the ~545x cut |
| Spatial subsampling | The exact pixel position of a detection matters less than its rough location | Tolerance to small shifts and distortions |
All three are restrictions - the network is less free than an MLP. That is the point. Inductive bias is the art of forbidding the network from learning things that cannot be true of the data.
AlexNet and ResNet: the two numbers everyone misquotes
8 min liveTwo papers turned convolution's promise into the modern field: AlexNet proved the promise scales, ResNet proved depth could be made trainable. Both come with a famous number, and both numbers are routinely quoted wrong. Tonight you learn the numbers with their fine print, because precision here is a professional tell.
LiveAlexNet 2012: the GPU moment, quoted exactly4 min▶
Krizhevsky, Sutskever and Hinton entered the ILSVRC-2012 image recognition challenge with a 60-million-parameter convolutional network and, in the paper's own words:
Now the fine print, which is course content, not a footnote:
- 15.3 was their ENTRY - a 7-CNN ensemble. The single network scored 18.2 percent on validation. Say "their entry scored 15.3", never "the network scored 15.3".
- The runner-up at 26.2 was classical computer vision - hand-engineered features, the best the old paradigm had. That is what makes the gap historic: it was a paradigm gap, not a tuning gap.
- The accurate shorthand: AlexNet's entry cut top-5 error by over 10 points, roughly 40 percent relative. Never say "halved" - it overshoots.
Five to six days on two consumer gaming GPUs. The bitter economics lesson of 2012 was not the architecture - LeNet had the promise 14 years earlier - it was that compute plus data had finally caught up to the promise. When you evaluate a deep learning pitch, ask the AlexNet question: does this team have the data and compute to fund the architecture they chose?
LiveResNet 2015: the degradation problem4 min▶
By 2015 the field had a strange embarrassment. He et al. open their paper with it:
The embarrassment, precisely: stack more plain layers and training error got worse. Not test error - training error. A 56-layer plain net fit the training data worse than a 20-layer one. That rules out overfitting entirely (an overfitter aces training and flunks test). This is the degradation problem: an optimization failure, a network that mathematically contains the shallower solution but that gradient descent cannot steer to it.
The residual fix is one addition: out = x + F(x). Each block now learns the difference from identity, and doing nothing (F = 0) is trivially learnable. Result: 152 layers, "8x deeper than VGG", and first place at ILSVRC 2015 with a top-5 error of 3.57 percent.
- Precision trap: 3.57 is - again - an ensemble number. The committee habit runs right through this history.
- Mechanism trap: "ResNet fixed vanishing gradients" is wrong. Batch normalization had largely handled vanishing gradients already; the paper's claim is optimization tractability - identity shortcuts make the deeper solution easy to find.
- Scope trap: the celebrated 1000+ layer experiments are on CIFAR, not ImageNet.
Self-studyThis is session 3's spiral lesson, in architecture form3 min▶
Remember the spiral: the 1x8 net sat at 55 percent looking like a capacity problem, then the same net hit 100 with a hotter rate and more epochs. The capacity was there; the optimizer had not arrived. The degradation problem is the same story at scale: the 56-layer plain net contained the 20-layer solution (set 36 layers to identity), but gradient descent could not find it.
Residual connections are the architectural version of that lesson: instead of asking the optimizer to work harder, reshape the problem so the good solutions are easy to reach. Capacity is not just what a network can express - it is what the optimizer can find. ResNet moved "find" closer to "can".
Embeddings: the layer you actually ship
4 min liveBack in session 2 you met the 1986 backprop paper's real pitch - not the algorithm, the payoff: hidden units "come to represent important features of the task domain". Four decades later that sentence is an industry. The layers before the output are a learned coordinate system where similar things sit near each other - and word embeddings, product embeddings, and customer embeddings are exactly that: hidden-layer activations, extracted and sold as a feature.
LiveFrom 1986's promise to a product3 min▶
Train a network on any task and the late hidden layers become a map: inputs the task treats similarly end up close together. Extract that vector and you have an embedding - a learned feature that other models, searches, and recommenders can reuse. The classifier was the training exercise; the representation is the asset.
Mango Lane could embed products from co-purchase sequences: train a net to predict what gets bought together, then harvest the hidden layer. Products bought in similar contexts land near each other - dresses near skirts - without anyone tagging categories. The recommendation course's matrix factorization is a shallow cousin of exactly this: two embedding tables and a dot product, no depth.
The honest boundary. Transformers and LLMs are this idea at industrial scale - embeddings of words and their contexts, stacked and refined. They belong to the GenAI shelf of this hub, and this course stops at the pointer. What you should carry across: when someone says "the model understands", the checkable claim underneath is "the embedding places similar things near each other".
Count what convolution saves
★ 18 min · everyone buildsNo code tonight - arithmetic and paper. The promise of Part 1 becomes a number you compute yourself, and by step 5 you will have designed your own embedding.
Price the open mind. A 28x28 grayscale image (784 pixels) into a fully-connected layer of 100 units: every unit sees every pixel, so 784 x 100 = 78,400 weights, plus 100 biases. Write it down - that is what "promises nothing" costs.
Price the promise. The same image through 16 conv kernels of 3x3: each kernel is 9 shared weights, so 16 x 9 = 144 weights, plus 16 biases. Write both numbers side by side and the ratio: 78,400 / 144 is roughly 545x fewer weights. That factor is bought entirely by the assumption that patterns repeat across the image.
Say what the kernel cannot see - one sentence. A 3x3 kernel is blind to anything wider than 3 pixels. Then say the fix: stacking layers grows the receptive field, because each layer's units see 3x3 patches of the layer below - depth is how a local promise ends up covering the whole image.
Sketch a residual block on paper. Take any two layers you like, wrap them as F, and draw the skip: out = x + F(x). Mark the skip arrow and note next to it: the gradient now has a highway - and "do nothing" costs the optimizer nothing, because F(x) = 0 recovers the input exactly.
Design your domain's embedding - one sentence. Name the entities (products? customers? support tickets?), and say what similarity should mean (bought in the same baskets? churn for the same reasons?). If you can write that sentence, you have specified an embedding; the network's job is only to make it measurable.
What to bring to session 5. Your parameter counts. Session 5 is about what a network does when you give it far more parameters than data - and its first exhibit is a net you will price the same way you priced these.
Before session 5
30 min- Redo the parameter count for a 5x5 kernel (25 weights each, 16 kernels). Notice the ratio against fully-connected barely moves - the saving comes from sharing, not from the kernel being tiny.
- Stack receptive fields by hand: two 3x3 conv layers - how wide a patch of the original image can a second-layer unit see? (Draw it: 5x5.) Three layers? (7x7.) Write the pattern.
- Rehearse the two corrections aloud: "15.3 was their entry - a 7-CNN ensemble; the single net was 18.2 on validation" and "degradation is a training failure, not overfitting - deeper plain nets got worse train error". Once each, out loud. They will come up in real meetings.
- Find the embedding already near you: one system at work that ranks, recommends, or searches. Ask what representation it compares. If the answer is hand-built rules or raw features, you have found where a learned embedding would go.
- Optional: skim section 1 of the ResNet paper (arXiv:1512.03385) - just the figure showing 56-layer vs 20-layer training curves. You now know exactly what you are looking at.
Three questions
Clear all three and session 4 gets stamped in your passport.
1 · Your by-hand count showed 16 conv kernels using ~545x fewer weights than a fully-connected layer on the same image. Why is this a promise about images rather than a free lunch on all data?
Shared weights encode "a pattern worth detecting here is worth detecting everywhere" - true of images, where a vertical edge means the same thing at any position. On data without that structure (session 6's tabular churn table, for instance), the restriction removes capacity the data actually needed. Inductive bias is a bet: 545x cheaper when it pays, dead weight when it does not.
2 · Before ResNet, a plain 56-layer network performed worse than a plain 20-layer one. What made this the "degradation problem" rather than ordinary overfitting?
An overfitter aces training and flunks test. The 56-layer plain net got worse on the training set itself, despite mathematically containing the 20-layer solution. He et al.'s fix reformulates layers "as learning residual functions with reference to the layer inputs" - making "do nothing" trivially learnable. And note the adjacent trap: this is optimization tractability, not vanishing gradients - BatchNorm had largely handled those.
3 · A slide claims "AlexNet halved ImageNet error: 15.3% vs 26.2%". What is the precise version?
The paper reports a "winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry" - and that winning entry averaged 7 CNNs, with the single network at 18.2 percent validation. 26.2 to 15.3 is not halving; "cut top-5 by over 10 points, roughly 40 percent relative" is the honest shorthand. Say "their entry" and you will be more precise than most conference speakers.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 4 cheat sheet · pin this
Tonight's architectures earn their parameters. Session 5 is about what happens when they do not: a network with more parameters than data points will happily memorize everything - including, famously, completely random labels. The diagnosis, the medicine cabinet in the right order, and the strange plot twist where bigger sometimes recovers.