Taming the net

You already saw the shape in session 3's stuck-shapes gallery: training accuracy keeps climbing while validation goes flat, then drifts down. Train-up-test-flat. That gap is overfitting, and it is the only diagnosis tonight - everything else is treatment.

Every item in tonight's cabinet attacks that same gap from a different side: more data makes the training set harder to memorize; early stopping quits before the memorizing starts; dropout makes memorizing unreliable; weight decay makes it expensive; a smaller net makes it impossible. The order matters - and Part 2 gives you the order with sources attached.

• Generalization gap - train score minus test score. The number every treatment tries to shrink.
• Regularization - anything that trades training fit for holdout performance, from extra data to weight penalties.
• Interpolation threshold - the capacity at which a model can fit the training data exactly. Part 3's landmark.
• Label noise - wrong labels in the training set. Where double descent shows itself most clearly.

• Random labels: the nets reach perfect training accuracy. There is no pattern to learn - this is pure memorization, and the architecture does it without complaint.
• With regularizers on: essentially the same. The standard toolkit does not remove the memorization capacity.
• Random pixels: still fits. The result is not about image structure at all.

The punchline is a negative result about theory: if the same architecture can memorize noise perfectly, then classical complexity measures - counting capacity - cannot explain why that architecture generalizes on real labels. The capacity to memorize was always there. Something else decides whether it gets used.

Cite it as ICLR 2017 (or the 2021 CACM republication) - never 2016, whatever a slide deck tells you.

The paper shows classical tools are insufficient - it does not show generalization is unexplainable. The working answers under active research: the implicit bias of SGD (gradient descent prefers certain solutions among the many that fit), margin-based measures, and Part 3's double descent. The honest summary for a meeting: "we can measure that these nets generalize, and we can rule out the textbook explanation; the replacement theory is still being written."

Practical consequence, and the bridge to Part 2: since the architecture will not protect you from memorization, you have to - with data, stopping rules, and the rest of the cabinet.

(a) More data. Karpathy's recipe puts it beyond argument:

Every other treatment trades something away. Data does not - it directly makes the training set unmemorizable and the patterns unmistakable. It is only ever skipped because it is inconvenient, which is not a statistical argument.

(b) Early stopping. You already do this: session 3's habit of watching the validation curve. Stop when validation stops improving - the net was learning patterns first and noise second, so quitting early keeps the patterns and skips the cram. The ensemble course's early-stopping law on boosted trees is the same medicine administered to a different species.

(c) Dropout (Srivastava et al., JMLR 2014), mechanism verbatim:

• The theory: training with random deletions approximates averaging exponentially many thinned networks. Note the verb - it is an approximation argument, not a literal ensemble like the tree course's forests.
• Modern usage: mostly fully-connected heads and transformers; less in conv layers, where normalization layers took over the job.

(d) Weight decay, done right. For years people wrote "L2 regularization" and "weight decay" interchangeably. Loshchilov and Hutter's AdamW paper (ICLR 2019) showed the equivalence has a boundary:

With Adam, the L2 penalty gets rescaled by the adaptive step sizes - exactly the weights with large gradients get under-regularized. The fix is decoupling the decay from the gradient update, which is why torch.optim.AdamW is today's default and plain Adam plus L2 is the quiet bug in a thousand training scripts.

Smaller model is last on the shelf for a reason Part 3 makes vivid: capacity you remove is gone, and the relationship between capacity and generalization is stranger than the textbook says.

Nakkiran et al. then showed the effect in modern deep nets, both model-wise (grow the network) and epoch-wise (train longer) - and found regimes where, strangely, more data hurts. The scope, which is content and not a disclaimer: the effect is most visible with label noise near the interpolation threshold, and it does NOT mean bigger is always better or that regularization is obsolete.

You will see the shape with your own eyes next session: the 3x32 overparameterized net (holdout 68) beats the 2x16 memorizer (holdout 60) on the exact same 200 rows - a measured dip-and-recovery, live. And note who still wins that showdown: the booster at 73. Recovery is not victory.