learn-deep-learning-with-phoebe / Session 6 of 6
Learn Deep Learning with Phoebe · Session 6 of 6

When deep learning

Five sessions built a real network and made it trainable. Tonight it faces the honest question: against the ensemble course's tuned gradient booster, on byte-identical data, who wins? The showdown runs live - the nets top out around 68 to 70 percent holdout, the booster holds 73 - and the loss is the most instructive result in either course, because the same architecture family that loses this fight owns images, speech and language outright. The finale is the modality map that explains both facts at once, without tribal loyalty to either side.

🔴 Capstone Everyone who climbed sessions 1-5 Pairs with the ensemble methods capstone 45 min
0-3 · Welcome 3-18 · The modality map 18-40 · Build-along: the showdown 40-45 · Q&A
Part 0

The question this course owes you

You can now build, train, diagnose and tame a neural network. The last skill is knowing when NOT to reach for one - and the answer is not a vibe, because the ensemble course left a benchmark sitting on this exact dataset: a tuned gradient booster at 73.0 percent holdout. Tonight your networks challenge it on the same 300 customers, same seed, same split, live.

🎫 Clear all three quiz questions for the final stamp Live = presented in the 45 minutes Self-study = the full depth, after class

★ What you walk out with tonight. A measured answer to nets-versus-trees on small tabular data - yours, not a blog's. The modality map that predicts the result and its mirror image. And the one-paragraph verdict you can deliver in any architecture meeting without joining a tribe.

LiveWhy losing tonight is the point3 min▶

A deep learning course that only showed you deep learning winning would be an advertisement. The research is unambiguous about where the wins live: Grinsztajn and colleagues showed tree-based models "remain state-of-the-art on medium-sized data (~10K samples)" on typical tables, and Shwartz-Ziv and Armon titled their cross-benchmark study "Tabular data: Deep learning is not all you need". Two hundred training rows of churn is the trees' home ground, and tonight's showdown confirms it with your own clicks.

The same evidence base says the mirror image: on images, speech and language, deep learning did not just win - it ended the previous field. The skill is holding both results and knowing WHY they differ, which is what the modality map is for.

Part 1 · covers the modality map

Inductive bias meets data structure

8 min live

Why do the same networks that lose tonight own vision and language? Because architectures carry promises about structure, and the promise either matches the data or it does not. That single idea organizes the whole map.

Deep learning's home ground Images: nearby pixels relate convolution promises exactly that (s4) Audio and text: order carries meaning sequence models promise exactly that All three: raw signal, no ready features hidden layers LEARN the representation - the 1986 pitch, industrialized The trees' home ground Columns already MEAN something orientation matters; axis cuts fit (s6 of trees) Junk columns are normal trees ignore them free; nets must learn to Truth is steppy, not smooth nets are biased toward smooth solutions - Grinsztajn's three challenges, verbatim One rule covers the map: reach for the architecture whose built-in promise matches the data's actual structure. Raw perceptual signal: the promise-makers win. A table of business columns: the promise is wrong, and honesty about that is cheap.
🔍 Click to zoom - the map: architectural promises versus data structure
LiveThe day the map became obvious: AlexNet, quoted precisely4 min▶

The clearest historical evidence for the left column is 2012. Krizhevsky, Sutskever and Hinton's ImageNet entry - a deep convolutional network trained "between five and six days on two GTX 580 3GB GPUs" - achieved, in the paper's own words, "a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry".

Two precision notes that keep the story honest: the 15.3 belonged to their multi-network ensemble entry, with a single network at 18.2 on validation - so say "their entry", not "the network". And the gap is over ten points, about 40 percent relative - dramatic enough without the "halved the error" inflation it usually gets. The runner-up was the best of classical hand-engineered features; the winner learned its features from pixels. The left column of the map, decided in one table.

The representation thread, tied off. Session 2 quoted the 1986 claim that hidden units "come to represent important features of the task domain". AlexNet is that sentence at industrial scale, and session 4's embeddings are it as a product. Deep learning's real product was never predictions - it is learned representations, which is precisely what a six-column table does not need.

Self-studyWhere the boundary blurs, honestly3 min▶
  • Scale moves the line. Grinsztajn's scope is explicit: medium-sized, around 10K samples. With millions of rows or heavy pretraining, the tabular gap narrows - the claim is scoped, not eternal.
  • Hybrids are normal. The 2024 competition record (per the ensemble course's sources) includes gradient-booster-plus-network stacks among winners; the map picks the LEAD architecture, not a religion.
  • Foundation models are probing the right column. The ensemble capstone covers TabPFN's honest two-sided verdict; it applies unchanged here, and it is that course's story to tell.

The disciplined position: the map gives the default; measurement on your own data - tonight's method - settles every specific case.

Build-along

The showdown, live

★ 22 min · everyone builds

Four networks against the booster's benchmark. The data generator below is copied byte for byte from the ensemble course's engine - same seed, same 300 customers, same 200/100 split - so this is the rare model comparison with literally zero data excuses. Every row trains its network live when you press run.

THE EXPECTED FINISH ORDER, HOLDOUT % · IF YOUR RUN DIFFERS, RE-CHECK WHICH ROW YOU PRESSED logistic regression 64.0 · the linear floor MLP 1x8 65.0 · one point of bend MLP 2x16 60.0 · train 100: memorized 200 rows MLP 3x32, overparam 68-70 · biggest net, best net ◆ tuned booster (bench) 73.0 · the trees hold the table Same 100 held-out customers judge every row. The 2x16 dip and the 3x32 recovery are both real, and both teach.
🔍 Click to zoom - the expected order, so your live run has a reference

Run logistic regression first: 67.5 train, 64 holdout. No hidden layer, so this is the linear floor - and note how respectable it is. Six meaningful columns take a linear model a long way, which is already the right column of the modality map speaking.

Run the 1x8: 65 holdout. One hidden layer buys one point over linear. The bend that cracked XOR wide open barely matters here, because this table's truth is mostly additive with a couple of steps - structure the trees exploit natively.

Run the 2x16 and read both numbers: train 100, holdout 60. The net memorized 200 rows perfectly and got WORSE than the linear floor on unseen customers - session 5's overfit diagnosis, live. On 200 noisy rows, 600-plus parameters of smooth flexibility is a liability.

Run the 3x32 overparameterized, trained long: 68 to 70, depending on your browser. Bigger than the memorizer, and BETTER - the strongest net on the board; session 5's double-descent teaser is the reading. The range is itself a lesson: 1,500 epochs of a chaotic training run amplify the tiny differences between browsers' floating-point math libraries, so the same seeded code lands a point or two apart across engines. Determinism has layers. And on every engine, it sits under the booster.

Now read the bench row it all answers to: 100 / 73.0. The ensemble course's tuned booster, on this data. Best net 68-70, booster 73 - Grinsztajn's finding, reproduced on your own screen: trees remain state of the art on typical medium-sized tabular data. Write both numbers down; they are your evidence, not anyone's slide.

Deliver the verdict paragraph out loud, once: "On our small table the tuned booster wins 73 to 68-70, consistent with the published benchmarks. The same network family wins outright on images and language because there its architectural promises match the data. We route by structure, and we re-measure when the data changes." That paragraph, with your numbers in it, is the course.

What the showdown does NOT say. It does not say nets are weak - the 3x32 losing by five on 200 rows of churn is the same family that ended classical computer vision. It says architecture is a promise about structure, and promises are checked against data, not against enthusiasm.

You ran the full showdown yourself and wrote down both headline numbersYour 3x32 number versus 73, from your own browser, not from a blog.
You can state the modality map's one rule and apply it to a new dataset in a minuteMatch the architecture's promise to the data's structure.
You can explain the 2x16's collapse AND the 3x32's recoveryMemorization on 200 rows; the overparameterized regime differs - and both still lose to the trees here.
Your AlexNet story survives a pedantTheir ENTRY hit 15.3 top-5 vs 26.2; single net 18.2 val; two GTX 580s, five to six days.
You know which claims are scoped and can recite the scopeTrees win on TYPICAL, MEDIUM-SIZED (~10K) tabular data - and scale moves the line.
Part 2 · covers the standing decision

The default, the challenger, the re-measure

5 min live

The showdown method - shared data, live fits, written numbers - is worth more than tonight's result, because results expire and methods do not. Three standing rules turn it into practice.

1 · Default by modality perceptual signal: deep nets business tables: trees the map picks the opener 2 · Challenge on YOUR data shared split, fixed seed, both families, numbers written an afternoon, settles a year 3 · Re-measure on change 10x more rows, new features, pretraining available - the boundary is scoped, so redo it No tribal loyalty survives contact with rule 2. That is the rule's entire job. Tonight was rule 2 executed once. The habit is executing it every time the stakes justify an afternoon.
🔍 Click to zoom - the three rules that outlive tonight's numbers
LiveThe arc, closed3 min▶

Session 1: three numbers and a squash, and the theorem read closely. Session 2: every weight got its blame, and the history got its correction. Session 3: the loop closed and the spiral separated capacity from optimization. Session 4: architectures as promises - convolution's locality, residuals making depth trainable, embeddings as the product. Session 5: the net tamed when it memorized. Tonight: the promise checked against a table, and the honest loss recorded next to the honest reasons.

You built the thing, you know why it wins where it wins, and you measured where it does not. That combination - construction plus calibration - is rarer than either half, and it is what this course was for.

Homework

After the course

45 min
  1. Run the showdown on real data. One labeled table from your work, 80/20 split, fixed seed: logistic, a small MLP, a tuned gradient booster (the ensemble course's homework built one). Write the three numbers and the winner.
  2. Write your modality inventory: the data types your team actually handles, each mapped to its default architecture with one sentence of why. This document ends most future architecture debates before they start.
  3. Rebuild tonight's verdict paragraph with YOUR numbers from item 1, and deliver it to a colleague. If it survives their questions, you own it.
  4. Watch one Karpathy Zero-to-Hero lecture - the micrograd episode. You will recognize everything, which is the point: you built it on paper first.
  5. Set the re-measure trigger: a calendar note at the next 10x data milestone or new-modality project, quoting rule 3. The boundary is scoped; keep it measured.
Check yourself

Three questions

Clear all three for the final passport stamp.

1 · The 2x16 network hit train 100 but holdout 60 - below even logistic regression - while the bigger 3x32 recovered to 68. What is the honest reading?

Both results are real and reproducible - the seed is fixed. The dip-then-recovery with growing size is the shape Belkin and colleagues named; the ceiling below the booster is the shape Grinsztajn measured. Two famous findings, one leaderboard, your browser.

2 · What makes tonight's nets-versus-trees comparison unusually trustworthy?

Most model-war anecdotes compare different data, different splits, different tuning effort. Tonight removed every excuse except the models: shared generator, shared split, fixed seeds, tuned representatives on both sides. Small, yes - the scope caveat stands - but clean, and clean beats big for teaching what a fair comparison even looks like.

3 · A startup pitches you: "our churn model uses deep learning, the same technology as ChatGPT, so it beats classical methods." Your course-grade response?

Technology pedigree is not performance. The modality map plus rule 2 turns the pitch into a checkable claim: shared split, both families, written numbers. If their net genuinely beats a tuned booster on churn-shaped data, excellent - that is a measurement, and measurements outrank both slogans and tonight's result.

Sources covered

What this session teaches, and where it came from

✓Grinsztajn et al. 2022The scoped state-of-the-art claim and the three challenges, reproduced live on shared data
✓Shwartz-Ziv & Armon 2022The title claim, abstract-level - deep learning is not all you need on tables
✓AlexNet, NIPS 201215.3 vs 26.2 quoted from the PDF, with the ensemble-entry and stale-abstract precision traps carried
✓The showdown's measured canon64 / 65 / 60 / 68-70 vs the booster's 73, re-fitted live in this browser; the deep run's 68-70 range is cross-engine float variation, taught as such
◐TabPFN and tabular foundation modelsThe ensemble capstone owns that verdict; it applies here unchanged and is not re-taught

Every fact on this page, and its verification tier, is recorded in the course's source map.

Session 6 cheat sheet · pin this

The showdownLogistic 64, 1x8 65, 2x16 60, 3x32 68-70 - booster 73. Byte-identical data, zero excuses.
The map's one ruleReach for the architecture whose built-in promise matches the data's structure.
AlexNet, preciselyTheir entry: 15.3 top-5 vs 26.2 runner-up; single net 18.2 val; two GTX 580s, five to six days.
The dip and the recovery2x16 memorizes and sinks; overparameterized 3x32 recovers - and still loses here. Both lessons are real.
Every claim, scopedTrees win on typical, medium-sized (~10K) tabular. Scale moves lines - re-measure on change.
The three rulesDefault by modality. Challenge on your own data. Re-measure when the data changes.