Leader track 🧭 · six sessions, no code
For whoever funds the project and asks why it is slow. What decides velocity, which metric to insist on, how to read an error spreadsheet, and the one-page memo.
Strategy is deciding what not to try
Why iteration speed decides projects more than model choice does, build the first system in a week, and the twelve percent that everybody at Lanternfish had an opinion about.
1️⃣ Leader session 2One number, or two kinds
A single-number metric the team can optimise, optimizing against satisficing metrics, and when a metric has stopped measuring what you care about.
🔎 Leader session 3Look at 100 mistakes
Error analysis as a spreadsheet, the ceiling every project has, and the sheet that puts a one-week re-label job above an eight-week model.
⚖️ Leader session 4Where the error comes from
Unavoidable bias, avoidable bias and variance from three numbers, human-level performance as the yardstick, and one knob per problem.
🔗 Leader session 5The pipeline that hides its weakest part
Error analysis by parts: attributing the error to one stage so the team stops fixing the stage that is already fine.
📄 Leader session 6Get your teammates to read this
The one-page ML strategy memo, and where this discipline lands when the model is a language model.
Practitioner track 🔧 · ten sessions, one decision
One classifier, Lanternfish's species model, taken from a dev set that tells the truth to a scored order of work. Session 10 runs the whole decision on the bench.
Dev and test sets that tell the truth
Same distribution as the traffic you care about, sized for the difference you need to see: 100 examples cannot detect 0.1 percent, 10,000 can.
⚡ Practitioner session 2The first system in a week
Build quickly, then iterate. The 12 percent baseline and why waiting for the right architecture costs more than shipping the wrong one.
📋 Practitioner session 3The error-analysis spreadsheet
Sixty-four real errors, five categories, a share column and a ceiling column. The order of work falls out of it.
👁️ Practitioner session 4Eyeball and Blackbox dev sets
Ten percent you look at, ninety you never do. Hand rules against what you saw take one half to 3.8 percent and leave the other at 11.2.
📐 Practitioner session 5Bias and variance from three numbers
Train 3, dev 12, human-level 2: avoidable bias 1, variance 9. Which project each diagnosis funds.
📈 Practitioner session 6Learning curves, read honestly
Plotting training and dev error against data size, and the two shapes that tell you more data will or will not help.
🌍 Practitioner session 7Training and testing on different distributions
When it is right, the training-dev set that separates variance from mismatch, and what to do about mismatch.
🔬 Practitioner session 8The Optimization Verification test
Is the model scoring wrong, or the search failing to find the answer it would score highest? One test tells them apart.
🧩 Practitioner session 9Error analysis by parts
A detector then a classifier. Perfect inputs to each stage, and the error that belongs to only one of them.
🎛️ Practitioner session 10The what-to-fix-next bench
Five projects, two halves of a dev set, your order. Points per week against the interesting project, and the rules that only fix what you looked at.
Choose your path 🗺️
Three doors into the same decision. Both tracks start at their own session 1.
Your passport 🎫
Clear all three quiz questions in a session and it gets stamped. Sixteen stamps across both tracks, and they live in this browser only.
What is real here 🧾
A course about looking at your errors honestly should be exact about its own numbers.
| Claim | Status |
|---|---|
| The 1,000 dev images and 120 errors | Generated once from a fixed seed, identical for every learner. Each error carries its spreadsheet category: night blur 42, deer-vs-elk 24, half out of frame 22, empty frame 18, wrong label 14 |
| Eyeball 12.8 percent, Blackbox 11.2 | The split is by index and the errors were placed at random, so the halves differ: 64 against 56. That difference is the reason to keep two |
| The spreadsheet and its points-per-week column | Counted on the 64 Eyeball errors. Ceiling per chapter 14: share of errors times the share the project removes. Re-label 1.62 per week, specialist model 0.21 |
| Points-per-week order: 6.2 percent by week 6, 3.6 by week 18 | Computed by the bench as each project removes its stated share of its category, earliest-indexed errors first, so the arithmetic is deterministic |
| Most interesting first: 11.2 at week 6, 9.2 at week 8, 3.6 at week 18 | Same arithmetic, different order. Same destination |
| Hand rules: Eyeball 3.8, Blackbox 11.2 | A construction, labelled as one on the bench: rules written against 60 specific examples fix only those examples. Everything else on the bench is counting |
| Nothing models a person | Unlike a simulator, there is no human behaviour to estimate. Every number is a count |
| Machine Learning Yearning | Read from the deeplearning.ai PDF; chapter titles and the quoted passages are in the course map with chapter numbers. Orthogonalization is from Ng's Structuring Machine Learning Projects course and is labelled as such |
| Lanternfish | A fictional wildlife-camera startup. The classifier is Ng's cat detector moved one step sideways |
The knowledge map 🧠
Both tracks at a glance - hover a session to spotlight its concepts, click any node to jump in.