Where tonight sits in the track
Session 1 built the instrument: 1,000 frames drawn from the live cameras, split into an Eyeball half and a Blackbox half. Tonight builds the first thing that instrument measures. Not the best classifier the team can imagine, the fastest one that runs end to end, so that by the end of the week there are 120 real mistakes in a table and the team can stop speculating about what is wrong with a model that does not exist yet.
★ What you walk out with tonight. A baseline built in an afternoon with a pretrained backbone and default hyperparameters. A dev_errors table with one row per wrong dev frame, its prediction, its label and an empty category column. The number every later project is priced against: 88 percent on the honest dev set, 120 errors, 12.8 percent on the Eyeball half and 11.2 on the Blackbox. And the habit of looking at ten wrong frames before forming a single opinion about the fix.
LiveThe first system exists to produce errors3 min▶
A team starting a new classifier knows almost nothing about where it will fail. Will night frames be the problem, or two species that look alike, or empty frames, or labels the field team got wrong? Every one of those is plausible before the first model runs, and every one of them has a champion on the team. The only thing that settles the argument is a dev set with mistakes on it, and the only way to get one is to build something, anything, and score it.
So the first system is optimised for one property: it finishes. It is measured on the real dev set from day one, and its errors are written down as it makes them. Whether it is good is almost beside the point. Its 120 mistakes are the first evidence the project has ever had, and the value of the week is that evidence, not the model.
Lanternfish's founding engineer wanted to get the architecture right first: a custom two-stage detector with a species head, a bespoke augmentation pipeline, a training loop written from scratch. Six months later it was elegant, it trained, and nobody could say which kind of frame it got wrong, because nobody had looked. Scored on the honest dev set in session 1 it came in at 88 percent. A fine-tuned pretrained backbone gets there in an afternoon.
Self-studyWhy speed of iteration decides projects3 min▶
- Every idea is a loop iteration. Build, measure on the dev set, decide. A team that closes the loop weekly tries fifty ideas in a year; a team that closes it quarterly tries four.
- The first iteration is the slowest. Data pipeline, training loop, scoring script, error table all have to exist before the loop turns once. Everything that follows reuses them.
- Most first ideas are wrong. Not the team's fault; nobody can know the error distribution before seeing it. A fast first system makes being wrong cheap.
- Speculation compounds. A design chosen without dev-set evidence is a design whose flaws will also be discovered without evidence, later, at higher cost.
- The exception is real but rare. A domain the team has shipped in many times, where the error categories are already known from the last project, can justify a longer first build. New domain, new camera, new species list: build fast.
Build your first system quickly, then iterate
9 min liveChapter 13 of Machine Learning Yearning is titled "Build your first system quickly, then iterate", and the title is the whole instruction. The reasoning underneath it is about information: until there is a dev set with mistakes on it, every design argument is speculation, and the fastest way to replace speculation with evidence is to ship something that makes mistakes on the real dev set.
LiveUntil there are errors, every design argument is speculation3 min▶
Consider the five opinions on the Lanternfish team about why the classifier is wrong: night blur, deer that look like elk, animals half out of frame, empty frames read as animals, wrong labels. Before a dev set exists with mistakes on it, all five are equally defensible, and the loudest one wins. After the first system runs, they are counts: 42, 24, 22, 18 and 14 out of 120. The argument does not end because someone was persuasive; it ends because someone looked.
This is why the baseline is measured on the real dev set from its first run rather than on a training-set holdout or a demo folder. A mistake on the dev set is evidence about the field. A mistake anywhere else is evidence about somewhere else.
LiveWhat "quickly" means in practice3 min▶
Quickly is not sloppy; it is a set of decisions made in favour of finishing:
- A pretrained backbone, fine-tuned, not an architecture designed for the problem. The backbone already knows what fur and antlers look like.
- Default hyperparameters. The library defaults were chosen by people who ran thousands of experiments. Tuning them before knowing what the model gets wrong is tuning blind.
- The simplest data pipeline that runs. Resize, normalise, done. Augmentation is a project with a ceiling, chosen in session 3, not a default.
- One week at most. If the first system is not scoring on the dev set by the end of the week, the plan was too ambitious for a first system. Cut scope until it finishes.
- Errors written to a table on the first scoring run. Not a log file, not a notebook cell. A table the next session can query.
The Lanternfish baseline that replaced the six-month architecture was under forty lines: a ResNet from a model zoo, three epochs of fine-tuning on the labelled camera frames, a scoring loop over the 1,000 dev frames. It ran on a laptop overnight. The next morning the team had 120 rows to disagree about instead of five theories.
Self-studyThe baseline is the number everything else is priced against3 min▶
Once the first system has scored, one set of numbers becomes the reference for the rest of the project:
| Number | Lanternfish baseline | What it is used for |
|---|---|---|
| Dev accuracy | 88 percent, 1,000 frames | The headline. Every project promises to move it. |
| Dev errors | 120 | The rows of the session 3 spreadsheet. |
| Eyeball half | 64 of 500 wrong, 12.8 percent | The half you look at. The ceilings are computed on it. |
| Blackbox half | 56 of 500 wrong, 11.2 percent | The half you never look at. The number you report. |
A project's ceiling in session 3 is a share of these errors. Its cost is weeks. Its points per week is ceiling divided by cost. None of that arithmetic can start until the baseline exists, which is another way of saying that a slow first system delays every good decision after it by exactly as long as it took.
What the first system is allowed to be bad at
8 min liveA baseline is allowed to be embarrassing. It is not allowed to be unmeasured, and it is not allowed to have been fixed before anyone counted what was wrong with it. The difference between a baseline and a prototype is entirely about what it is measured on.
LiveA baseline is not a prototype3 min▶
Both are built fast. The difference is the audience. A prototype is built to be shown: a demo folder of good-looking frames, a stakeholder, a confidence threshold nudged until the output looks right. A baseline is built to be measured: scored on the real dev set from session 1 on its first run, with every mistake recorded, and never tuned to look good on the frames someone happened to be watching.
The failure mode is a prototype that gets promoted to a baseline. Its number was produced on frames chosen to flatter it, so every later project is priced against a fiction, and the first honest dev-set score arrives as a shock months later. Lanternfish's 94-percent web-photo number was exactly this.
LiveFixes made before measuring are fixes chosen by conviction3 min▶
The first scoring run makes the temptation acute. Ten wrong frames are on screen, three of them are night blur, and someone already knows a denoising trick. Two days later the denoiser is in and the baseline has never been recorded. The problem is not the denoiser; it may well be a good project. The problem is that it was chosen on three frames by the person who happened to know a trick, and nobody can now say whether night blur was 33 percent of the errors or 5.
The discipline is simple and slightly painful: record the baseline, write the errors to the table, and make no fix until session 3's spreadsheet has put a share and a ceiling on every category. Every fix made before that point is a fix chosen by conviction rather than by count, and conviction is what the whole spreadsheet exists to replace.
Lanternfish's team wanted the deer-versus-elk specialist model the moment they saw their first misread elk. On the spreadsheet a week later that category was 19 percent of the Eyeball errors and the specialist model offered 0.21 points per week, the lowest on the sheet. The re-labelling job nobody had proposed was 14 percent of the errors and 1.62 points per week.
Self-studyThe errors table is the first system's real deliverable3 min▶
The model file will be replaced within weeks. The table it produced will be queried for the life of the project. Give it the shape session 3 needs:
- frame_id - the key back to the image, so a human can open it. An error you cannot look at is an error you cannot categorise.
- half - eyeball or blackbox, carried over from the session 1 split. Session 3 only reads the Eyeball rows; session 4 explains why.
- predicted and actual - the model's answer and the label. The pair is often the category: elk predicted for deer is one row of a pattern.
- category - null. Every row. A human fills it in while looking at the frame, and the fact that it is empty tonight is the point: nothing has been decided yet.
- One row per wrong frame, nothing else. Correct frames belong in the scoring log, not here; the spreadsheet is about mistakes.
A baseline that produced a number but no table has done half the job. The number says how wrong the model is; the table says how, and only the second one funds a project.
Self-studyThe fast first system will sometimes be embarrassing2 min▶
A pretrained backbone with default settings will call a badger a fox and a moving branch a boar. Somebody will screenshot it. The reflex is to hold the baseline back until it is presentable, and the reflex is wrong: every embarrassing error is a row in the table, and the pattern behind the embarrassment is a project with a ceiling. Lanternfish's 18 empty frames called animals looked ridiculous in a demo and turned into a two-week motion filter worth 1.6 points, the second-best points per week on the sheet.
The embarrassment is information. A first system that is never embarrassing was either tuned before it was measured, or measured on frames chosen to spare it.
The baseline in an afternoon
★ 20 min · everyone buildsBring the dev set from session 1 and any labelled training frames you have. If you are on the Lanternfish exercise pack, it ships the labelled camera frames, the 1,000-frame dev set with its dev_half column, and the baseline's frozen predictions, so everyone's table lands on the same 120 rows. If you fine-tune your own backbone instead, your count will differ by a few frames and every step below still works.
Fine-tune a pretrained backbone with defaults. A model-zoo ResNet, the default optimiser, three epochs, no tuning. If you find yourself opening the hyperparameter documentation, close it. That is a session 5 decision, and it will be made against a number.
Score it on the 1,000-frame dev set and write every mistake to a table. One row per wrong frame: frame_id, half, predicted, actual, and a category column that is null in every row. Do not fill the category in tonight, even when it is obvious.
Confirm the counts per half. The split from session 1 was random, so the two halves will not carry the same number of errors. Lanternfish's do not: 64 in the Eyeball half and 56 in the Blackbox. Write both down; the gap between them is a fact about sampling, not about the model.
Record the baseline where the team will see it. Accuracy 88 percent, 120 errors, 12.8 and 11.2 by half, the date, and the model file's hash. Every project from session 3 onward is priced against this row, and a baseline that lives in one person's notebook is a baseline the team will argue about later.
Open ten Eyeball errors in a viewer and look at them. Not categorise, look. Notice what you notice: the grain on the night frames, the antlers hidden by a branch, the frame with nothing in it that the model called a boar. Write nothing in the category column. The looking is practice for session 3; the writing is session 3.
Ten frames is enough to feel the pull. Three of the ten will be night blur and someone will know a fix. That pull is exactly what the spreadsheet in session 3 is for: it turns "I saw three" into "it is 33 percent of the errors and worth 0.84 points a week", and the fix gets funded on the second number, not the first.
Before session 3
30 min- For one classifier you own, find its baseline: the first honest dev-set number, the date, and the model it came from. If nobody can produce it, the project has been pricing its work against a number that does not exist.
- Check whether your baseline was measured on the real dev set or on something friendlier. A demo folder, a training holdout or a hand-picked sample makes it a prototype number, and the first honest score is still ahead of you.
- List every fix made to the model before anyone counted its errors by category. For each one, write down who chose it and on how many examples. Most are one person and under ten.
- Build your own
dev_errorstable with the five columns from tonight, and bring the Eyeball rows to session 3 with the category column still empty.
Three questions
Why build the first system in a week with a pretrained backbone and default hyperparameters, rather than designing the right architecture first?
Chapter 13: build your first system quickly, then iterate. The first system's job is to produce the errors that decide what to build next, not to be good.
Two teams each build a classifier in a week. One scores it on the real dev set and writes the errors to a table; the other tunes the threshold until a demo folder looks right. Which has a baseline?
Speed is necessary but not sufficient. What makes a baseline is what it is measured on. A prototype promoted to a baseline prices every later project against a fiction.
The Lanternfish baseline has 64 errors in the Eyeball half and 56 in the Blackbox half. What does the gap tell you?
12.8 and 11.2 percent are both readings of the same model on two random halves of one dev set. The gap is sampling, and writing both down is what lets session 4 spot a gap that is not.