Learning curves, read honestly

Chapter 28 of Machine Learning Yearning is titled "Diagnosing bias and variance: Learning curves". The construction is simple enough to do in an afternoon. Take the training pool. Train on 10 percent of it, then 25, then 50, then all of it. After each run, score two things: the error on the training examples that run actually saw, and the error on the fixed dev set, which never changes. Plot both against training set size, on the same axes. Draw the human-level error rate as a horizontal line across the whole chart.

Three lines, one picture. The dev curve falls as data grows. The training curve rises as data grows, because a model finds it harder to fit 38,000 frames well than 3,800. The human-level line does not move, because a person's error rate has nothing to do with how much data your model saw. Where those three end up relative to each other is the diagnosis.

New readers usually expect both curves to fall. Training error rises with training set size, and the reason is worth holding onto:

• At 100 examples a model of any real capacity can nearly memorise the set. Training error is near zero and means nothing about the world.
• At 38,000 examples the same model has to find structure that generalises across all of them, and it will get some wrong. Training error climbs toward the error rate the model is actually capable of.
• The gap between the two curves is the variance. It starts wide, because memorising is easy and generalising is not, and it narrows as data grows.
• The height the training curve settles at is the bias. More data cannot push it below the error the model class can reach on this problem.

So the two curves converge from opposite directions, and the level they converge toward is the thing worth predicting.

If the dev curve is visibly still coming down between 50 percent and 100 percent of the training pool, the model has not yet extracted everything the data class has to give. Doubling the pool will move the number, and roughly by as much again as the last doubling did, which is what makes the slope worth writing down as a number rather than describing as "still dropping a bit".

The strong version of this reading is comparative. If the last doubling of data bought 1.4 points of dev error and the next doubling costs 152,000 pounds, that is a price per point, and it sits in the same spreadsheet as every other project with a ceiling and a cost. Session 3's points-per-week column is the same arithmetic wearing a different unit.

When the dev curve has levelled and the training curve has levelled just below it, both far above the human-level line, the model is failing on examples it has already seen. No quantity of new examples fixes that. This is avoidable bias, and the honest response is the one from session 5: a model with more capacity, a longer training schedule, a better input representation, or a feature the current model has no way to see.

The trap is that this shape looks disciplined. Two smooth flat lines, a stable number, a team that has "converged". A converged model well above human-level has converged on the wrong thing.

A training curve hugging the bottom of the chart while the dev curve sits ten points above it is variance, drawn rather than inferred. The model can fit what it has seen and cannot carry it across to what it has not. More data is the direct fix, and the curve tells you it will work because the gap has been narrowing over the range you plotted.

Read the slope before committing. A gap that is narrowing quickly says the next tranche of frames is well spent. A gap that has barely moved between 50 and 100 percent says the model is memorising faster than you can buy examples, and augmentation or regularisation is the cheaper route to the same place.

Four practical rules, and skipping any one of them produces a chart that is worse than no chart because it looks authoritative:

• Plot training error on the same axes. A dev-only curve cannot distinguish bias from variance, and those two readings fund opposite projects.
• Sample the smaller training sets randomly, not by taking the first N rows. A frame table ordered by camera or by capture date gives you a subset that is one forest in one season.
• Average several random draws at each size. Three is the working minimum at the small end. One draw at the small end is a coin flip presented as a data point.
• Keep the dev set fixed across every point on the curve. It is the one thing that must not vary, or the curve measures two changes at once.

Training error at each point is scored on the subset that run actually saw, not on the whole pool. Scoring the small runs against the full training pool turns the training curve into a second dev curve and the diagnosis disappears.

The left-hand points on a learning curve are the least reliable and the most visually influential, because they set the apparent steepness of everything to their right. A model trained on 10 percent of a pool is at the mercy of which 10 percent: one draw lands three good night cameras, the next lands a month of rain. Lanternfish's three draws at 10 percent came back at 25, 22 and 14 percent dev error. An 11-point spread, on a curve whose entire story from left to right is 8 points.

Averaging three draws is the working minimum and five is better at the smallest size. The cost is real, because it is three trainings instead of one at every point, but the small runs are also the cheap ones, so most of the extra compute buys you the part of the chart that most needs it.

Here is the sentence teams say: "the curve is flat, so more data will not help". Here is the sentence the curve supports: "the curve is flat between 19,000 and 38,000 examples, so another 38,000 would not have helped much". Those are different claims. The plot can only speak about the range you plotted. A curve that has levelled over a doubling can still be falling over a tenfold increase, and plenty of them are.

The reason this matters is that the flat reading closes a door. Once "more data does not help" is in a deck, nobody proposes data again for a year, including the person who could have got 400,000 frames from a partner network for nothing. Write the range into the sentence and the door stays open at the right price.

A dev error of 12 percent means nothing on its own. If a trained field biologist scores 2 percent on the same frames, 12 percent is a large gap and there is a lot of headroom. If a biologist scores 11 percent, because half the frames are a tail in the dark at forty metres, then 12 percent is close to the ceiling and the interesting question is no longer the model at all.

Drawing that line across the chart converts a curve into a diagnosis. It shows how much of the remaining error is even addressable, and it stops a flat curve at 12 percent from being read as failure when it is close to what the task allows. Lanternfish's human-level number is 2 percent, measured by having two biologists label 200 dev frames independently, which is a morning of work and the highest-value morning on the whole project.

• Training error 3, dev error 12, human-level 2. The training curve sits one point above human-level and the dev curve sits nine points above the training curve.
• Avoidable bias 1. The training curve is nearly on the floor the task allows. There is almost nothing to win by making the model fit its own data better.
• Variance 9. The whole visible gap on the chart is between the two curves, which is the shape in the third panel above.
• So the diagnosis is variance, and more data is a live option rather than a reflex.
• The slope at the right edge decides which kind of fix. Steep, and more camera frames are the cheapest point of error on the table. Flat, and the money goes to occlusion augmentation or a regularisation pass instead, both of which attack the same variance without buying a single label.

Note what the curve does not settle: whether the training pool and the dev set are even the same kind of image. Lanternfish trains on web photos as well as camera frames, and that turns part of what looks like variance into something else. Session 7 splits it.