Slice, augment or collect

A slice is any subset that shares a characteristic you can name: one digit, one writer, one camera, one country, one customer tier. The MIT notes score it by the average loss of held-out examples inside it, which here becomes the share of each digit's training rows whose four-fold, out-of-sample neighbours agree with the label. The 9s come out at 57 of 71, 80.3 percent, then the 4s at 85.2, the 8s at 86.0 and the 1s at 87.7. Those are exactly the digits the declared plant confused, so the slice table is telling the truth about where the data is still thin or still muddled.

The tempting shortcut is to print per-digit test accuracy and pick the lowest. The MIT rule is blunt: evaluation examples "should only be used for computing evaluation scores, not for any modeling decisions." Choosing which slice to edit is a modelling decision. Make it on test, and the next test score is partly a record of your choice rather than a measurement of it; with 65 test 9s, a handful of lucky images can steer the choice and then flatter it. This is evaluation leakage into a decision, a cousin of the feature leakage in Feature Engineering ↗, not the same thing.

Digits are a slice you can name in advance. The ones that hurt in production are usually the ones nobody named: a scanner model, a lighting condition, a dialect. The MIT notes give a procedure for finding them: sort the held-out examples by loss, take the high-loss end, and cluster it. A cluster that keeps appearing is a candidate slice; you then look at its members and give it a name a person can act on.

Two cautions carry over from tonight's setup. The "held-out" examples must be out-of-fold training rows, not the locked test set, for the same reason as above. And a cluster of high-loss rows may be a cluster of wrong labels rather than a thin slice, which is why the relabel pass comes first. Reading a hundred of those rows by hand and tagging each one is error analysis, and its method lives in ML Strategy p3 ↗.

Once a weak slice is found, the MIT notes list four remedies: use a more flexible model; over-sample or up-weight the minority subgroup; "Collect additional data from the subgroup of interest"; and engineer features that help the model tell the slice apart. The first is model-side, and here the model is frozen. The last is feature work, which changes what the model sees rather than which rows it learns from, and it belongs to Feature Engineering ↗.

That leaves the two data-side remedies, and the rest of tonight is about them. Over-sampling a slice with transformed copies is Part 2. Collecting new rows, from the slice or in general, is Part 3. Class imbalance, outliers and distribution shift have their own MIT lectures; they are named here and not taught.

Augmentation fixes coverage. It helps when the slice is thin, when its real variation is something a transform can imitate, and when new real rows are slow or expensive to get. It does not fix labels: every copy inherits the label of its source. On this seed the shift of the 9s bought two test digits, 478 to 480, and the 9s themselves went from 56 to 59 correct of 65 when the test set was finally read. The overall total rose by only two while the 9s rose by three, so the other digits lost one between them: an extra slice's worth of rows moves the boundary for its neighbours too.

Then the honest dip. Over the twenty-seed sweep, relabel plus shift averages 94.52 against 94.45 for relabel alone: +0.07 points, about a third of one test digit. By the time the 9s are shifted, the main problem on this bench, wrong labels, has already been dealt with, and a one-pixel shift at 8 by 8 is a four-pixel shift of the original 32 by 32 scan, a coarse move that adds little new coverage. The +0.4 on one seed is two digits; the +0.07 is the claim.

Before any transform ships, write the invariance as a sentence and try to break it. "A digit shifted by one pixel keeps its label" holds for every digit here. "A digit flipped left to right keeps its label" fails for 2, 3, 4, 5, 7 and 9. "A digit rotated 180 degrees keeps its label" turns a 6 into a 9. Mixup, in the MIT notes, is the case that proves the rule by breaking it on purpose: it blends two inputs and blends their labels in the same proportion, because the blended image no longer belongs to either class.

The test is one line per transform in the edit log: the invariance claimed, one input where it fails, and whether that input can occur in the slice you are augmenting. If you cannot name a failure case, you have not thought about the transform long enough.

The order of edits matters. Tonight the shift runs on the relabelled 800, where the 9s are mostly real 9s. Run it on the given labels instead and the look-alike 4s that the declared plant labelled 9 would be shifted too, four copies each, every copy carrying the wrong label into a denser patch of the training set. Augmentation multiplies whatever the labels say. Clean first, then augment, and write that order into the spec in session 6.

The same logic applies to generative augmentation. A generator trained on noisy labels produces plausible new rows with the same confusions, only more of them.

Three routes, printed by the build-along below. Collect the 497 pool rows and use their labels as given: 454 correct, 90.8 percent, below the 457 baseline. Collect them and run confident learning over all 1,297: 478, 95.6 percent. Leave the pool alone and clean the 800: 478, 95.6 percent. On this seed, cleaning the 800 matched collecting 62 percent more rows and cleaning all of them.

The twenty-seed sweep says the same thing with more authority than one seed can. More rows with the same noise average +0.25 points over the baseline, and are worse than it in 8 of the 20 seeds. Collect-and-clean averages 94.60 against 94.45 for relabelling the 800 alone, a gap of under one test digit. Ng's slide said cleaning and doubling were about equally effective; here, with systematic noise, collecting without cleaning bought roughly nothing, and collecting with cleaning bought roughly what cleaning alone did. That is one bench and one kind of noise, not a law. The sweep is what you would run on your own data before quoting either.

One more knob, from the session 4 bench: after relabelling, drop the ambiguous rows, the ones where no digit wins a majority of the ten out-of-fold neighbours. On the 800 that drops 110 rows and scores 475, 95.0 percent, worse than the 478 of relabelling alone; the twenty-seed mean agrees, 93.61 against 94.45. On the 1,297 it drops 143 rows and scores 483, 96.6 percent, the best score on the bench. The sweep did not run the 1,297 version, so read that 483 as one seed.

The mechanism is scarcity. Ambiguous rows sit on the boundary between digits, which is exactly where a nearest-neighbour model needs examples. With 800 rows, pruning 110 of them removes evidence the model cannot replace. With 1,297, there are enough clean boundary rows left that removing the muddled ones sharpens the boundary instead of thinning it. Ng's long-tail point is the same rule at scale: big-data problems with a long tail of rare events "are also small data problems" in the tail, and a pruning rule tuned on the head can starve the tail.

The slide states the 500-example, 12-percent case as reasoning, and the curve on the next slide, clean against noisy labels from 250 to 1,500 examples, names no dataset. So it is a claim to test, not a result to cite. Two things the bench adds: the noise here is systematic, look-alike swaps drawn from the images that most resemble the other digit, which is the kind that more data reinforces rather than averages out; and the collection route only matches cleaning when the collected rows are cleaned too. If your noise is closer to random, new rows may do more than they did here. That is the reason to measure, not to assume.