The data-edit bench

Test accuracy is the share of the 500 test digits the frozen model gets right after the edit. One test digit is 0.2 points, so a difference of one or two digits on a single seed is not a finding. The readout also prints the difference against the baseline, and the ceiling: the same model trained on the true labels gets 489 of 500, 97.8 percent. That ceiling is knowable only because the plant is declared.

Training rows says how many rows the edit left. Removal shrinks the set; relabelling keeps it; augmentation and extra data grow it. Reading accuracy without this number hides what an edit cost.

Per-digit test accuracy sits under the readout, with the change against the baseline for each digit. It is there to be read, never to choose an edit: session 5 picks the weak digit on the training set.

Detector precision is the share of flagged rows that really were planted errors. Recall is the share of the 133 planted errors the detector flagged. Wrong labels left counts training rows whose label still differs from the true digit after the edit, including any a relabel got wrong. Accuracy can saturate while these keep moving, which is why the bench reports both: a detector that finds a third of the errors can still lift accuracy, and you would never know it had missed two thirds.

Under the readout the bench prints where the flags go, the off-diagonal of the confident joint, and the per-class thresholds, then draws every flagged row from its own 64 pixels with a rust ring when it was a planted error and a grey ring when it was a false alarm. The last gallery is the planted errors the detector missed. Look at them: they are the 7s that look most like 1s.

Confident learning flags 98 of the 800 rows. 85 of them are planted errors, a precision of 86.7 percent, and it finds 85 of the 133, a recall of 63.9 percent. Removing the 98 takes the frozen model from 91.4 to 95.8 percent; relabelling them to the suggested digit takes it to 95.6. Every one of the 85 caught errors is relabelled to its true digit; the 13 false alarms are relabelled wrongly. Over twenty seeds removal averages +4.20 points and relabelling +4.48, and neither is ever worse than the baseline. On this seed removal edges ahead by one test digit; on the mean relabelling does. Call them about equal, and choose by what the removed rows would have cost to collect.

Dropping ambiguous rows after relabelling, the 110 rows whose ten neighbours reach no majority, leaves only 23 wrong labels in training, the cleanest set on the ladder. It scores 95.0 percent, below plain relabelling at 95.6. The ambiguous rows are mostly correctly labelled digits that sit near a boundary, and near-boundary rows are exactly what a nearest-neighbour model needs to place the boundary. Fewer wrong labels is not the goal; a better model on the test set is. Try the same knob on the 1,297 rows in the knobs panel and it helps instead, which is session 5's small-data lesson.

On this seed, adding four one-pixel shifts of every training 9, the digit the relabelled training set agrees with least under cross-validation, takes the model to 96.0 percent, the best preset, and lifts test 9s from 56 to 59 of 65. Over twenty seeds the same edit averages +0.07 points over plain relabelling. That is the measurement: augmentation adds coverage of an invariance, and on 8 by 8 digits whose main problem is wrong labels, coverage is rarely the bottleneck. A one-pixel shift at this resolution is also a four-pixel shift of the original 32 by 32 scan, a coarse transform. Session 5 is about when augmentation is the right edit.

The plant is systematic. The 7s labelled 1 are the 7s that look most like 1s, and the pool's 7s labelled 1 are the same kind of 7. Adding them puts more wrong votes in exactly the neighbourhoods where the model was already confused. The bench shows 457 correct becoming 454, and the wrong labels in training rising from 133 to 219. Over twenty seeds the change averages +0.25 points and is worse than the baseline in 8 of 20: "about nothing", with the sign decided by the draw.

The honest nuance: this is not a law that more data never helps. With purely random noise, more rows do average the errors out, and the homework in session 1 asks you to see it. And more data that is cleaned too, the last preset, reaches 95.6 percent. The trap is more data from the same process, unexamined.

Confident learning needs out-of-sample predicted probabilities; the paper fixes four folds of cross-validation, and the MIT notes say "We use cross-validation to obtain predicted probabilities out-of-sample." With in-sample votes each training row is its own nearest neighbour at distance zero, so every row casts one vote for its own label before anyone else votes. The detector flags only 42 rows, 95.2 percent of them real errors, and finds 30.1 percent of the planted errors against 63.9 out of sample. Precision went up while recall halved, and a team reading only precision would call it an improvement. Accuracy lands at 94.8 against 95.6.

This is detector leakage, the row seeing its own answer, and it is a different animal from feature leakage, where a column carries the target. Feature leakage belongs to Feature Engineering ↗.