The data-edit spec

Only data edits. Relabel, remove, drop ambiguous rows, add shifted copies, add the 497 extra rows: anything the knobs on the session 4 bench allow, in any combination.

The model and the seed are frozen. kNN with k = 10, squared distance on the 64 raw pixels, seed 20210921 for the split and the plant. If you want to try k = 5, that is a model-centric experiment and belongs in a different document.

The test set is untouched. Not relabelled, not filtered, and not used to pick the edit. Reading per-digit test accuracy after the choice is fine; choosing by it is not.

Every edit is logged. A row that changed without a line in the log is a row nobody can defend. Part 2 is that log.

On 500 test digits, one digit is 0.2 points. So 96.6 against 96.4 is one digit, and one digit on one seed is not a win. It could flip if the split moved. The honest comparison is against the twenty-seed sweep, where it exists: relabel averages 94.45, relabel plus shift 94.52, and collect plus clean 94.60. Those three edits look different on the shipped seed, and over twenty seeds they are within 0.15 points of each other, less than one test digit. Treat them as about equal.

The 96.6 combination beats the best 800-row preset by three digits, 0.6 points, on one seed. That is a promising lead, not a finding: the sweep did not run it. Two combinations tie at 483, the 1,297 rows with relabel and drop ambiguous, and the 1,297 with remove, drop ambiguous and shift. When two edits tie, prefer the one with fewer steps and fewer rows: it is cheaper to explain and to rerun.

Trained on the true labels, the frozen model still misses 11 of the 500 test digits: 489 correct, 97.8 percent. Those misses are not label errors in training, because there are none. They are digits the model cannot separate: a 4 whose ten nearest training images happen to be 9s, a sloppy 8 that sits in the 3s' neighbourhood. No data edit to the labels can fix them. Only more coverage near those shapes, or a different model, could.

That makes the gap you are allowed to close 32 digits, from 457 to 489, and not 43. On this seed the best combination closes 26 of them. Quoting "96.6 against a perfect 100" would overstate what is left for the data to do.

The edit log is the field the other six exist to make trustworthy. One line per row the edit touched: the row id, the old label, the new label or "removed", what the detector said and how strongly, who reviewed it, and which guideline version they reviewed it against. With that line, any single change can be questioned and reversed. Without it, the only way to undo a bad cleaning pass is to throw the whole dataset away.

Here is a constructed sample, four lines long. The rows, labels and votes are real outputs of the session 3 detector on the 800 training digits; the reviewer column and the decisions on the two false alarms are constructed to show what a reviewed log looks like.

Row 63 is the one that matters. The detector suggested 8; the digit is a 3. On the bench, the 13 false alarms among the 98 flags were all relabelled wrongly, because nobody reviewed them. A reviewer column is what turns a detector's suggestion into a decision.

Freeze list. The model and every setting that decides its output: kNN, k = 10, squared Euclidean distance on 64 integer pixels, distance ties to the earlier training row, vote ties to the nearest tied neighbour, seed 20210921, no preprocessing, accuracy as the metric, and a checksum of the test file so anyone can prove it is the same 500 digits.

Test-set rule. Never edited, never used for decisions. The MIT notes: evaluation examples "should only be used for computing evaluation scores, not for any modeling decisions."

Guideline version from session 2, detector and settings from session 3 (four folds, per-class thresholds, out of sample), result table with both counters from session 4, and what not to claim: no one-seed, one-digit wins; no precision or recall quoted from a declared plant as if it were the real noise.

Ng's closing checklist makes a good acceptance test for the finished page. Good data is "defined consistently", has "cover of important cases", has "timely feedback from production data", and is "sized appropriately". The guideline field answers the first, the slice work of session 5 the second, the result table the fourth. The third is production monitoring, and Data Observability ↗ owns it.

A spec is only as good as its link to the exact rows it describes. The simplest working pattern: keep the raw labels read-only, store the edit log as a small table next to them, and build the training set by applying the log to the raw labels. Then the dataset version is the log's version, a diff between two versions is a diff between two logs, and a reviewer can read a change the way they read a pull request.

Three habits make it hold up. Give the edit log a version that moves with the guideline version, so a rule change forces a new log. Record the detector settings in the log header, so a rerun cannot silently use different folds. And never overwrite a label in place: an overwritten label is a change with no author and no reason.

Northcutt, Athalye and Mueller had exactly this problem on ten public test sets: an algorithm flagged candidates, and nobody knew which were real. Their design is the one to copy. Each flagged item went to five crowd workers on Mechanical Turk, and it counted as a label error if fewer than 3 of the 5 agreed with the given label. Across the ten datasets, 51 percent of the flagged candidates were confirmed on average, and the spread was wide: on MNIST, only 15 of 100 flags were real errors. Precision is a property of the detector on this data, not of the method.

The build-along runs the small version. From the 98 flags, a random sample of 20 with seed 6: 17 were real errors, an estimate of 85 percent. On the bench we can compare it with the true precision, 86.7 percent, because the plant is declared. At work you only have the 85. With 20 items the estimate is rough: one more or one fewer real error moves it by 5 points. Write the sample size next to the number on the spec, and check a larger sample before a precision figure decides anything expensive.

Run the same detector a second time on the relabelled 800. It flags 6 rows, and 5 of them are labels that are still wrong. That sounds efficient until you look at the denominator: 61 wrong labels were left after pass 1, so pass 2 finds 5 of them and misses 56. Test accuracy moves from 478 to 479, one digit, noise. Wrong labels fall from 61 to 57.

The rule: stop when a pass flags few rows and the audited precision of its flags falls, and move the effort to where the remaining errors actually come from. The 56 the detector cannot see are the look-alikes that sit deep in the other digit's neighbourhood; no amount of re-running finds them. They are a guideline problem (session 2) or a coverage problem (session 5). Write the thresholds down before the pass starts. A constructed example: "stop when a pass flags fewer than 10 percent of the previous pass's rows, or when a 20-flag audit finds fewer than half real."

It is tempting to review the top of the list, the flags the detector is most confident about. Those are the easiest ones, and their precision is higher than the rest. An audit of the top 20 estimates the precision of the top 20, and the spec would quote it for all 98. A random draw from all the flags, with the seed recorded, estimates the precision of the whole pass, and anyone can redraw the same sample to check the reviewer.

The same logic sets who reviews. The labelerrors.com design used five independent workers and a 3-of-5 rule so that one tired reviewer could not decide an item. For a small internal audit, two reviewers who disagree on a flag is itself a finding: the guideline has a hole there.