Finding the wrong labels

Take row 5 of the training set. It carries the label 1. Split the 800 rows into four folds, fit kNN on the three folds that do not contain row 5, and ask its ten nearest neighbours what it is: 8 of them say 7 and 2 say 1. That is a strong, honest signal that the label is wrong, and in fact the image is a 7 the plant labelled 1.

Now ask the same question in sample, with row 5 still in the training set. Its nearest neighbour is itself, at distance zero, carrying the label 1. Every row gets one guaranteed vote for its own label, and the look-alike rows the plant favours are exactly the ones whose neighbourhoods are already split. On the 800, the in-sample detector flags only 42 rows, 40 of them real, for precision 95.2 percent and recall 30.1 percent, against 63.9 percent out of sample. The high precision is not a virtue: it only flags the rows so wrong that even a self-vote cannot rescue them. Over twenty seeds the pattern holds, with in-sample recall averaging 32.45 percent against 59.57 percent out of sample.

This is detector leakage, a row's own label leaking into the judgement of that label. It is a cousin of feature leakage, which Feature Engineering covers ↗; the cure here is the fold split, nothing more.

Confident learning asks for an n by m table of predicted probabilities: here 800 rows by 10 digits. A kNN with k = 10 gives the simplest version there is: the share of the ten neighbours voting for each digit. Row 5's row of the table is 0.2 for 1, 0.8 for 7, zero elsewhere. The probabilities move in steps of 0.1, they are not calibrated in any formal sense, and it does not matter much, because the next step compares each row against a bar set by the same model on the same data.

The paper fixes k = 4 folds in its cross-validation experiments, and we do the same. Any model that outputs probabilities can replace kNN: logistic regression, a gradient-boosted tree, a fine-tuned network. What cannot change is that each row's probabilities come from a model fitted without it. The homework swaps the model and compares the flags.

MIT's Introduction to Data-Centric AI, taught by Anish Athalye, Curtis Northcutt and Jonas Mueller, runs the label-errors lecture on the same pieces: out-of-sample predicted probabilities via cross-validation, "The thresholds for each class are the average predicted probability of examples in that class", and confident counting. Its lab asks for "your own implementation of automatic label error identification using Confident Learning". The open-source cleanlab library packages the method, and it is not taught here: writing it by hand once is how you learn which choices, the fold count, the tie rule, the model, move the flag list, and it gives you something you can check line by line.

On the 800 training digits the bars come out as printed by the build-along: 0: 1.00, 1: 0.49, 2: 0.94, 3: 0.54, 4: 0.59, 5: 0.95, 6: 0.98, 7: 0.66, 8: 0.58, 9: 0.60. A row given 0 has to win all ten votes for 0 to look confidently like a 0, because 0s are easy and every other 0 does. A row needs only 0.49 of its votes for 1 to look confidently like a 1, because the plant has scrambled 1s and 7s and the model is rarely sure about either.

That is the point of the per-class bar. The paper argues it makes the method robust to "heterogeneous class probability distributions" and to "class-imbalance": a class the model is over-confident about gets a proportionally higher bar, so its rows are not all waved through, and a class it is unsure about gets a lower one, so its rows are not all flagged.

The confident joint C is a ten by ten table. For every row given label i, find each digit j whose probability clears t_j. If exactly one does, add one to C[i][j]. If several do, the collision rule of equation 1 applies: take the one with the highest probability among those over their bars. Diagonal cells are rows whose label the votes confirm; off-diagonal cells are rows where the votes confidently point elsewhere, and those are the flags.

On the 800 the diagonal holds 80, 46, 65, 43, 44, 59, 83, 49, 48 and 42, and the off-diagonals hold 98 rows. The big cells are the plant showing through: given 7 but suggested 1, 20 rows; given 9 suggested 4, 19; given 8 suggested 3, 17; given 1 suggested 7 and given 3 suggested 8, 14 each; given 4 suggested 9, 7. The small ones, 3 rows given 8 suggested 1 and single rows at 2 to 8, 3 to 7, 5 to 9 and 9 to 8, are where the detector disagrees with a label the plant never touched, and Part 3 checks them.

A row whose probabilities clear no bar at all, such as a near-uniform spread of votes, lands in no cell. The paper counts that as a feature: the joint is "robust to pure noise or examples from an alien class". It is also, as Part 3 shows, where the missed errors hide.

The raw counts are not yet an estimate of the noise. Equation 3 of the paper calibrates C into Q, an estimate of the joint distribution of given and true labels: each row is rescaled so its sum matches the observed count of that given label, and the whole table is normalised to sum to 1. Q is what you would report as "an estimated share of 7s labelled 1".

The paper then compares several ways to choose which rows to prune. The baseline C_confusion flags every row whose argmax disagrees with its label, which the authors describe as "similar to a single iteration of INCV". C_{y,y*} flags the off-diagonals of the confident joint, which is what we compute tonight. Prune by Class removes, per given class, the rows with the lowest self-confidence; Prune by Noise Rate ranks them by margin, the gap between the best other class and the given one; C+NR takes the intersection of the two. None of these variants is computed here; the build-along implements C_{y,y*} and C_confusion only, so every number on this page traces to one of those two.

Score both detectors on the same out-of-sample votes against the declared plant. Confident learning flags 98 rows and 85 of them are planted errors: precision 86.7 percent, recall 63.9 percent. The argmax baseline, the paper's C_confusion, flags a row whenever its given label is not among the top-voted digits. It flags 141 rows and catches 109: precision 77.3 percent, recall 82.0 percent.

The baseline finds more errors and raises more false alarms: 32 of its 141 flags are correct labels, against 13 of 98. Per-class thresholds trade some recall for precision, and which side of the trade you want depends on what happens to a flag. If every flag goes to a human reviewer, precision is reviewer time. If flags are relabelled automatically, every false alarm writes a new error into the data: all 13 of confident learning's false alarms were relabelled wrongly, while all 85 true catches went to their true digit. On twenty seeds of the same setup confident learning averages precision 88.76 percent and recall 59.57 percent, so tonight's seed is typical.

The 48 planted errors confident learning misses are not random. They are the look-alikes: the 7 drawn so like a 1 that its neighbours split, the 1 that half its neighbours call 7, the 9 that could pass for a 3. Their votes are spread, so the true digit's probability falls short of its bar, the given label falls short of its own, and the row lands in no cell of the joint. The same rule that protects the joint from pure noise hides these rows.

Part of the reason is the plant itself. Confident learning assumes class-conditional noise: the chance a label is flipped depends on the true class, not on the particular image. Our declared plant bends that assumption on purpose, because it picks the flips from the images that look most like the other digit, which is what tired labellers actually do. The test-set study reached a similar conclusion about real benchmarks, where errors are "primarily systematic mislabeling, not just random noise". Treat a detector's recall on image-dependent noise as lower than its paper numbers, and measure it.

Without a declared plant, the only way to know precision is to check a sample of flags by hand. Northcutt, Athalye and Mueller did it on the test sets of ten benchmarks: each flagged item went to five crowd workers, and it counted as an error only if fewer than 3 of 5 agreed with the given label. On average 51 percent of the algorithmically flagged candidates were confirmed as real errors. On MNIST, Table 2 shows 15 errors among 100 flagged, a precision of 15 percent. On clean data most flags are false alarms; on noisy data most are real. The same algorithm, run the same way, gives very different precision depending on how dirty the data was to begin with.

The paper's gallery of confirmed errors also shows the failure modes a reviewer will meet: an image that shows part of an object labelled as the whole, an object seen from an unusual viewpoint, and, on digits, a figure drawn so badly that any label is a guess. The authors add that their corrected labels "are not 100% perfect". A flag is a request for a human look, never a verdict.