When labellers disagree

Ng's slide on making data quality systematic is three sentences long, and every word is doing work: "Ask two independent labelers to label a sample of images. Measure consistency between labelers to discover where they disagree. For classes where the labelers disagree, revise the labeling instructions until they become consistent."

Notice what it does not say. It does not say "find the bad labeller" or "take the majority". The unit of repair is the instruction, and the unit of measurement is the class. A sample is enough, because the goal is to find which parts of the rule are ambiguous, not to relabel the whole set twice. The iguana example on the same deck shows how little it takes: the instruction "Use bounding boxes to indicate the position of iguanas" produces boxes that differ from labeller to labeller, and every one of them follows the instruction.

An error is a label that contradicts the rule: a 3 labelled 8 when the guideline is clear that it is a 3. An inconsistency is two labels that each satisfy the rule, because the rule does not decide the case. You cannot call "Um... today's weather" wrong until someone has written down whether hesitations are transcribed and how. Before that, it is one of several defensible answers.

This matters for what you do next. Errors are fixed row by row, and session 3 builds a detector that finds them. Inconsistencies are fixed once, in the guideline, and then every affected row is relabelled against the new rule. Running an error detector on an inconsistent dataset flags rows whose label is "wrong" only relative to the model's guess at what the rule should have been.

When two labellers disagree the tempting move is to rank them, retrain the weaker one or replace them. That treats the symptom. If two careful people read the same instruction and diverge, the instruction admits both readings, and a third careful person will diverge too, in a direction you cannot predict. Replacing people does not remove the ambiguity; it only changes which reading wins this month.

Treat the guideline the way an engineering team treats an API contract. It has a version number. Every class has a definition, a list of edge cases with a ruling for each, and examples of both sides of every boundary. A disagreement is a bug report against the contract. The fix is a new version, and rows labelled under the old version are marked so they can be rechecked. Session 6 writes that version into the data-edit spec, so the guideline outlives the team that wrote it.

People problems do exist. The careless labeller C in tonight's build-along is one. But you can only tell a careless person from an ambiguous rule once the rule is clear enough that careful people agree on it, which is why the rule comes first. The disputed rows themselves usually come out of error analysis, which ML Strategy p3 ↗ covers.

Look at the pair with the higher kappa. A and B agree on 670 of the 800 rows, and 68 of those 670 are wrong together: both labellers wrote the same wrong digit. That is not bad luck. They share a guideline, so they share its blind spots; a 7 without a clear top bar looks like a 1 to both of them for the same reason. Kappa compares the labellers with each other and never with the truth, so a shared confusion raises it.

So kappa measures consistency, and consistency is the thing Ng's protocol asks for, which makes it the right tool for finding ambiguous classes. It is the wrong tool for certifying a dataset. A high kappa on a class tells you the rule is being read the same way; it does not tell you the rule is right. For that you need either a ground truth sample checked by someone who knows the domain, or a detector that looks at the images, which is session 3.

The per-digit table points at where to start. Every disagreement between A and B falls on one of the three look-alike pairs:

Three limits are standard in the measurement literature, and each one shows up in real labelling projects.

Prevalence. Chance agreement p_e is computed from each labeller's label frequencies. When one class dominates, say most rows are "no defect", both labellers use that label most of the time, so p_e is high, and even a high raw agreement can produce a modest kappa. The same labelling quality gives different kappas on datasets with different class balance, so a kappa from one project does not transfer to another.

Few classes. With two classes, chance agreement is 50 percent when both labellers use the two labels equally often, and higher when one label dominates, so the gap between raw agreement and kappa is large. With ten balanced classes, as tonight, it is small. Compare kappas only between tasks with the same number of classes and similar balance.

It is one number for all classes. A kappa of 0.8 can hide one class at perfect agreement and another near chance. Always report the per-class table alongside it. Cohen's kappa is also defined for exactly two labellers; for more, extensions such as Fleiss' kappa exist, with the same caveats. Published verbal scales ("substantial", "almost perfect") are conventions, not measurements, so set your own threshold per class from what a disagreement costs.

Tonight's three constructed labellers vote on each of the 800 rows. The majority label is right on 89.8 percent of them, and it leaves 82 wrong labels against 133 for A alone. Trained on the vote, the frozen model scores 483 of 500, against 457 on A's labels. That is a real gain, and it costs three labels on every row.

Both flaws show up in the same run. On 25 rows all three labellers wrote a different digit, a three-way split with no majority; the build-along falls back to A's label and says so, which is a choice, not a rule. And C, who guesses on a fifth of rows, gets exactly the same vote as A and B. Where A and B share a confusion, C's occasional correct answer is outvoted two to one, which is how the 68 rows wrong together survive the vote.

The MIT notes describe the step beyond the vote. First, estimate each annotator's quality as their agreement with the consensus, then let better annotators count for more. The classic model for this is Dawid-Skene, which learns a full confusion matrix per annotator: not just "C is wrong 20 percent of the time" but "B writes 1 when the truth is 7". The notes also give its limit: it cannot give a reliable consensus for rows labelled by only one annotator, which in practice is most rows of most datasets.

CROWDLAB closes that gap by adding a model. It combines any classifier's predicted probabilities with the annotations, so a singly labelled row still gets a second opinion from the model, and multiply labelled rows get the annotators weighted by how reliable they have proved. Neither is computed here; both are named so you recognise them when a vendor quotes them. Note what neither can do: if every annotator shares the same confusion, weighting them differently does not recover the truth.

The build-along runs a constructed experiment: suppose a revised rule, "a 7 has a top bar, a 1 is a single stroke", ends the 1 and 7 confusion and nothing else. It fixes 48 rows of A's labels, and the frozen model goes from 457 to 472. The vote got to 483, higher, but it paid for two more complete passes over all 800 rows. The rule fixed one of three confused pairs with one rewrite and a recheck of the 1s and 7s; rules for 3/8 and 4/9 are the obvious next edits. The effect of the rule is constructed, the images and the model scores are real.

Sheng, Provost and Ipeirotis (2008), "Get Another Label?", study when repeated labelling improves label quality and model quality. As reported from the paper's abstract, the answer is that it pays under conditions rather than always; read the paper before budgeting a second or third pass. The order that follows from tonight is Ng's: measure agreement on a sample, rewrite the rule where it breaks, and pay for extra labels only on the classes that stay inconsistent after the rewrite.