The six sessions 🎯 · from "the data is the problem" to a spec someone can audit
The running data is the UCI Optical Recognition of Handwritten Digits set: 1,797 real 8 by 8 images, one frozen k-nearest-neighbour model, a clean test set nobody edits, and a declared label-noise plant. Real cases join it from Andrew Ng's 2021 talk, the Data-Centric AI Competition, MIT's data-centric AI course, and the label errors found in ten famous benchmark test sets.
Hold the model, fix the data
Code plus data, two loops, and the evidence including where the data loop barely moved. Build-along: freeze a model, plant declared noise on real digits, and measure the three numbers that bound every edit.
🗳️ Session 2 · advancedWhen labellers disagree
Measure agreement beyond "percent the same", see where majority vote goes wrong, and rewrite a label guideline until two careful people land on the same answer.
🔎 Session 3 · advancedFinding the wrong labels
Confident learning by hand: out-of-sample probabilities, per-class thresholds, the confident joint, and why flagging every disagreement finds more errors and more false alarms.
🎛️ Session 4 · bench nightThe data-edit bench
Eight data edits on one frozen model, computed in your browser, each with two counters: test accuracy and how many planted errors the detector found. Plus the more-data trap and a leak.
🧩 Session 5 · advancedSlice, augment or collect
Find the weak slice without peeking at the test set, tell augmentation that adds coverage from augmentation that adds decoration, and settle collect against clean for small data.
📜 Session 6 · capstoneThe data-edit spec
Beat the bench by data edits only, then write the spec: the freeze list, the test-set rule, the edit log, the precision audit and the stopping rule that make an edit outlive the model.
Your passport 🎫
Clear all three quiz questions in a session and it gets stamped. Six stamps means you can freeze a model, measure how much its data is costing, find and fix wrong labels with a detector you can audit, and write the edits down so they survive the next model. Stamps live in this browser only.
What this is not, so you land in the right place 🧭
| If your question is | Go here instead |
|---|---|
| How do I look at a hundred mistakes and sort them by cause before deciding what to fix? | ML Strategy, the error-analysis spreadsheet ↗ |
| Which dev sets do I look at, and is my problem bias or variance? | ML Strategy, sessions p4 and p5 ↗ |
| How do I build features, and how do I stop a column from leaking the target? | Feature Engineering ↗ |
| How do I watch data quality in a pipeline, with alerts, freshness and drift? | Data Observability ↗ |
| How do I find duplicate customers across systems at scale? | Entity Resolution ↗ |
| The errors are found, a lot of them are the data, and I need to fix it without touching the model and prove each fix worked. | you are in the right place |
What is honest here 🧾
The images and every number the bench prints are real computations. The noise is planted on purpose and declared in full, because that is the only way to count what a detector missed. Where a figure comes from a paper or a talk, the page says which; where it comes only from secondary write-ups, the page says that instead.
| Claim | Status |
|---|---|
| Every accuracy, flag, threshold and count on the bench | Really computed in your browser from the 1,797 UCI digits (Alpaydin and Kaynak, 1998), and reproduced to the row by an independent Python run |
| The 133 wrong labels, and the detector's precision and recall | Constructed. A declared plant of six look-alike confusions; precision and recall exist only because the answer key is known |
| The shipped ladder | One seed of twenty, the one closest to the twenty-seed mean; the mean is quoted wherever the two disagree |
| Ng's three-project table and the small-data claim | Read from his 2021 slide deck; the small-data claim is stated there as theory and is quoted as a claim |
| Label errors in ten benchmark test sets | Read from Northcutt, Athalye and Mueller (2021), NeurIPS; the rates are their estimates after human review |
| The competition's fixed model, image size and caps | Reported from participants' write-ups; the task and the deadline were read on the official page |
| The two-labeller tables in session 2 | Constructed, and labelled so on the page |
The knowledge map 🧠
The whole discipline at a glance - hover a session to spotlight its concepts, click any node to jump in.