Where we start
You have already done the error analysis. Someone has looked at a hundred mistakes, filled the spreadsheet, and found that a large share of them trace back to the data: labels that are wrong, labels that two people would write differently, and whole kinds of input the training set barely covers. The question now is what to do about it without touching the model, and how to prove each change helped. That starts with a setup where only the data can move.
★ What you walk out with tonight. A frozen model, a locked test set and a declared noise plant on real digits, the three numbers that bound every data edit (91.4 percent given, 97.8 percent true, 90.8 percent with more of the same), and the habit of asking "what did this edit do on the untouched test set?" before anyone argues about a new architecture.
Two loops, one system
5 min liveAndrew Ng's 2021 talk that named the movement opens with one line: an AI system is code plus data. Both halves can be iterated. The model-centric loop holds the data fixed and changes the code; the data-centric loop holds the code fixed and changes the data. Neither is new. What is new is treating the second loop as an engineering discipline with its own tools and its own measurements, instead of a chore done once before the real work.
LiveThe two views, in Ng's own words2 min▶
The slide that frames the whole field puts two views side by side. The model-centric view: "Collect what data you can, and develop a model good enough to deal with the noise in the data." The data-centric view: "The consistency of the data is paramount. Use tools to improve the data quality; this will allow multiple models to do well."
The last clause is the practical one. A model tuned to survive a messy dataset is tuned to that mess. A dataset cleaned so that labels are consistent makes the next model, and the one after it, better too. That is why the edits you make in these six sessions are written down as a spec at the end: they outlive any one model.
LiveWhat "holding the model" actually means2 min▶
Holding the model fixed is stricter than it sounds. The architecture, every hyperparameter, the training seed, the preprocessing and the evaluation metric are all frozen before the first data edit. If any of them moves, you can no longer say which change bought the gain. Tonight the model is k-nearest neighbours with k = 10 and squared distance on the raw pixels, chosen because it has no training randomness at all: the same rows always give the same predictions, in Python and in your browser, to the last test image.
The test set is frozen harder still. It is never relabelled, never filtered, and never used to choose an edit. The MIT data-centric AI course puts it bluntly: evaluation examples "should only be used for computing evaluation scores, not for any modeling decisions." Choosing which slice to augment by looking at test accuracy is a modelling decision, and session 5 shows how to make it on the training set instead.
Self-studyWhere the name came from, and what is not new3 min▶
Ng gave the talk "MLOps: From Model-centric to Data-centric AI" at a DeepLearning.AI live event in March 2021, and the same year DeepLearning.AI and Landing AI ran the Data-Centric AI Competition that inverted the usual format. MIT's course "Introduction to Data-Centric AI", taught by Anish Athalye, Curtis Northcutt and Jonas Mueller, turned the idea into lectures and labs in 2023 and 2024, and defines the field as treating dataset improvement as "a systematic engineering discipline" rather than "ad hoc trial/error and intuition".
None of the techniques are new. Label-noise research, repeated labelling and data augmentation all predate 2021 by decades. What the name changed is where the effort is booked: a data edit becomes a versioned, measured change with a reviewer, the same way a code change is.
Steel sheets, 39 kinds of defect. Ng's opening case is a Landing AI project inspecting steel for defects: a baseline at 76.2 percent accuracy against a 90.0 percent target. The team asked the room whether to improve the code or the data, and then showed what each had bought. Part 2 has the table, including the two projects where the answer was much less dramatic.
What the data loop bought, and where it barely moved
5 min liveThe famous number is the steel project: model-centric work added nothing, data-centric work added 16.9 points. It is usually quoted alone. The same slide carries two more projects, and they tell a more useful story: the data loop won all three, but by 16.9, 3.06 and 0.4 points. A method that sometimes buys half a point is still a method; it is not magic, and the size of the gain depends on how much of the error was in the data to begin with.
LiveLabel errors sit in the test sets too3 min▶
Northcutt, Athalye and Mueller ran confident learning over the test sets of ten of the most-used benchmarks, then had every flagged item checked by five crowd workers. They estimate "an average of at least 3.3% errors across the 10 datasets", with at least 6 percent of the ImageNet validation set mislabelled. The rate is not uniform: 0.15 percent on MNIST, 5.85 percent on CIFAR-100, 10.12 percent on QuickDraw. Carefully curated datasets were cleaner; datasets built by automated collection were not.
The consequence is the one that matters for a practitioner. On ImageNet with corrected labels, ResNet-18 outperforms ResNet-50 if the share of originally mislabelled test examples rises by just 6 percent. The model you would have shipped depends on whether your test labels are right, which is why the test set here is clean by construction and why session 6 asks you to audit yours.
| Test set | Size | Flagged by the algorithm | Confirmed by humans | Estimated error |
|---|---|---|---|---|
| MNIST | 10,000 | 100 | 15 | 0.15% |
| CIFAR-10 | 10,000 | 275 | 54 | 0.54% |
| CIFAR-100 | 10,000 | 2,235 | 585 | 5.85% |
| ImageNet (validation) | 50,000 | 5,440 | 2,916 | 5.83% |
| IMDB | 25,000 | 1,310 | 725 | 2.90% |
Self-studyWhat the competition inverted2 min▶
The Data-Centric AI Competition page states the inversion plainly: in most competitions you build a model for a fixed dataset; here you "improve a dataset given a fixed model", with techniques "such as fixing incorrect labels, adding examples that represent edge cases, apply data augmentation". Submissions closed on 4 September 2021 and were scored on accuracy. The task was a noisy set of handwritten Roman numerals. The fixed model is reported as a cut-down ResNet50 on 32 by 32 images, with a starting set of about 1,500 images and a cap of 10,000; those three details come from participants' write-ups, not from an official page read for this session.
Tonight's setup is the same shape on Arabic digits: a model nobody may touch, a noisy training set, a clean test set, and a score.
A benchmark that picks the wrong model. If a team compares two vendors' models on a test set where 6 percent of labels are wrong, the ranking can flip with the fix. Northcutt and colleagues show it on public benchmarks with public models; the same thing happens silently on an internal test set that was labelled once, in a hurry, three years ago.
Start after the errors are found
5 min liveData-centric work is the second half of a loop. The first half, looking at the model's mistakes and sorting them by cause, is error analysis, and it has its own discipline: the spreadsheet of a hundred errors, the dev sets you look at and the ones you do not, the split between bias and variance. Those live in ML Strategy, sessions p3 to p5 ↗. The work here picks up at the moment the sheet says "a lot of this is the data" and asks: which edit, and how do we know it worked?
LiveThe four edits, and which session owns each3 min▶
Rewrite the label rule when two careful people would label the same example differently. The fix is the guideline, not the row, and session 2 is about measuring that disagreement before arguing about it. Relabel or remove a row when the label is simply wrong. Session 3 builds the detector that finds those rows without reading all of them, and session 4 measures what each choice buys. Add targeted rows, collected or augmented, when a kind of input is thin in the training set. Session 5 is about finding that slice without peeking at the test set, and about when collecting more beats cleaning what you have.
Ng's closing checklist names the same four properties: good data is "defined consistently", has "good coverage of inputs x", has "timely feedback from production data", and is "sized appropriately". The third is monitoring data in production, and Data Observability ↗ owns it.
Self-studyWhy small data is where this pays most2 min▶
Ng's slide makes a strong claim: with 500 examples of which 12 percent are noisy, cleaning the noise and doubling the set to 1,000 are "about equally effective", and "there is significant room for improvement in problems with <10,000 examples". It is stated as theory, with a clean-against-noisy curve on the next slide that names no dataset. Treat it as a hypothesis to test on your own data, which is exactly what the build-along below and session 5 do. He adds that big-data problems with a long tail of rare events, web search, self-driving and recommenders, "are also small data problems" in the tail.
What to ask for, if you lead the team. Before approving a new architecture, ask for three numbers on a test set that has not been touched: the current model on the labels you have, the same model on a hand-corrected sample, and the same model with the extra data the team is asking for. If the second gap is larger than the third, the next sprint is a data sprint.
Freeze the model, plant the noise, measure the gap
★ 22 min · everyone buildsFive steps in numpy and scikit-learn on the real UCI handwritten digits. Every number printed below is what the code produces; the same split, plant and model run in your browser on the session 4 bench, and a separate reference implementation agrees with it to the last test image.
Load the real digits. 1,797 images of 8 by 8 pixels, each pixel a count from 0 to 16, from the UCI Optical Recognition of Handwritten Digits dataset. The ten classes are nearly balanced.
Split once, with a seed, and write the seed down. 500 test images that will never be edited, 800 to train on, and 497 held back as "more data" for the tempting fix in step 5.
Freeze the model. k-nearest neighbours with k = 10 on squared pixel distance. Distance ties go to the earlier training row and vote ties go to the digit of the nearest tied neighbour, so every run gives the same answer. Score it once on the true labels: this is the ceiling the plant takes away.
Plant the noise, and declare it. Six confusions a tired labeller makes: 1 and 7, 3 and 8, 4 and 9, each way at its own rate. Within each digit the flips are drawn from the images that look most like the other digit, measured as distance to each digit's mean image, because that is where real labellers err.
Measure the gap, then try the tempting fix. The same model on the labels you were given, and then on 497 more rows from the same labellers. Keep all three numbers; they bound everything the next five sessions do.
| Same frozen model, same 500 test digits | Training rows | Correct | Accuracy |
|---|---|---|---|
| The true labels (knowable only because the plant is declared) | 800 | 489 | 97.8% |
| The labels you were given | 800 | 457 | 91.4% |
| 497 more rows from the same labellers | 1,297 | 454 | 90.8% |
Read the third row twice. Sixty-two percent more data made the model 0.6 points worse on this seed. Over twenty seeds of the same setup it averages +0.25 points and is worse than the baseline in 8 of the 20, which is the honest summary: more rows with the same systematic confusions buy roughly nothing, because each new row teaches the same wrong boundary again. Cleaning the 800 you have, session 4 will show, buys about four points.
Before session 2
30 min- Write your freeze list. For a model you own, list every setting that must not move while you edit data: architecture, hyperparameters, seeds, preprocessing, metric, and the exact test file with its checksum.
- Change the seed in step 2 and re-run steps 2 to 5. Write down all three numbers. Which one moved most, and does the third row still sit near the second?
- Plant uniform noise instead: flip 16.6 percent of training labels to a random other digit and re-run step 5. Compare the more-data row with tonight's. The difference is the difference between random and systematic error.
- Bring ten rows your team argued about. Session 2 is about the rows where two careful people disagree, and a real disputed example is worth more than any constructed one.
Three questions
Clear all three and session 1 gets stamped in your passport.
1 · A teammate proposes choosing which digit to augment by looking at per-digit accuracy on the test set. Why is that off the table?
The MIT notes put it as a rule: evaluation examples are for computing scores, "not for any modeling decisions". Session 5 finds the weakest digit by cross-validated agreement on the training rows, and the test set stays a measurement.
2 · Ng's slide shows data-centric work adding 16.9, 3.06 and 0.4 points on three projects. What is the right way to quote it?
Quoting only the steel row overstates the method; quoting only the surface row understates it. The full table is the honest claim, and it is also the more useful one, because it tells you to measure how much error is in your data before promising a gain.
3 · Tonight, 497 more rows from the same labellers took the frozen model from 91.4 to 90.8 percent. What explains that?
Random noise averages out as rows are added; systematic confusion is reinforced. One seed can show a small loss or a small gain, and the sweep is what says "about zero". C has a grain of truth, one test item is 0.2 points, which is why the claim rests on twenty seeds rather than one.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 1 cheat sheet · pin this
Before any row is called wrong, someone has to agree what right is. Session 2 is about labellers who disagree: how to measure agreement beyond "percent the same", how majority vote goes wrong when one labeller is careless, and how a label guideline gets rewritten until two careful people land on the same answer. Bring the ten rows your team argued about.