Why a quality session sits inside a feature course
Session a1 argued that features decide return on investment before models do. Session a2 showed how a leak fakes a good result. This session is the quieter, more expensive cousin of both: the data is not leaking and nobody is cheating, the inputs are simply dirty, and the cost shows up as a permanent drag rather than a scandal. The good news is that dirty data is a budget problem, and budget problems are the kind leaders are actually equipped to solve.
The four kinds of bad data 15 min live
Google's crash course names four categories of unreliable example, and the value of the list is that each one has a different owner and a different fix. When your team says "the data is messy," your first job is to make them say which of these four, because the answer determines who pays.
LiveThe four categories, in Google's own examples6 min▶
Read these out loud in your next data review. The examples are deliberately mundane, which is the point - none of them require a villain:
- Omitted values. "A census taker fails to record a resident's age." Nothing was corrupted; a value simply never arrived. Your team then has to decide what to put in its place, and every choice they make is a guess your business will act on.
- Duplicate examples. "A server uploads the same logs twice." The rows are all individually correct. The dataset is still wrong, because the duplicated slice now counts twice in every average, every rate, and every model weight.
- Out-of-range values. "A human accidentally types an extra digit." One order becomes ten times its real size. A single fat finger can move a monthly mean, and a model trained on it learns from the typo.
- Bad labels. "A human evaluator mislabels a picture of an oak tree as a maple." Here the input is perfect and the answer is wrong, which is why no amount of cleaning the inputs will find it.
LiveWhy three lanes are cheap and one is not5 min▶
Look again at the two lanes in the diagram. Omitted values, duplicates and out-of-range values all share one property: a rule can find them. A null count, a uniqueness check, a plausibility range. These are engineering tasks with a finish line, and they belong in an automated check that runs every day rather than in a person's notebook.
Bad labels have no such rule. The row looks perfectly well formed; only the outcome attached to it is wrong. Finding them takes re-labelling a sample by hand, agreement scores between reviewers, or statistical detective work on rows the model gets confidently wrong. That is a staffing decision, not a script.
This is also where the time actually goes. Google's crash course warns what happens when nobody owns the problem upstream: "you'll spend enormous amounts of your time tossing out bad examples and cleaning up the salvageable ones." Note the verb - your time, meaning your team's, every cycle, forever.
Self-studyWhat "clean" actually means before a model is involved3 min read▶
Two habits worth carrying into any review, both of them boring and both of them cheap:
- Look at your data before you model it. Summary statistics tell you the shape; a scan of actual rows tells you the truth. Most out-of-range values are visible to a human in about thirty seconds and invisible in a mean.
- Write the check down as a rule, not a memory. "Order value is never negative and never above $5,000" is a sentence that can run nightly. "Ravi usually spots the weird ones" is a sentence that leaves when Ravi does.
The four categories are also a good vocabulary test for a vendor or a new hire. If someone describes data quality only as "missing values," they are seeing one lane of four.
Upstream vs downstream: the spend decision 24 min live
Every quality problem can be handled in two places. Downstream means the data science team patches it in code, every project, every retrain. Upstream means somebody changes the form, the pipeline, or the process so the bad data is never created. Both are real options. They have very different cost curves, and choosing between them is a budget call that sits with you, not with the analyst who noticed the problem.
| Problem | Patch downstream (recurring) | Fix upstream (one-off) | Who owns the fix |
|---|---|---|---|
| Missingness | Imputation guesswork - every model invents a plausible value and inherits the guess | Instrument the form so the field is captured at source | Product plus the team that owns the form |
| High cardinality | Encoding complexity - special handling, more code, more ways to get it wrong | A governed lookup with an agreed, maintained list of values | Data governance plus the domain owner |
| Drift-prone columns | Constant re-training as the meaning of the column keeps moving underneath | A stable definition, versioned, with changes announced | The system owner who keeps changing it |
| Bad labels | A silent ceiling on every model built from the data, forever | Labeling quality assurance - sampling, review, agreement checks | Operations plus whoever funds the labelling |
LiveThe leader's rule, in one line6 min▶
Patching downstream is a recurring cost paid by data science forever. Fixing upstream is a one-off paid by engineering.
That asymmetry is the whole session. A downstream patch feels cheaper because it is invisible on any budget line - it is absorbed into the sprint, the notebook, the retrain, and nobody ever writes an invoice for it. An upstream fix is visible, has an owner, has a cost, and therefore has to be argued for. The cheaper-looking option is usually the more expensive one, precisely because its cost is hidden.
Three practical tests before you commit money upstream:
- How many teams pay the tax? One team patching one column is tolerable. Five teams each writing their own patch means five slightly different definitions, which is now a governance problem too.
- Does the patch require a guess? Deduplication is deterministic. Imputation is not. Anywhere the patch invents information, you are shipping an assumption into decisions.
- Will the fix hold? An upstream fix only pays off if the process that produces the data actually changes. If the field goes back to optional next quarter, you bought nothing.
LiveWhere downstream patching is the right answer5 min▶
The rule is not "always spend upstream." Downstream is correct when the problem is small, local, and cheap to detect:
- A one-off historical mess. Data from a system you already retired cannot be fixed at source. Clean it once, document what you did, move on.
- A single low-stakes model. If exactly one internal report depends on the column, the recurring cost is genuinely small.
- When the upstream owner is outside your reach. A vendor feed you cannot change is a downstream problem by definition - but then the recurring cost belongs in the vendor conversation at renewal.
What makes a leader useful here is naming the choice out loud. The failure mode is not choosing downstream; it is drifting into downstream because nobody framed it as a decision with a price tag.
LiveMaking the upstream case to the person who has to pay5 min▶
The awkward part of this decision is that the team who benefits and the team who pays are almost never the same team. Data science gets the relief; engineering, product, or operations gets the work. Three moves that make that conversation land:
- Price the recurring cost, roughly. Not to the dollar - to the order of magnitude. "Four analysts, half a day each, every retrain, four times a year" is a number a sponsor can compare against a two-week engineering change. An unpriced tax always loses to a priced fix.
- Show the divergence, not the effort. Two teams reporting different figures for the same metric is more persuasive than any hours estimate, because it threatens something the sponsor already cares about: trust in the numbers.
- Ask for the smallest durable change. Making one field required is a change a product owner can actually schedule. Rebuilding the intake system is a change that gets deferred for two years while the tax keeps running.
And say the timeline honestly. An upstream fix improves data from the day it ships forward; the history stays as dirty as it always was. That is still worth doing, but a sponsor who expects the back catalogue to heal will feel misled in six months.
Bad labels are the one problem money must go upstream for. The other three categories are annoying but recoverable - a patched pipeline can still produce a good model. Bad labels are different: if the answers in your training data are wrong, they put a hard ceiling on model quality that no amount of feature work, no better algorithm, and no larger budget for modelling can lift. Google's example is a human evaluator mislabelling a picture of an oak tree as a maple, and there is no clever transformation that turns that maple back into an oak. When a team tells you the model has plateaued and the features are already good, ask who audits the labels before you approve another modelling sprint.
The quality questions for a data review ◐ 10 min read
Four questions, one per row of the table above. They are deliberately answerable - if your team cannot answer them, that itself is the finding.
Self-studyThe four questions, and what a good answer sounds like6 min read▶
What fraction is missing per column, and is it random? A single overall number hides everything. The follow-up matters more than the number: missing at random is a modelling inconvenience, but missing because of something - only unhappy customers skip the field - means the gap itself carries information, and imputing it quietly erases a signal.
How many distinct categories does this column have, and is the list governed? Twelve values with an owner is a dimension. Twelve hundred free-text values with no owner is a maintenance liability that grows every month. Ask who can add a new value and whether anyone reviews it.
Which columns changed definition in the last year? Nobody volunteers this. Ask it directly, and ask where the change was announced. A column whose meaning shifted silently is the reason a model that worked in March stopped working in September, and it will keep happening until definitions are versioned.
Who audits the labels? If the answer is a name, ask how often and on what sample size. If the answer is "nobody," you have found the ceiling on every model in that domain, and you have found it before spending another quarter of modelling effort against it.
Self-studyRetiring features is a quality decision too3 min read▶
Google's "Rules of Machine Learning" includes a rule that reads like housekeeping and behaves like risk management. Rule 22: "Clean up features you are no longer using."
Unused features are not free. Each one is a column somebody still maintains, still monitors, still has to explain in an audit, and still might reintroduce into a model by accident. Feature debt behaves exactly like technical debt: it accumulates quietly, and the interest is paid by whoever inherits the system.
Practically, this means a deprecation process needs to exist before you need it - a way to mark a feature unused, a waiting period, and then an actual removal. Session a4 puts that process inside a wider governance frame.
Official sources covered
This leader track teaches the decision-relevant core of the canonical sources - Google's crash-course data modules and Rules of Machine Learning, sklearn's preprocessing and pitfalls guides - with no code and no notebooks. Certificates and graded exercises stay with the official providers. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Which set names the four categories of bad data from Google's crash course?
The four come with deliberately ordinary examples: a census taker failing to record a resident's age, a server uploading the same logs twice, a human typing an extra digit, and an evaluator mislabelling an oak tree as a maple. The first three a script can catch; the fourth cannot be caught by a rule.
2 · Why are bad labels the expensive category, even when every input column is perfectly clean?
A mislabelled row is structurally valid, so no data quality script flags it, and the model faithfully learns the wrong answer. Detecting the problem takes people - sampling, review, agreement checks - which is why it is the one category where money has to go upstream.
3 · Your team has been imputing a missing field for three years and five teams now write their own version of the patch. What does the ownership rule say?
Five teams patching means five slightly different guesses about the same missing value, so it is a definition problem as well as a cost problem. The downstream patch looks cheap only because its cost is absorbed invisibly into every sprint, and Google warns exactly what that adds up to: "you'll spend enormous amounts of your time tossing out bad examples and cleaning up the salvageable ones."