The model is only as good as what you feed it
There is a comforting myth that machine learning is about clever algorithms, and that a better algorithm rescues a shaky project. It almost never does. A model learns entirely from the examples you hand it, so if those examples are biased, stale, mislabelled, or secretly contain the answer, the model faithfully learns the wrong thing - and does it confidently. When Lumen Skincare's models went wrong, the algorithm was never the culprit; the data was. This session teaches you to look where the failures actually live, so you can interrogate a project before it burns a quarter.
Features, labels, and garbage in, garbage out 8 min live
Every supervised model is trained on a table. Each row is one example - for Lumen, one website session. The columns split into two kinds. Features are the inputs: everything you know about the visitor before the outcome (their channel, whether they are new or returning, their prior spend). The label is the answer you want to learn to predict (did they convert?). The model studies how the features relate to the label. And here is the iron law: if the data is biased, stale, or mislabelled, the model learns the garbage faithfully. Garbage in, garbage out is not a slogan here - it is mechanical.
LiveWhat a feature is3 min▶
A feature is one input column - one thing you know about the case before the outcome. Choosing good features (the craft your builders call "feature engineering") matters far more than choosing the algorithm. A model with great features and a plain algorithm beats a fancy algorithm fed weak features almost every time.
- Numbers - like prior 30-day spend, or pages viewed. Often the strongest signals.
- Categories - like channel (email, paid search, social...) or device. The model learns which values lean which way.
- Derived features - things you compute, like "days since last order." Often where the real signal hides.
Lumen's conversion model runs on a handful of honest features per session: prior_30d_spend (by far the strongest - past spend predicts future buying), new_vs_returning (returning visitors convert about twice as often), channel, product_category, device, and pages_viewed. None of them is exotic. The model's whole skill comes from combining these plain, knowable facts - not from a clever algorithm.
LiveWhy label quality is everything3 min▶
The label is the answer the model learns to copy. So if your labels are wrong, the model learns to reproduce the wrong answers - and no algorithm can fix that, because it has no other source of truth. This is why "garbage in, garbage out" is literal: the model cannot be more correct than the answers you taught it.
- Biased labels - if past approvals favoured one group unfairly, a model trained on them learns to repeat that unfairness, and dresses it up as objective.
- Stale labels - answers from a world that no longer exists (pre-launch, pre-pandemic) teach patterns that have expired.
- Mislabelled examples - if "converted" was recorded inconsistently, every downstream number is built on sand.
Data leakage: the silent killer 7 min live
Here is the failure that fools even good teams. Data leakage is when a feature secretly contains the answer, or could not actually be known at the moment you predict. The model looks brilliant in testing - because it is quietly peeking at the answer - and then falls apart in production, where that peek is gone. It is the most dangerous mistake in this whole course precisely because the numbers look great right until it ships. And a related trap: more columns is not more signal. Some features are pure noise.
LiveHow to smell leakage3 min▶
You do not need to read code to catch leakage. You need one question, asked of every feature: "would we actually know this at the moment we make the prediction?" If the honest answer is "no, that gets recorded later," you have found a leak.
- Answers in disguise - a column like
was_refundedonly exists after a purchase happened. Using it to predict a purchase is circular. - Timing violations -
account_closed_datetells you the customer left; using it to predict churn is predicting the past from the future. - The tell - a suspiciously perfect score. Real problems are hard; 99% usually means the model found a shortcut to the answer, not a genuine pattern.
Lumen once built a churn model that hit 99% accuracy - the team was thrilled. One feature was a status flag that only got set after a customer had already cancelled. The model was not predicting churn; it was reading a column that said "this person churned." Brilliant in the lab, worthless live - the flag simply is not there for a customer who has not left yet. Once they removed it, the honest accuracy dropped to something modest and real.
LiveMore features != more signal2 min▶
The instinct to "just add every column we have" feels safe and is wrong. Irrelevant features do not sit there harmlessly; they add noise the model can latch onto, making it easier to overfit (Session 2) and harder to trust. Parsimony - the fewest features that carry real signal - beats a kitchen-sink of columns.
- Signal features genuinely relate to the answer, like prior spend to order value.
- Noise features have no real relationship; the model may fit their random wiggles as if they mattered.
session_hour, the hour of day a visit happened. It looks plausible - surely evening shoppers differ? In this data it carries essentially no signal; it is deliberately near-useless. Adding it does not help the model and gives it one more way to chase noise. When someone says "let's add more data," ask "more signal, or just more columns?"
This week ◐ 25 min total
- Audit a feature list. Find one model in your org and get its list of features. For each, ask "would we know this at the moment we predict?" Flag anything that fails.
- Hunt a feature from the future. Look specifically for a column recorded after the outcome - a status, a resolution, a closed date. If you find one in a model's inputs, you have found a likely leak. Bring it to Session 4.
- Question a trusted metric. Name one number your org relies on and ask what labels it is built on. If those labels were set inconsistently or by a biased process, the metric may be quietly wrong.
Three questions before you go 🎯 ◐ 90 seconds
1 · What is data leakage?
Leakage is a feature that leaks the answer - it contains the outcome or is only recorded after it. It makes a model look brilliant in testing and useless in production, where that information is not available yet.
2 · A model is trained on two years of labels set by a biased past process. What happens?
A model can only be as good as the examples it learns from. Biased labels teach the model to reproduce the bias, confidently and at scale. The algorithm has no other source of truth to correct against.
3 · Your team wants to add every column you have, including session_hour, to a model. Good idea?
More features is not more signal. A near-useless column like session_hour adds noise the model can chase, raising the risk of overfitting. Parsimony - the fewest features that carry real signal - wins.
What this session covers
This session teaches the framing behind the data-quality and leakage warnings that every serious ML course hammers - the judgement a decision-maker needs. The hands-on fix in Python (building a Pipeline so transformers are fit on training data only) lives in Builder Session 9; the sources below go deeper.