What this last session is for
Everything so far has been diagnosis: why features carry the return, how leakage fakes success, where quality costs get paid, what governance actually protects, and when a learned feature is worth its price. Tonight is the operating manual. A review you can run without reading code, a spec your team can fill in without a meeting, and a synthesis short enough to repeat from memory in front of an executive who has none of this context.
The feature review ritual 15 min live
Six questions, always in this order. The order matters: each one catches a different failure, and the early ones catch the failures that make the later answers meaningless. Run them on any model that comes to you and you will find the problems that code review does not, because none of these are code problems.
1 · What is the base rate, and what does the held-out score say? Never the training score alone. A training score answers "did the model fit what it saw", which is always flattering and never sufficient. The base rate tells you whether the score is impressive at all - on a 3.2% conversion problem, 96.8% accuracy is what you get by predicting "no" every time. Ask for both numbers before you react to either.
2 · For each feature, when is the value written relative to the outcome? This is the leakage question, and it is the most important one on the list. sklearn's definition is the standard to hold them to: leakage is "when information that would not be available at prediction time is used when building the model", producing "overly optimistic performance estimates" and "poorer performance when the model is used on actually novel data". A feature is only a feature if it exists before the thing you are predicting.
3 · What did every transform fit on? Scalers, encoders, imputers and selectors all learn statistics from data. If they learned from the test set, the test score is fiction. sklearn's rule is blunt: "Always split the data into train and test subsets first, particularly before any preprocessing steps", and "Never include test data when using the fit and fit_transform methods."
4 · Does the split respect time and entities? If the model will predict the future, the test set has to be the future. If a customer can appear on both sides of the split, the model has met them before and the held-out score is not held out. Neither failure produces an error message.
5 · Which features are we no longer using? Rule #22 of the Rules of Machine Learning: "Clean up features you are no longer using." Unused features are pipelines nobody maintains, storage nobody questions and failure modes nobody expects. Feature debt is real debt, and this is the only question on the list that saves money rather than preventing embarrassment.
6 · Who owns each feature definition? A name, not a team. If two dashboards disagree about "active customer", the model is quietly using one of the two definitions and nobody has decided which. Ownership is what turns a column into something that can be corrected.
LiveWhy question 2 outranks the other five5 min▶
Every other failure on the list announces itself eventually. A bad split gets caught when the numbers look strange. Feature debt shows up in a cost review. A missing owner surfaces the first time two people disagree.
Leakage does the opposite. It announces itself with a better score, which is why it survives code review, demo day and the launch meeting, and only fails once it is somebody's production system. sklearn documents this with a number worth carrying: on completely random data with no signal at all, running feature selection before the split produced 0.76 accuracy. Done honestly, the same pipeline scored 0.50, and 0.43 under cross-validation. Twenty-six points of accuracy manufactured out of pure noise.
So the question has to be asked feature by feature, not once for the model. "When is this value written, relative to the outcome we are predicting?" Ask it of the strongest feature first. That is where the leak usually is, because a leak is what makes a feature look strong.
LiveThe importance-list trap4 min▶
Teams often bring a feature importance chart to a review. Two cautions from sklearn's own documentation before you read one:
- Tree importances are biased. Impurity-based importance is "strongly biased" and will "favor high cardinality features (typically numerical features) over low cardinality features such as binary features". A tall bar can be an artifact of how many distinct values a column has.
- Importance is model-specific, not truth. "Permutation importance does not reflect the intrinsic predictive value of a feature by itself but how important this feature is for a particular model." And: "Features that are deemed of low importance for a bad model (low cross-validation score) could be very important for a good model."
The reviewable version of the question: was this computed on the held-out set? sklearn's rule is the one to hold: "Features that are important on the training set but not on the held-out set might cause the model to overfit."
Self-studyRunning the review without technical depth3 min read▶
You are not auditing code, and you should say so. The six questions work because each has an answer a person can state in a sentence, and an evasive answer is itself the finding.
- "We can get you the held-out number" means it has not been looked at recently.
- "It's in the pipeline somewhere" for question 3 means nobody has traced what fit on what.
- "The whole team owns it" for question 6 means nobody does.
None of that requires you to read a line of code. It requires you to ask the same six questions every time, so that a team's preparation for your review becomes the review's real output.
What a feature spec must contain 12 min live
The six questions need somewhere to land, and that place is a feature spec: one short block per feature, written once and maintained. This is exactly the artifact the practitioner capstone produces, so if both tracks have run, your team already knows how to fill it in. Seven fields, and every one of them exists because something went wrong without it.
| Field | What it says | What it prevents |
|---|---|---|
| Name + definition | The feature name and one sentence describing what it measures | Two teams computing "active customer" differently and neither noticing |
| Source | The table and column it comes from | A feature nobody can trace back when the number looks wrong |
| Write-time relative to the outcome | When the value is populated, compared to when the thing being predicted happens | Leakage. This is the field the whole spec exists for |
| Transformation | What was applied: scaled, encoded, binned, imputed, derived | Silent differences between how the feature is built for training and for serving |
| Owner | A person, not a team | A definition drifting for six months with nobody accountable |
| Refresh cadence | How often the value is recomputed, and how stale it may be at prediction time | A model served with a feature that was current last quarter |
| Deprecation status | Live, deprecated, or retired - and since when | Feature debt, and Rule #22 becoming a slogan rather than a practice |
The test of a spec is not whether it is complete. It is whether it survives three events. A team change: the person who built the feature leaves and someone else has to maintain it. An incident review: the model produced something wrong on a Tuesday and you have four hours to explain why. An audit: an external party asks what the model uses and how each input is defined. A spec that answers all three is worth the twenty minutes it takes to write. One that only satisfies a checklist is documentation theatre, and everybody in the room knows it.
LiveWhy the write-time field is the load-bearing one4 min▶
Six of the seven fields are hygiene. The write-time field is protection, and it is the one nobody writes down unless the template forces them to.
It works because it converts an abstract worry into a factual question with a checkable answer. "Is there leakage in this model?" invites a shrug. "When is post_purchase_flag written, relative to the purchase?" has one correct answer, and once it is written on a page next to the feature name, everyone in the review can see the problem at the same moment.
This is also the field that connects to the governance session. Rule #29 of the Rules of Machine Learning states the structural version: "The best way to make sure that you train like you serve is to save the set of features used at serving time, and then pipe those features to a log to use them at training time." Feature stores implement that idea, and one of their stated purposes is to "Avoid data leakage by generating point-in-time correct feature sets". The spec is the human-readable version of the same discipline, and you can have it this quarter without buying anything.
Self-studyWorkshop: draft one spec before the next review15 min exercise▶
Pick the single feature your team believes is the strongest in a live model. Fill in all seven fields yourself, from what you can find out without writing code, and note where you get stuck.
- If you cannot fill in write-time, you have found your first review item, and it is the important one.
- If you cannot fill in owner, the definition is unowned regardless of what the org chart says.
- If refresh cadence turns out to be different in training and in serving, you have found training-serving skew without needing a tool to detect it.
Bring the half-finished spec to the team rather than a request for documentation. A leader filling in six fields and asking for the seventh gets a different response than a leader asking for a document.
Course synthesis 12 min live
Five sessions, five sentences. If you remember nothing else from this track, remember these, because each one changes a decision you will actually make.
a1 · Features carry the return. The model-selection debate that eats a planning meeting is usually worth less than an afternoon of feature work. Fund the inputs, not the algorithm shopping.
a2 · Leakage is the failure that looks like success. It arrives as a better score, so it passes every review built to catch bad news. sklearn's own noise experiment manufactured 26 accuracy points out of nothing at all.
a3 · Quality is a budget with an upstream or downstream choice. You are always paying for it. The only question is whether you pay once at the source or repeatedly in every pipeline that consumes it.
a4 · Skew and point-in-time correctness are governance, not tooling. Rule #29 says to train on the features you served. A feature store is one way to implement that discipline, and it is a bad substitute for having decided to have it.
a5 · Learned features are a trade, not an upgrade. Embeddings buy signal from text that no column reaches. They cost storage, latency and explainability. Rule #17 keeps the cheap option on the table: start with directly observed and reported features.
Self-studyHow the two tracks fit together3 min read▶
The leader track gave you the decisions. The practitioner track gives your team the craft: numeric transforms, encoding, datetime decomposition, the leakage session, interactions, missing data, selection, the embeddings bridge, and a capstone that ends in the feature spec from Part 2.
They meet at the spec. Your six questions are what the spec is designed to answer, which means the two tracks produce a shared artifact rather than a shared vocabulary. That is a much stronger form of alignment, and it is the reason the capstone exists in the shape it does.
Where next ◐ 5 min read
Three directions, depending on who you are sending and what you are trying to fix.
The practitioner track of this course - for anyone who builds features. Ten sessions on the same running case, with a live workbench, the leakage trap you can press, and a written feature spec as the capstone deliverable. If you have run the review from Part 1 and the answers were thin, this is the fix.
learn-model-evaluation-with-phoebe - for judging the result. This course owned the inputs. That one owns the verdict: metrics, thresholds, and the cost of being wrong in each direction. The natural next conversation after "the held-out number is 0.81" is "0.81 of what, and what does a false positive cost us".
learn-intro-ml-with-phoebe - for the modelling mechanics. If the vocabulary in a model review is still doing you damage, this is the on-ramp. It builds the Lumen case this course improved, and it makes the train and held-out split concrete rather than abstract.
Self-studyWhat this course deliberately does not cover2 min read▶
Stated plainly so you know where the edges are:
- Deep-learning representation learning - a different course, and a different set of trades than the embeddings bridge in a5.
- Lag, rolling and seasonal time-series features - the forecasting course owns those properly.
- Feature-store implementation and operations - a4 taught the concept and when it pays. The build is a data-engineering topic.
- Automated feature engineering tools - named, not taught. None of them answer the write-time question for you.
- Certificates and graded exercises - those stay with the official providers, and the source list on every page tells you where to find them.
Official sources covered
This closing session pulls from every source the track used, since the review and the spec are the synthesis of all of them. Certificates and graded exercises stay with the official providers. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Of the six review questions, which one outranks the rest, and why?
Every other failure surfaces eventually. Leakage does not: sklearn's definition names information "that would not be available at prediction time", and their noise experiment produced 0.76 accuracy on data with no signal at all, against 0.50 done honestly. Ask the write-time question of the strongest feature first - a leak is often what made it look strong.
2 · What must a feature spec contain?
Seven fields, each present because something went wrong without it. The write-time field is the load-bearing one - it turns "is there leakage" into a factual question with a checkable answer. The test of the spec is whether it survives a team change, an incident review and an audit.
3 · A team brings you a model with 96.8% accuracy on a problem where 3.2% of sessions convert. What do you ask?
Review question 1, exactly as written. The base rate tells you whether a score is impressive at all, and the held-out number is the only one connected to production. A training score answers "did the model fit what it saw" - always available, always flattering, never enough.