How this track works
Ten sessions, one running case: Lumen Skincare's conversion model - the classifier the intro-ML course trained on 40,000 checkout sessions with a ~3.2% conversion rate. That course built the model; this course decides whether it is any good, what it is worth in dollars, and when to ship it. b1-b5 are the classic half (matrix, threshold, curves, regression and ranking, then the cost matrix - the session everything funnels into). b6-b8 are the LLM half (quality metrics, the grader problem, speed and cost). b9 makes evaluation continuous, b10 assembles the full eval sheet. Sibling courses: learn-intro-ml teaches the sklearn mechanics; learn-evals teaches the LLM eval harness hands-on. This course owns the metrics themselves.
Four cells, two kinds of pain 7 min live
A binary classifier's predictions land in exactly four buckets. Two are correct (true positives, true negatives). Two are mistakes - and they are different mistakes, paid for by different people. A false positive at Lumen: the model flags a session as "will convert," marketing spends outreach money, nobody buys. A false negative: a real converter goes unflagged, no nudge arrives, a ~$74 order quietly does not happen. Every metric in this course is some ratio of these four cells - the matrix is the raw truth, the metrics are summaries with opinions.
LiveThe accuracy paradox, on Lumen's numbers5 min▶
Lumen's conversion rate is ~3.2%. So consider the laziest possible model: predict "will not convert", always. Its confusion matrix has two occupied cells - all the true negatives, all the false negatives - and its accuracy is 96.8%. A model that never finds a single customer outscores, on accuracy, most models that actually work.
Google's ML Crash Course states the general form: on data where a class appears 1% of the time, a model predicting negative 100% of the time scores 99% accuracy "despite being useless." This is not a curiosity - most valuable classification problems ARE rare-event problems: fraud, churn, conversion, defects, disease. Which means:
- Accuracy is a rough progress indicator for balanced data only (Google's own metric table).
- On imbalanced data, accuracy mostly measures the base rate, not the model.
- sklearn ships
balanced_accuracy_score(the macro-average of per-class recall) precisely because plain accuracy gives "inflated performance estimates on imbalanced datasets" - its docs' words.
The 99%-accurate fraud model. A bank's new fraud model reported 99.2% accuracy and got fast-tracked. First month in production it caught almost nothing: fraud was 0.4% of transactions, and the model had learned to say "legitimate" nearly always. Nobody had asked for the confusion matrix. One glance at the FN cell would have killed the launch in review - which is exactly the glance you now always take.
Self-studyThe matrix in sklearn, 60 seconds2 min read▶
The intro-ML course covers this hands-on (its b7); here is the refresher:
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
cm = confusion_matrix(y_test, model.predict(X_test))
# rows = actual, columns = predicted (sklearn's convention - CHECK it,
# other libraries and half the diagrams online transpose it)
ConfusionMatrixDisplay(cm).plot()One habit worth stealing: print the matrix with labels, never as bare numbers. A transposed reading silently swaps precision and recall - a real and embarrassing class of bug.
Every metric answers one question 6 min live
The four cells combine into ratios, and each ratio answers exactly one question. Learn the QUESTION, not the formula, and you will never mix them up again: Accuracy - what fraction of everything did we get right? Precision - when we flag someone, how often are we right? Recall - of the real converters, how many did we catch? The fourth question - what does each mistake cost? - has no classic name, and it is the one the business actually asks. It gets its own session (b5).
LiveWhich question does YOUR problem ask first?4 min▶
The fastest metric-literacy drill there is - name the dominant error cost, and the lead metric follows:
- Cancer screening, fraud, safety incidents - a miss is catastrophic, an extra check is cheap: lead with recall.
- Spam filtering, auto-takedowns, alerts a human team must triage - a false alarm burns trust or labor: lead with precision.
- Lumen conversion outreach - a miss costs ~$74, a false alarm ~$2: recall-leaning, and b5 makes it exact.
Notice what you just did: you chose a metric by choosing whose pain matters more. That is not a technicality - it is a business decision wearing a math costume, and it should be made by someone who knows it is being made.
Self-studyWhere the LLM half will rhyme with this3 min read▶
Hold onto the precision/recall pair - it comes back wearing new clothes in the LLM sessions:
- Microsoft's Foundry docs define groundedness as the precision aspect of a RAG answer (no fabricated content) and completeness as the recall aspect (no critical information missed) - the same two questions, asked of sentences instead of sessions (b6).
- An LLM judge that calls answers "good" has its own precision and recall against human labels - and its own documented biases (b7).
- Even pass@k vs pass^k (from the agents course) is the same shape: capability ceiling vs reliability floor.
One matrix, many costumes. If you can read the four cells, the whole metric zoo becomes familiar.
Drag the threshold, watch the cells trade ★ 12 min · everyone runs it
Below is Lumen's conversion model evaluated on a real 2,000-session slice - the scores are fixed, the thresholding and every metric are computed live in your browser. The model outputs a score per session; the threshold decides which scores count as "will convert." Drag it and watch the four cells - and every metric built on them - move together.
Read the default (threshold 0.50): precision 21.5%, recall 63.6%. Most flags are wrong, and a third of buyers are missed - and this is a REASONABLE model on a 3.2% base rate. Calibrate your expectations to rare events.
Drag right to ~0.9: precision rises toward 100%, recall collapses. Every flag is right, almost nobody is caught. Note accuracy stays serenely high the whole way - the paradox from Part 1, live.
Drag left to ~0.1: recall hits 100%, precision falls to single digits. You caught everyone by flagging half the store.
Say the lesson out loud: precision and recall trade through the threshold. A model does not HAVE one precision and one recall - it has a curve of them, and someone picks a point.
Leave the two buttons alone tonight. They are b5's story - if you already pressed one, you have met Goodhart's law early.
Break accuracy on purpose ★ 8 min · explore
Push the threshold to 0.98 - the model now flags almost nobody, which is nearly the "always say no" model from Part 1. Watch the tiles: accuracy sits near 97%, F1 and recall are wreckage, and the error cost climbs. One slider position, three different verdicts, depending on which tile you trust.
At threshold 0.98, write down all five tiles. Rank the tiles by how alarmed they are. Accuracy is the calmest - and the most wrong about the situation.
Find the threshold where F1 peaks by feel (drag slowly around 0.55-0.65). Notice the error-cost tile does NOT bottom out at the same place. Hold that thought - it is the entire plot of b5.
Set the FN cost box to 5 (pretend a missed buyer barely matters). Watch the cost-vs-F1 disagreement shrink. The metrics did not change - your business did. That is why no metric is "correct" absent a cost story.
This slider is a real production control. sklearn's own docs now say the default 0.5 cutoff is "most certainly not ideal for most use cases" and ship TunedThresholdClassifierCV to tune it against a business scorer. Teams that treat the threshold as a fixed constant are leaving the single cheapest model improvement on the table - no retraining, one number.
Try it yourself - this week ◐ 20-30 min total
- Drag the evaluator until you can predict, before looking, which direction each of the five tiles moves when the threshold rises. Say each one's reason in the four-cell vocabulary.
- Draw the confusion matrix from memory for a problem YOU own - name the two error types in business language and put a rough dollar or hour figure on each. Bring it to b5; it becomes your cost matrix.
- Find one metric quoted in a report or vendor deck this week and ask the Part 2 question: which mistakes does this number hide? Write two sentences.
- Optional refresher if the sklearn side felt rusty: intro-ML b7 (classification metrics hands-on) - this course assumes that muscle memory.
Official sources covered
This track teaches the working core of the canonical metric sources - sklearn's evaluation guides, Google's crash-course classification module, Ng's metric doctrines, the key papers, and the LLM-era vendor guidance - with a live evaluator on the key pages. Certificates and graded assessments stay with the official providers (linked each session). This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Lumen's "always predict no" model scores 96.8% accuracy. Why?
Accuracy on imbalanced data mostly measures the base rate. Rare-event problems - fraud, churn, conversion, disease - are exactly where accuracy flatters useless models. Read the matrix instead.
2 · Marketing complains: "half the sessions you flag never buy." Which metric are they complaining about?
"When we flag, are we right?" is precision's question - TP / (TP + FP). Recall would be the complaint from finance: "we missed buyers." Matching the stakeholder's pain to the metric is the literacy this course trains.
3 · Why does dragging the threshold change precision and recall in opposite directions?
The scores never change - only the cut line does. Fewer, more confident flags = cleaner flags but more misses. A model has a CURVE of precision-recall pairs; the threshold picks the operating point, and b5 shows who should pick it.