The number that lies loudest
When a model is finished, it does not hand you a verdict - it hands you a report card full of numbers, and it stays silent about which one matters. Left alone, most rooms grab the single figure they already understand: accuracy. That is exactly the trap. For Lumen Skincare, whose visitors convert only about 3.2% of the time, a model that predicts "no buyer" for every single person is 96.8% accurate and completely worthless - it finds nobody. This session is the antidote: a short, jargon-free toolkit for reading the report card so no metric can bluff its way past you.
The accuracy paradox, and the matrix that tells the truth 10 min live
Accuracy is "what fraction did we get right?" It sounds unimpeachable, and on a balanced problem it is fine. On a rare event it collapses. Lumen converts about 3.2% of sessions, so a lazy model that always guesses "no" is right 96.8% of the time and catches zero buyers. The fix is to stop staring at one headline number and look at the confusion matrix - the four-cell table every other metric is built from.
Try it yourself. The tool below is Lumen's real conversion model. Drag the decision threshold and watch accuracy stay stubbornly high while the buyers it actually catches - the recall - swings wildly. That gap is the whole lesson.
LiveWhy 96.8% is a lie3 min▶
Accuracy answers "what share of all predictions were correct?" On Lumen's data that is dominated by the 96.8% of sessions that never convert. A model can win all of those by simply saying "no buyer" every time - and it will score 96.8% while finding exactly zero of the buyers you actually care about.
- The base rate rules everything. When one outcome is 30x rarer than the other, guessing the common one is almost always "right" and always useless.
- The valuable answers are the rare ones. You are hunting the 3.2%, so a score that rewards ignoring them is measuring the wrong thing.
- Ask the follow-up. "96.8% accurate - how many of the real buyers did it actually find?" That one question deflates the paradox instantly.
Lumen's marketing team once celebrated a "97% accurate" conversion model in a leadership deck. It held up right until someone asked how many of the month's buyers it had actually flagged. The answer was none - the model had learned to say "no" to everyone, banked the easy 96.8%, and found not a single customer. The applause stopped.
LiveThe confusion matrix3 min▶
The confusion matrix is a simple 2x2 tally of what the model predicted against what actually happened. Four cells, and every classification metric you will ever meet is built from them:
- True positive (TP) - a real buyer the model correctly flagged. The win.
- False positive (FP) - flagged as a buyer, never bought. A wasted offer or discount.
- False negative (FN) - a real buyer the model missed. A lost sale.
- True negative (TN) - correctly left alone. The easy majority on Lumen's data.
When someone shows you a single number, ask for the matrix. It is the only view that cannot hide a failure, because it shows you the buyers you missed right next to the ones you caught.
The metrics that matter, in plain words 9 min live
Once you have the matrix, four words carry almost all the meaning. Learn them as promises, not formulas. And learn the one that quietly runs the show: the threshold, the dial that decides how bold the model is about calling someone a buyer - a business choice, never a default of 0.5.
LivePrecision vs recall (and the threshold dial)3 min▶
Two promises, one trade-off:
- Precision - of those we flagged as buyers, how many really bought. High precision means few wasted offers.
- Recall - of the real buyers, how many we caught. High recall means few missed sales.
- F1 - the balance of the two, a single number for when you care about both roughly equally.
- ROC-AUC (~0.71 for Lumen) - how well the model ranks buyers above non-buyers, from 0.5 (a coin flip) to 1.0 (perfect). A real but modest signal here.
The dial that trades one for the other is the threshold. A model outputs a probability; the threshold is the cutoff above which you call it "buyer". Lower it and you catch more buyers (recall up) but flag more people who never buy (precision down). Raise it and the reverse. It defaults to 0.5, but 0.5 is arbitrary - the right cutoff depends on what a mistake costs you.
LiveJudging a regression in dollars3 min▶
When the model predicts a number instead of a category - like Lumen's order-value model - the report card changes. The two figures to know:
- RMSE - the typical size of the model's miss, in the same unit as the target. For Lumen, dollars. The dumb baseline of "just predict the average order" has an RMSE of about $41; the full model brings that down to about $28. That drop is the model earning its keep.
- R2 - the share of the variation the model explains, from 0 to 1. Useful to a data scientist, near-meaningless to a boardroom.
Always compare against the baseline: a number alone ("RMSE $28") means nothing until you know that guessing the average was $41. And report the dollars, not R2, to a CFO - "our typical miss shrank from $41 to $28 an order" lands; "R2 of 0.52" does not.
Lumen's threshold is set by arithmetic, not habit. A false positive - a discount sent to someone who was never going to buy - costs a few dollars of margin. A false negative - a real buyer who never got the offer - costs a whole sale. Because a missed buyer hurts far more than a wasted discount, Lumen deliberately lowers the threshold to lift recall, accepting more wasted offers to catch more buyers. The metric follows the money, not the default.
This week ◐ 25 min total
- Ask for the matrix. For one model in your org, ask the team for its confusion matrix, not just its accuracy. Look at the false negatives - the cases it missed - and decide if you can live with them.
- Pick your promise. For a real use case, decide whether precision or recall matters more, and why - what does a wasted alert cost versus a missed one? Write it in one sentence.
- Demand the baseline. Whenever someone quotes a metric, ask "what is the baseline?" A score only means something next to "predict the mean" or "always guess no".
- Optional: find a past deck where a headline accuracy number oversold a model. Bring the story to Session 5.
Three questions before you go 🎯 ◐ 90 seconds
1 · A Lumen model is 96.8% accurate at predicting conversion. Why might that be worthless?
This is the accuracy paradox. With a 3.2% base rate, guessing the common outcome ("no buyer") is right 96.8% of the time while finding none of the buyers you care about. Accuracy is the wrong score on imbalanced data.
2 · What does "recall" tell you about a conversion model?
Recall reads across the actual-buyer row: of all the real buyers, how many did the model catch? "Of those we flagged, how many bought" is precision, not recall.
3 · A regression model has an RMSE of $28, versus a "predict the average" baseline of $41. What does that tell you?
RMSE is the typical miss in dollars. $28 versus a $41 baseline means the model shrank the average error below what guessing the mean would give - it is genuinely adding value. Accuracy is a classification metric and does not apply to a dollar prediction.
What this session covers
This leader session teaches the framing behind the "how do we know a model is any good?" week of the major intro courses - the confusion matrix, the classification metrics, and the dollar view of regression. The code that computes them (classification_report, roc_auc_score) lives in Builder Session 7; the sources below go deeper.