How the leader track works
Six sessions, 45 minutes each, and every one ends the same way: with questions to ask your teams, not homework to code. The shared case is Lumen Skincare's conversion model - an $18M/yr DTC brand whose classifier scores every checkout session for "will this person buy?" (about 3.2% actually do, average order ~$74). a1 and a2 cover the classic side: why single numbers mislead, and how thresholds and costs turn metrics into money. a3 and a4 translate the LLM metric zoo - quality, speed, cost - into plain language. a5 names the three ways evaluation projects die, and a6 hands you the governance playbook. One more thing worth knowing: the practitioner track of this course teaches your data scientists to WRITE the evaluation sheets. This track teaches you to READ them - and to notice what is missing.
| Session | The decision it equips you for |
|---|---|
| a1 · Why one number lies | Tonight - reading any model report without being fooled by its headline number |
| a2 · Precision, recall, and money | Who sets the threshold, and what optimizing the wrong metric costs in dollars |
| a3 · LLM quality metrics, plain | Reading groundedness, hallucination, and judge scores with healthy skepticism |
| a4 · Speed and cost | Latency and price as first-class metrics, not infrastructure footnotes |
| a5 · The three project killers | Spotting the failure patterns that quietly sink evaluation efforts |
| a6 · Your eval governance playbook | The scorecard to demand, the vendor grilling, the review cadence |
The 96.8% model that finds nobody 15 min live
Lumen's conversion rate is about 3.2%. So here is a model your team could build in zero minutes: always say "will not buy." It is right about 96.8% of the time - a grade most real models would envy - and it finds exactly zero customers. That is the accuracy paradox, and it is not a curiosity: fraud, churn, conversion, defects, and disease are all rare-event problems, which means the problems where models are most valuable are exactly the problems where accuracy flatters useless models.
LiveThe paradox in one boardroom sentence7 min▶
Google's ML Crash Course states the general form almost word for word: on data where a class appears 1% of the time, a model predicting negative 100% of the time scores 99% accuracy "despite being useless." The boardroom version: on rare-event problems, accuracy mostly measures how rare the event is, not how good the model is.
- Google's own guidance: accuracy is a rough progress indicator for balanced data only.
- Lumen's base rate is 3.2% - so 96.8% accuracy is the score for doing nothing at all.
- The valuable problems - fraud, churn, conversion, safety, disease - are almost all rare-event problems. The lie lives exactly where the money is.
The 99%-accurate fraud model. This story pattern repeats across the industry: a bank's new fraud model reports 99%+ accuracy and gets fast-tracked. First month in production it catches almost nothing - fraud was a fraction of a percent of transactions, and the model had learned to say "legitimate" nearly always. Nobody in the approval chain had asked to see the mistakes broken out by type. One glance at the missed-fraud line would have stopped the launch. That glance is what this session installs.
LiveFour cells, four line items on your P&L8 min▶
Every yes/no model produces exactly four outcomes. Your teams call them TP, FP, FN, TN; you can call them by what they cost. At Lumen the two mistakes have wildly different prices - and no single number tells you which mistake it is making.
| Outcome | In business language | Who feels it | Price at Lumen |
|---|---|---|---|
| True positive | Flag-and-right: flagged a session, they bought | Everyone smiles | the win |
| False positive | Flag-and-wasted: flagged, nobody bought | Marketing's budget | ~$2 wasted outreach |
| False negative | Missed-buyer: a real buyer, never nudged | Finance's revenue line | ~$74 lost order |
| True negative | Correctly-ignored: no flag, no purchase | Nobody - and that is fine | free |
Hold the asymmetry: a miss costs about 37 times a false alarm here. Session a2 turns that ratio into a dollar figure your teams can optimize - and shows why the metric they usually optimize ignores it completely.
Self-studyWhen a matrix lands in your inbox: a 60-second reading drill3 min read▶
Sooner or later a team will actually send you the four cells. Read them in this order and you cannot be fooled:
- 1 · Find the base rate first. Add up the "actually happened" column and divide by the total. Everything else is judged against it.
- 2 · Go straight to the miss cell. The false negatives are where the expensive silence lives - the 96.8% model hides all its damage there.
- 3 · Check the labels before trusting anything. Different tools draw the matrix rotated or transposed - confirm which axis is "model says" and which is "actually happened" before you read a single count.
- 4 · Compare to the do-nothing model. Would always-say-no produce a similar-looking matrix? If yes, the model is decoration.
- 5 · Ask for prices on the two error cells. Counts without costs are half a story - and a2 is about the other half.
Metric literacy as self-defense 7 min live + self-study
You do not need to compute a single metric to defend yourself against one. You need exactly one question, asked out loud, every time someone quotes a single number: "which mistakes does this number hide?" Accuracy hides which of the two error types it is making. A precision figure hides the buyers being missed. A recall figure hides the outreach being wasted. Every summary is a choice about what not to show - your job is to ask for the part that was chosen away.
LiveThe one question - and the rare case where accuracy is fine4 min▶
"Which mistakes does this hide?" works on any metric because every metric is a ratio of the four cells - it keeps some, drops others. Andrew Ng teaches the same defense to engineers as "error metrics for skewed datasets": on rare events, stop trusting the average and look at the error types separately. The toolmakers agree - scikit-learn, the standard ML library, ships an alternative to plain accuracy precisely because accuracy gives "inflated performance estimates" on imbalanced data. Their words, in the documentation.
Is accuracy ever fine? Yes, when BOTH of these hold - and notice how rarely they do:
- Balanced classes - the outcomes are roughly 50/50, so the do-nothing model scores 50%, not 96.8%.
- Equal costs - both mistake types hurt about the same, so it is fair to average them.
Lumen fails both tests: 3.2% base rate, and a 37x cost gap between the two errors. Most problems worth modeling fail both.
LiveThe leader's five questions for any model report3 min▶
Ask these in order. A strong team answers all five in under two minutes; a weak report survives none of them.
| # | The question | What a good answer sounds like |
|---|---|---|
| 1 | What is the base rate? | "3.2% convert - so doing nothing scores 96.8%, and here is how far we beat that." |
| 2 | Show me the confusion matrix. | Four labeled cells with counts, already in the deck - not "we can pull that." |
| 3 | Who chose the threshold, and why? | A name and a reason. "The default" means nobody chose - a2 is about this. |
| 4 | What does each error type cost? | Two different prices, even rough ones - a miss and a false alarm are different invoices. |
| 5 | Evaluated on what slice? | Which data, from when, and whether it looks like next quarter's customers. (a5 goes deep here.) |
Self-studyPrecision vs recall: just listen to who is complaining3 min read▶
Your teams will say "precision" and "recall" constantly. You never need the formulas - you need to know whose complaint each one is:
- Marketing complains = precision. "Half the sessions you flag never buy" - the flags are polluted. Precision asks: when we flag, are we right?
- Finance complains = recall. "We missed buyers worth $74 each" - real money walked past unflagged. Recall asks: of the real buyers, how many did we catch?
The two complaints trade against each other through the threshold - flag more people and marketing suffers, flag fewer and finance does. You will feel that trade in your own hands in the demo below, and a2 shows who should be choosing the balance point (spoiler: someone who knows both prices).
Self-studyFive red flags in a metric slide3 min read▶
Pattern-match these before your next vendor meeting or internal review. None of them proves the model is bad - each one proves a question was skipped:
- Accuracy with no base rate next to it. You now know why: without the base rate, the number is unreadable. Ask question 1.
- One blended score and nothing else. A single number is a summary with an opinion. Ask which mistakes it hides.
- No mention of a threshold anywhere. Every yes/no model has the dial - silence means the default is doing the deciding.
- "Up to" phrasing. "Up to 99% accurate" describes the best point on a curve under the friendliest conditions - ask for the operating point they actually run at.
- Error types never separated. If misses and false alarms are never shown apart, nobody has priced them - and the 37x asymmetries stay invisible.
The pattern behind all five: a summary was offered where a decision should be. The five questions from the previous card are the antidote, one for one.
Drag a real threshold ★ 20 min · everyone tries it
Below is Lumen's actual conversion model, evaluated on a 2,000-session slice - the scores are real, and every tile recomputes live in your browser as you drag. The slider is the model's threshold: how sure it must be before it flags a session as "will buy." No code, no setup. Drag it and watch which numbers panic and which stay eerily calm.
Read the default tiles first (threshold 0.50): precision 21.5%, recall 63.6%. Most flags are wrong AND a third of buyers are missed - and this is a perfectly reasonable model on a 3.2% base rate. Calibrate your expectations: rare-event models live in this range, and a vendor promising 95% on both is describing a unicorn.
Drag the slider right and watch two things at once: accuracy stays serene - drifting calmly in the high 90s - while recall dies, collapsing toward zero as the model flags almost nobody. That is Part 1's paradox, live: the headline number cannot feel the buyers being lost.
Now drag left toward 0.1: recall climbs to 100% - every buyer caught - while precision falls to single digits, because you flagged half the store to do it. Feel the trade in your hand: this is marketing's complaint and finance's complaint on one slider.
You will notice two buttons under the slider. Leave them alone tonight - they are the entire plot of a2, and they are worth $704. If you already pressed one: yes, the "best" metric and the cheapest outcome disagree. Hold that thought until next session.
This week, in your own meetings ◐ 15 min total
- Find one single-number claim in a report, dashboard, or vendor deck this week and ask the question out loud: "which mistakes does this number hide?" Write down what happens next.
- Ask one team for the confusion matrix behind a model you sponsor. If it arrives quickly with labels and counts, good sign. If it takes days, you have learned something more important than the numbers.
- Ask what the base rate is for a prediction problem you fund. Then compute the do-nothing score yourself: 100% minus the base rate. That is the floor any accuracy claim must clear by a lot.
- Put rough prices on the two mistake types for one decision your org automates - even "$2-ish vs $74-ish" precision is enough. Bring them to a2; they become the cost matrix that picks the threshold.
Official sources covered
The leader track teaches the decision-relevant core of the canonical metric sources - Google's crash-course classification module, scikit-learn's evaluation guidance, and Ng's metric doctrines - in business language, with the live evaluator doing the math. Certificates and hands-on assessments stay with the official providers (and with the practitioner track). This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A model at Lumen scores 96.8% accuracy while finding zero customers. What causes this?
On rare events, accuracy mostly measures the base rate, not the model. Doing nothing scores 96.8% at Lumen. That is why the valuable problems - fraud, churn, conversion, disease - are exactly where accuracy flatters useless models.
2 · Marketing storms in: "half the sessions your model flags never buy anything." Which metric is that complaint about?
Polluted flags are precision's complaint, and it comes from whoever pays for the false alarms - here, marketing's outreach budget. Missed buyers would be finance's complaint, and that one is recall. Match the stakeholder to the metric and you never mix them up.
3 · A vendor deck leads with one impressive number. What is the single question that exposes it?
Every metric is a ratio of the four cells - it keeps some mistakes visible and drops others. Asking which ones were dropped forces the confusion matrix onto the table, and the four cells with prices attached tell you what the summary chose not to say.