Learn Model Evaluation with Phoebe

Two tracks into metrics that actually pay

Every metric is an opinion about which mistakes matter - and most teams inherit the opinion without noticing. This course covers both metric families in one arc: the classic canon (precision, recall, F1, ROC, calibration, cost matrices) and the LLM-era system (groundedness, hallucination rates, LLM-as-judge, latency and cost per query). A leader track teaches you to read any model report without being fooled. A practitioner track evaluates Lumen's real conversion model end to end - and a live evaluator runs in your browser: drag a real threshold on 2,000 scored sessions, press "Optimize F1" and watch it cost $704 more than the money-optimal point, then calibrate an LLM judge yourself and catch its verbosity bias red-handed.

2tracks
16sessions
704$ the Goodhart gap
45min per session

The leader track 🤝 · for executives, PMs & anyone who reads model reports · 6 x 45 min · no code

Metric literacy as self-defense: the accuracy paradox, the threshold as a business decision someone is silently making, LLM quality numbers in plain English, speed and cost as product features, and the three failure patterns that quietly kill AI ROI - each session ends with the questions to ask your teams.

🎭 Leader 1 · easy

Why one number lies

A 96.8%-accurate model that finds zero customers - the accuracy paradox on real numbers, and the first questions that expose it.

▶ Leaders start here 💰 Leader 2 · easy

Precision, recall, and money

The threshold is a business decision wearing a math costume - the $704 demo, and who in your org should be picking the operating point.

▶ The Goodhart button 🗣️ Leader 3 · medium

LLM quality metrics, in plain English

Groundedness, hallucination rates, relevance - what the numbers mean, the two different things "hallucination" measures, and why judge scores are not measurements.

The judge caveat ⚡ Leader 4 · medium

Speed and cost

Time-to-first-token, cost per query, throughput - the operational metrics users actually feel, and the satisficing architecture that keeps quality honest about them.

Latency is a feature 💣 Leader 5 · deciding

The three project killers

No golden set before building, optimizing a proxy unwired to money, static evals in a drifting world - and the governance moves that prevent each.

The ROI session 📋 Leader 6 · deciding

Your eval governance playbook

The scorecard to demand before any launch, vendor-grilling questions, the two-hop proxy chain, and the review cadence that keeps evals alive.

Strategy session

The practitioner track 🧪 · for DS, analysts & AI engineers · 10 x 45 min · Python + the live evaluator

One running case, evaluated harder every session: Lumen's conversion model (built in intro-ML) plus an LLM assistant on the same brand. Classic half b1-b5 funnels into the cost matrix; LLM half b6-b8 covers quality, graders, and operations; b9 makes it continuous; b10 ships the full eval sheet. Hands-on RAGAS machinery lives in the sibling learn-evals course - this one owns the metrics.

🔲 Practitioner 1 · easy

The confusion matrix, cold

Four cells, two kinds of pain, the accuracy paradox on Lumen's 3.2% base rate - and your first drag of a real threshold.

▶ Practitioners start here 🎚️ Practitioner 2 · easy

The threshold dial

Precision and recall trade through one number - F-beta when costs lean, balanced accuracy, and why a model has a curve, not a score.

P/R fluency 📈 Practitioner 3 · medium

Curves + calibration

ROC-AUC vs PR-AUC under imbalance (the Saito-Rehmsmeier result), reliability diagrams, and the Brier-score caveat sklearn itself warns about.

The imbalance toolkit 📏 Practitioner 4 · medium

Regression + ranking metrics

MAE vs RMSE vs MAPE on Lumen's order values, R² without self-deception, NDCG for anything ranked - and the which-scoring-function doctrine.

The other targets 🧮 Practitioner 5 · the heart

The cost matrix

THE session: prices on the error cells, Elkan's rule, TunedThresholdClassifierCV - and the button that maximizes F1 while losing $704.

▶ The $704 button 🔍 Practitioner 6 · hands-on

LLM quality metrics

The faithfulness formula, groundedness on a 1-5 scale, relevance - and the world-vs-document distinction that untangles "hallucination".

Two hallucinations ⚖️ Practitioner 7 · hands-on

The grader problem

Code vs LLM vs human graders, the four documented judge biases, the 80%-equals-human-agreement bar - and a judge you calibrate yourself, live.

▶ 58% → 88% demo ⏱️ Practitioner 8 · hands-on

Operational metrics

TTFT, inter-token latency, the identity that ties them together, cost per query, the batching tradeoff - operations as first-class eval criteria.

The serving math 🌊 Practitioner 9 · hardest

Drift + continuous evaluation

Capability vs regression evals, golden-set-first discipline, the training/serving skew ladder, and the two-hop proxy chain that explains "offline up, revenue flat".

Evals as a process 🏆 Practitioner 10 · hardest

Capstone: the eval sheet

Write the full evaluation spec for one classic model and one LLM system - metrics, prices, floors, gates - and defend every line of it.

Ship the sheet
easy - everyone medium hands-on / deciding hardest

Choose your path 🗺️

Two doors into one conviction: a metric without a cost story is an opinion you have not read yet. Leaders learn to read them; practitioners learn to write them.

🤝 Leader (read any report) a1→ a2→ a3→ a4→ a5→ a6
🧪 Practitioner (write the sheet) b1→ b2→ b3→ b4→ b5→ b6→ b7→ b8→ b9→ b10
🤖 LLM-only fast lane b1→ b5→ b6→ b7→ b8→ b9
🚀 Full journey (read AND write) a1-a6→ b1-b10

The knowledge map 🧠

Both tracks at a glance - hover a session to spotlight its concepts, click any node to jump in.