The eval sheet: eight slots, no blanks 8 min live
Everything this course taught compresses into one page with eight slots. Each slot exists because a session showed what happens without it - the paradox (b1), the unpriced threshold (b5), the untrusted judge (b7), the invisible latency (b8), the eval that never ran again (b9). Fill all eight or say honestly which are open.
1 · The task + base rate. What is predicted or generated, on what population, with what prevalence. A metric without its base rate is unreadable (b1).
2 · The cost matrix, with named price owners. FP and FN in dollars or hours, and WHO signed each number. Unsigned prices are the sheet's most common blank (b5).
3 · One optimizing metric. The single number you actually maximize - the money when possible. One, for iteration speed (Ng's doctrine, b5).
4 · Satisficing constraints. Floors and ceilings with owners: recall floors, precision floors ops can absorb, toxicity ceilings. Constraints have prices too (b5).
5 · The golden set. Size, where the cases came from, when it last grew. 20-50 real cases before building - the highest-ROI hour (b9).
6 · The grader spec. Code, LLM-judge, or human per metric; if a judge: its model version AND its human-agreement number. A judge score without agreement data is hope (b7).
7 · Operational budgets. p95 TTFT, cost per query, throughput - as eval criteria, not infra footnotes (b8; Anthropic lists latency and price as success-criteria categories).
8 · The continuous plan. CI gate, production-sampling cadence, re-eval triggers - including "the model version changed", yours or the vendor's (b9).
LiveWhy one page, and why owners3 min▶
Two design rules make the sheet work in real organizations:
- One page, hard limit. A sheet nobody reads protects nobody. Eight slots fit on a page; the supporting notebooks can be as long as they like, linked from it.
- Every line has an owner who can veto it. The product owner owns the prices, DS owns the metrics and graders, platform owns the CI gate and budgets. A number without an owner is a number nobody will defend in the incident review - which is exactly when the sheet is read.
The sheet is the meeting. Teams that adopt this format report the same effect: the fights move from launch week to sheet-writing week, where they are cheap. Marketing arguing the FP price down, ops arguing the precision floor up, finance asking why latency has no budget - that argument IS the evaluation working. Silence at sheet time becomes an incident later.
Sheet 1: the Lumen conversion model ★ 10 min · everyone writes
Fill the eight slots for the classic model, live: 1 predict `converted` per checkout session, base rate 3.2%. 2 FP $2 (marketing signed), FN $74 (finance signed). 3 minimize expected error cost on the held-out slice. 4 precision >= 8% (ops absorbs the alert volume), no customer segment's recall below 80%. 5 the 2,000-session evaluation slice + quarterly refresh. 6 grader = code (label comparison - classic problems get exact graders free). 7 n/a for a batch scorer - scoring latency budget only. 8 re-tune threshold quarterly and on any drift alert. Now verify the sheet against the evaluator - it opens at the sheet's chosen operating point.
Confirm slot 3's answer: the evaluator opens at t=0.22 - the cost-optimal threshold, $1,146 on the slice. Check slot 4 holds: precision 8.8%, above the 8% floor. Barely - which means the floor is a REAL constraint, not decoration.
Press Optimize F1 once more, as the red team will: F1 0.706 at t=0.61, cost $1,850. The sheet's justification for its ugly-looking operating point is this exact $704 gap - write it into the sheet's notes line.
Change the FN price to $40 and press Optimize $: the decision moves. The sheet is alive - when finance re-prices a lost order, the threshold re-tunes. That is slot 8 working, not a flaw.
TunedThresholdClassifierCV(model, scoring=make_scorer(lumen_gain)) - the b5 snippet IS the sheet's optimizing metric in code. The sheet documents; the code enforces; CI (slot 8) makes sure they never drift apart.
Sheet 2: the Lumen RAG assistant ★ 12 min · everyone writes
Same eight slots, LLM entries: 1 answer customer + analyst questions over Lumen's docs and data, 30 intents. 2 a wrong-but-confident answer costs a support escalation (~$15) or a bad business decision (priced per intent tier); a refused answerable question costs a session. 3 maximize task success on the golden set. 4 faithfulness >= 0.9, relevance floor, zero PII leaks (a binary constraint - some floors are absolute). 5 30 cases from real support transcripts, grown monthly from production misses. 6 faithfulness graded by an LLM judge - REQUIRED: judge model version pinned + human agreement >= 85% on the anchor set. 7 p95 TTFT <= 800ms, cost <= $0.02/query. 8 regression suite holds 100% in CI; full re-eval + judge re-calibration on ANY model version change. Slot 6 is the one most sheets fake - so run it for real:
Run the raw judge: 58% agreement - this judge FAILS the sheet's >= 85% requirement. An eval built on it would be measuring verbosity, not faithfulness. The sheet just rejected its own grader; that is the sheet working.
Switch to the length-controlled judge: 88% - passes the bar. Write BOTH numbers into slot 6: the bar, and the measured agreement. Now the faithfulness floor in slot 4 means something.
Note the discipline this created: before this course, "faithfulness 0.92 ✓" looked like a fact. Now you read it as "0.92 according to WHOM, agreeing with humans HOW often?" - the b7 reflex, institutionalized in a slot.
Where the machinery lives. This sheet's slots 5-6 are implemented with the tooling the sibling learn-evals course teaches hands-on: RAGAS for the faithfulness pipeline, golden-set harnesses, tracing. The sheet is deliberately tool-agnostic - vendors change (OpenAI's evals platform sunsets November 2026), the slots do not.
Defend every line ★ 8 min · pair up
The ship gate: each sheet line gets the hostile question it must survive. Pair up - one reads a line, the other attacks it. A line that folds gets fixed or honestly marked open; a sheet that survives is an evaluation.
LiveThe red-team question bank6 min▶
- Slot 2: "Who signed the $74 - and would they sign it again today?" (Prices age. Quarterly re-validation is slot 8's job.)
- Slot 3: "Why is your optimizing metric not just F1 like every other team?" (Answer with the $704 gap, from the evaluator, on demand.)
- Slot 4: "What does the precision floor cost?" (Every constraint moves the optimum - you measured it in b5's Demo 2.)
- Slot 5: "When did the golden set last grow, and from what?" ("At launch, from brainstorming" is a failing answer - production misses are the feed.)
- Slot 6: "What is your judge's agreement number, and when did you last re-measure it?" (No number = no metric. Judge model changed = re-calibrate.)
- Slot 7: "p95 or average?" (Averages hide the tail the users actually feel.)
- Slot 8: "What happens when the base model silently updates?" (Version pins + the full-suite trigger, or you find out from customers.)
- The whole sheet: "Which online metric is this a proxy for, and who watches THAT?" (The two-hop chain - hop one celebrated as hop one only.)
Self-studyWhere to go next2 min read▶
- learn-ai-evals-with-phoebe - the LLM eval harness hands-on: RAGAS pipelines, LLM-judge implementations, tracing, CI wiring. This course's slots 5-6-8, in code.
- learn-intro-ml-with-phoebe - the sklearn mechanics under sheets like Sheet 1, including the model this course evaluated.
- The leader track (a1-a6) - the same material as governance: hand it to whoever reads your sheets.
- The honest frontier: fairness metrics (demographic parity, equalized odds) are the next column this sheet grows - named here, taught in a future course.
The real capstone - this week ◐ 60 min, worth it
- Write the eight-slot sheet for ONE system you actually own - classic or LLM. Use the b1 and b5 homework artifacts you already built (your confusion matrix, your prices, your eval line) as slots 1-4.
- Get one real signature: take slot 2 to the person who owns the money and have them confirm or correct the prices. Their correction is the most valuable data point this course produces.
- Run the red-team bank against your own sheet, alone, honestly. Mark the lines that folded as OPEN rather than papering over them - an honest gap beats a decorative number.
- Schedule the first quarterly review in your calendar now. Slot 8 dies by forgetting, never by decision.
Official sources covered
The capstone synthesizes the whole track. Certificates and graded assessments stay with the official providers. This page draws on:
Three questions before you go 🎯 ◐ 90 seconds
1 · Sheet 1 ships at threshold 0.22 with F1 0.161 - the worst-looking F1 on the card. Why does it survive the red team?
The sheet declares its optimizing metric (expected cost at signed prices) and can produce the evaluator evidence on demand. Ugly satellite metrics at the priced optimum are expected. (C is false - the optimum follows the cost RATIO, which can point either direction.)
2 · Sheet 2's raw judge showed 58% human agreement. What does the sheet require you to do?
A grader below the bar invalidates every number it grades - fix the grader (here: length control, 58% -> 88%), never the standard. Averaging two uncalibrated judges just averages two unknown biases.
3 · Which slot answers "what happens when the vendor silently upgrades the base model"?
The continuous plan owns change. The golden set being model-independent is exactly why re-RUNNING it on the new model is the detection mechanism - and the judge is a model too, so it re-calibrates (b7).