The third project killer 8 min live
The course spine names three project killers, and this is the last one: static evals in a drifting world. The story is always the same. The system passed its eval at launch. Then the data drifted, the vendor silently upgraded the model underneath it, and users found phrasings nobody had tested. Six months later a ticket arrives: "the AI stopped working" - and nobody can say when it stopped, or why, because after launch day nothing was measuring. The fix is not a better metric. It is turning evaluation from an event into a loop.
LiveAnthropic's split: capability vs regression evals5 min▶
Anthropic's agent-eval guidance draws one line through every test you will ever write, and the line is the question the test answers:
- Capability evals ask "what can it do?" They start at low pass rates and climb as you build - a capability eval at 35% is a roadmap, not a failure.
- Regression evals ask "does it still work?" They hold at ~100%, and any dip blocks the merge. A regression eval at 96% is not "pretty good" - it is an alarm.
Most teams only ever build the first kind, run it once, and call the project evaluated. The second kind is what keeps b1-b8's numbers true next quarter: yesterday's hardest capability cases graduate into today's regression suite, and the suite runs on every change - your code, your prompts, your data, and (Part 2's point) the vendor's model.
The same guidance names the working posture: eval-driven development - write the evals BEFORE building the capability, the way test-driven developers write the test before the function. The eval defines "done"; the build chases it. Teams that eval after building inevitably eval what they built instead of what was needed.
LiveGolden-set-first, and the pass@k recap5 min▶
If you adopt one habit from tonight, adopt this one: before any build or fix, spend one hour collecting 20-50 cases from real failures into a golden set. It is the highest-ROI hour in applied ML - it converts "the AI seems off" into a number you can watch move, and it is the seed the whole loop above grows from. The first project killer (no golden set before building) and the third (static evals) have the same cure: the set exists early, and it keeps growing.
For agents, recall the two aggregate scores from the agents course - same data, opposite questions:
- pass@k: at least one of k trials succeeds - the capability CEILING. "Can it ever do this?"
- pass^k: all k trials succeed - the reliability FLOOR. "Can I depend on it?"
Capability evals naturally report pass@k flavors; regression evals should think in pass^k, because production does not get k attempts.
The vendor upgrade nobody announced. A support-bot team saw resolution quality slide over a weekend. No deploy, no data change - the API provider had upgraded the underlying model in place. The team had no regression suite pinned to a model version, so the first detector was a customer complaint thread. A 40-case golden set on a nightly cron would have caught it Saturday morning, with the delta labeled and dated. That suite now exists; it was written the following Monday, which is the expensive way to schedule it.
Three gaps, three owners 10 min live
"Offline says 0.83, production feels broken" is not one mystery - it is three possible gaps stacked in a ladder, and Rule #37 of Google's Rules of ML names each rung. Measure the same metric at four checkpoints - training data, holdout, next-day data, live traffic - and the LOCATION of the gap tells you which team owns the fix. Debugging all three rungs as one blob is how a two-day fix becomes a two-month standoff.
| Gap | What it means | Who owns it | The fix lives in |
|---|---|---|---|
| 1 · Train vs holdout | Overfitting - the model memorized instead of learning | Modeling | Regularization, simpler models, more data |
| 2 · Holdout vs next-day | Drift or stale features - the world moved since the snapshot | Data | Feature freshness, retraining cadence, drift checks |
| 3 · Next-day vs live | Serving skew - the pipeline itself disagrees with training | Platform | The serving path: logging, transforms, feature parity |
LiveRung 3 is the strange one - and the quote to keep4 min▶
Rungs 1 and 2 are statistics; rung 3 is not. A gap between next-day data and live traffic, in Rule #37's words, "typically signals engineering errors, since identical inputs should produce identical outputs." The model is fine, the data is fine - the serving pipeline is computing a feature differently than training did, dropping a field, or applying yesterday's encoder. That is why the ladder assigns it to platform, not to the modelers: no amount of retraining fixes a transform that disagrees with itself.
The diagnostic discipline: three gaps, three owners, three different fixes - never debug all three as one mystery. When someone says "the model got worse in prod," your first question is now: worse at which rung?
LiveVersion pinning, or: your vendor is on the ladder too4 min▶
The classic ladder assumed the model only changes when YOU change it. LLM systems break that assumption - the model can change under your feet. The discipline that restores it:
- Pin model versions - exact version strings in config, never "latest". An unpinned model is an unowned rung 3.
- Re-run the full suite on ANY model change - yours or the vendor's. A version bump is a deploy; treat it like one. Rule #24's companion habit applies: measure the delta between the new model and the current one on the same data before swapping.
- Judge models too - b7 made the judge a measurement instrument with a calibration number. A judge-model upgrade silently re-calibrates every score it has ever produced, so the judge's version is pinned and its human-agreement number is re-measured on change, same as any other model.
The two-hop chain, completed ★ build-along · 14 min
b5 planted the flag; tonight the full picture. Rule #39: your launch metrics are proxies. The chain is eval score -> online metric -> business goal, and each hop breaks independently, which makes it a diagnostic tool and not just a humility exercise. Offline up, revenue flat is usually hop 1-2: drift or serving skew (Part 2's ladder), or an eval metric detached from real user pain. Engagement up, satisfaction down is hop 2-3: the classic clickbait failure, where the online metric itself stopped meaning what you hoped. Name the hop before proposing the fix.
LiveProduction sampling in practice5 min▶
The loop's fourth box, concretely: score a slice of live traffic on the SAME metrics as your offline eval, and alert on deltas against the eval baseline. Not all traffic - a sample; not new metrics - the same ones, so a delta means the world moved, not the ruler. Practical shape:
- Sample 100-500 interactions a day (stratify by segment if b5's constraints are per-segment).
- Score with the same graders the offline suite uses - code graders directly; LLM judges with the pinned judge version from b7.
- Alert when a metric leaves its band vs the eval baseline; route the alert by the skew ladder's owner map.
- User feedback (thumbs up/down) is a cheap online signal with known biases - angry users click more than happy ones, and most users click nothing. Use it as a trend line and a failure-case miner, never as the metric itself.
Now build the nightly job - the whole session in one pseudo-snippet. This is deliberately boring code: a cron, a sample, a comparison, an alert. The value is not in the cleverness; it is in the fact that it runs every night, forever.
Sample yesterday's traffic and score it with the exact metrics and graders the offline suite uses. Same ruler, new data.
Load the eval baseline recorded against the PINNED model version. If nobody wrote the baseline down at ship time, tonight's first action item writes it down.
Compare, and alert on any metric outside its band - routing by owner: modeling, data, or platform, per the ladder.
Queue the failures for golden-set review. This line is the loop closing: production's surprises become next week's regression cases.
# nightly_eval.py - the continuous loop as a cron job
sample = sample_production_traffic(n=200, day="yesterday")
scores = run_metrics(sample, metrics=SAME_METRICS_AS_OFFLINE)
baseline = load_baseline(model_version=PINNED) # written at ship time
for name, value in scores.items():
delta = value - baseline[name]
if abs(delta) > ALERT_BAND[name]:
page_the_owner(name, delta) # skew ladder decides who
queue_for_golden_set(scores.failures) # the loop closes hereTry it yourself - this week ◐ 30-40 min total
- For one system you own, sort its existing tests into capability vs regression. If the regression pile is empty, promote your 5 hardest passing capability cases into it and wire them to block merges.
- Spend the golden hour: collect 20-50 real failure cases for a system that has none. Date the file - the freshness date goes on b10's sheet.
- Take one past "worse in prod" incident and place it on the skew ladder: which rung, which owner, and was it debugged that way at the time? Two sentences.
- Check one production model or LLM dependency for version pinning. If it says "latest," draft the one-line config change and the re-run-the-suite ritual that goes with it.
- Hands-on tracing, observability dashboards, and CI wiring for LLM evals live in the sibling course: learn-ai-evals-with-phoebe. This session gave you the doctrine; that course gives you the harness.
Official sources covered
Certificates and graded assessments stay with the official providers (linked each session). This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · In Anthropic's split, which statement describes a REGRESSION eval?
Capability evals ("what can it do?") start low and climb; regression evals ("does it still work?") hold ~100% and gate every change. Option C is the one-shot pattern - the third project killer this whole session exists to kill.
2 · Holdout metrics look fine, but the same metric sags on next-day data. On Rule #37's ladder, who owns the fix?
Rung 2 (holdout vs next-day) belongs to data: freshness, retraining cadence, drift checks. A is rung 1 (train vs holdout); B is rung 3 (next-day vs live), which "typically signals engineering errors." Three gaps, three owners.
3 · The offline eval score improved for a quarter, but revenue is flat. Which hop of the two-hop chain most likely broke?
Offline-up-revenue-flat is the signature of hop 1-2: the eval score moved but the online metric never followed. Hop 2-3's signature is different - engagement up, satisfaction down. Name the hop first; the fix follows from the location.