Why this hour is worth your calendar
A language model always answers. It answers when it knows, and it answers - just as fluently, just as confidently - when it is guessing. That single fact is why AI projects that dazzle in the demo can fail quietly in production: nobody could tell the difference between a right answer and a convincing wrong one, because nobody was measuring. Evaluation is how you replace "it seems fine" with a number you can defend. It is the least glamorous part of an AI project and the one that decides whether you can trust it.
The confidence trap 9 min live
The model's fluency is the same whether it is right or wrong. Without evaluation, a right answer and a confident wrong one are indistinguishable - and the wrong ones are the expensive ones. This picture is the whole reason eval exists.
LiveWhy a great demo proves almost nothing3 min▶
A demo is a curated sample of one. Someone picked the questions, saw the answers, and showed you the good run. That tells you the system can be right - not how often it is right, not where it fails, not what it does on the messy inputs real users bring.
- Selection bias is built in. Nobody demos the failure. The demo is the highlight reel, and you are being asked to fund the whole season.
- The failures are silent. A wrong answer does not throw an error. It just quietly misinforms a customer, misprices a quote, or misroutes a ticket - and looks exactly like a right answer while doing it.
- Evaluation is the anti-demo. Instead of one hand-picked question, you run a hundred representative ones and count. That count is the thing you can actually stand behind.
LiveThe cost of a confident wrong answer3 min▶
Not all errors cost the same, and that is the leadership lens on evaluation. A wrong answer in a brainstorming tool is a shrug. A wrong answer in a refund policy, a medical summary, or a financial figure is a refund issued incorrectly, a compliance incident, a lost customer. Evaluation is how you buy down the risk on the answers that matter.
- Frequency times severity. A rare error on a trivial task is fine. A rare error on a high-stakes task may not be. Eval lets you measure the frequency so you can weigh it against the severity you already understand.
- "Good enough" is a number, not a feeling. 95% correct might be plenty for draft suggestions and nowhere near enough for automated payouts. Evaluation turns that judgment into a threshold you set on purpose.
The support bot that aged badly. A team (anonymized) shipped a help bot that tested beautifully on launch day. Three months later a policy changed, the underlying documents drifted, and the bot kept confidently quoting the old rule. No error was ever thrown; the first signal was a rise in complaints. An online eval on live traffic would have caught it in a day - the subject of session a3.
Self-studyWhy you cannot outsource this to "the model is smart now"2 min read▶
A fair objection: models are so capable now, do we really need to measure? Yes - and more, not less. Capability raises the stakes: a more fluent model produces more convincing wrong answers, and gets trusted with higher-value decisions. Capability does not remove the need to measure; it raises the cost of not measuring. Anthropic's own guidance to builders is blunt about it - define success criteria, then build empirical evaluations against them, before you trust the system. The smarter the tool, the more that discipline earns.
The three questions evaluation answers 8 min live
Evaluation is not one activity - it answers three different questions at three different moments. A leader who knows the three can tell whether a team is doing all of it or just the easy part.
LiveShip it, changed it, still good - the whole discipline in three moments3 min▶
- Ship it? (offline eval) Before launch, run the system against a fixed set of representative questions with known good answers, and measure. This is the gate: a number you decide is high enough to ship.
- Better or worse? (regression eval) Every prompt tweak, model swap, or retrieval change gets re-run against that same set. AI systems are famous for a "fix" in one place silently breaking three others. The eval catches it before your users do.
- Still good? (online eval) After launch, sample real production traffic and keep measuring. The world drifts - documents change, users ask new things, the model gets updated under you - and only online eval sees it.
Live"Can't the model just grade itself?" - a preview3 min▶
Often, yes - and it is one of the most useful and most dangerous ideas in the field. Using a strong model as an automated judge (LLM-as-judge) is how eval scales past what humans can hand-grade. It genuinely works: on public benchmarks a top model agrees with human preferences around 85% of the time. But it has known blind spots - it can favor longer answers, favor whichever answer it sees first, and favor its own style.
- The leader takeaway: LLM-as-judge is legitimate and necessary at scale, but it must be calibrated and spot-checked against humans - not trusted blindly. Session a5 covers when to trust it; the builder track b5 shows how.
The metric that got gamed. A team (anonymized) optimized against an LLM judge that quietly rewarded longer answers. The system learned to pad. The scores climbed; the customers complained the answers had gotten wordy and vague. The fix was not a better model - it was catching the judge's verbosity bias and controlling for length.
Price a wrong answer ★ 12 min · pen and paper
Evaluation earns its budget when you can say what a wrong answer costs. Do it for your own use case - no code, a pen, and honest numbers.
Name one AI use case you own or are considering. Write the single most consequential question a user could ask it.
Write what a confident wrong answer to that question actually causes - in money, risk, or trust. Be concrete: "a refund issued that should not have been", not "a bad experience".
Estimate how often the system would need to be wrong for that cost to be unacceptable. That number is your ship threshold - the thing eval will measure against.
Ask: does anyone currently know the real error rate? If the answer is "no", you have just found the business case for this whole course.
The "how would we know" audit ★ 8 min · your next AI review
Four questions that reveal, in one meeting, whether a team is actually measuring or just hoping.
"How good is it, as a number, and against what set of questions?" A shrug or a demo in reply means there is no offline eval.
"When you changed the prompt last week, how did you know it did not break anything?" This is the regression-eval question. "We tried a few things" is not an answer.
"If it started giving worse answers in production tomorrow, how long until we noticed?" The honest answer for most teams is "when customers complain". Online eval shortens that to hours.
"Who owns the eval, and does it run automatically?" If eval is a person's occasional side task, it will lapse. It needs an owner and a schedule.
Before session a2 ◐ 30 min total
- Finish pricing the wrong answer for your top use case, and keep the sheet - a4 turns it into a ship scorecard.
- Run the risk-pricing prompt on a second use case and compare the two thresholds - notice how different "good enough" is for each.
- Use at least two of the "how would we know" questions in a real AI review this week.
- Optional: skim the builder track's b1 - you do not need the code, but seeing the live scorecard makes "measure it" concrete.
- For our top AI use case, what is the measured error rate, and against what set of questions?
- When we last changed a prompt or model, how did we confirm we did not make it worse?
- If quality dropped in production tomorrow, how long until we would know - and how?
- Who owns evaluation, and does it run on a schedule or only when someone remembers?
- What is our ship threshold for this use case - the number that says "good enough to trust"?
Official sources covered
The leader track teaches from published engineering guidance and course curricula - no vendor marketing. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A vendor's AI demo is flawless. The best reason this is weak evidence is...
Selection bias is built into any demo. Evaluation is the anti-demo: run many representative questions and count, rather than showing one good run.
2 · The three questions evaluation answers across a system's life are...
Offline eval gates the launch, regression eval guards every change, online eval watches production. Most teams do the first once and stop.
3 · Using a strong model as an automated judge (LLM-as-judge) is...
LLM judges reach ~85% agreement with humans and are how eval scales - but they favor longer answers, first-seen answers, and their own style, so they need calibration and human spot-checks.