What this course is, and what it is not
This is not a course about eval metrics. Two of its siblings already do that job properly: Learn Evals goes deep on retrieval and generation metrics, and Model Evaluation covers the classic and LLM-era metric canon side by side. This course assumes those exist and asks the one question they leave open: given all these numbers, do you ship?
Four parts, and the failure that comes from skipping each 7 min live
An eval is not a script. It is four separable decisions, and teams usually make three of them well and one of them by accident. The one made by accident is the one that fails in production.
LiveWhere the cases come from: real failures beat imagination4 min▶
Anthropic's engineering guidance on agent evals is blunt about the starting point: begin with 20 to 50 simple tasks drawn from real failures, because early changes have large effect sizes and you do not need statistical power to see them. That advice is about iteration speed, not about launch evidence - and conflating the two is how a 20-case set ends up in a go/no-go deck. Both things are true at once:
- 20-50 cases is the right size to start improving a bot. You are looking for 30-point swings, and 30-point swings are visible at n=30.
- 20-50 cases is nowhere near enough to launch on. You are now claiming a 5-point difference from a bar, and that needs an order of magnitude more. Session a4 and b9 do this arithmetic properly.
Where the cases actually come from, in descending order of value: production logs of things that went wrong; support tickets and escalations; the questions your own team asks the bot to check it is alive; and last and least, questions you invented at your desk. Invented questions carry your assumptions about what users ask, which is exactly the bias the eval exists to catch.
Self-studyAnthropic's "volume over quality" principle, and its limit3 min read▶
Anthropic's published eval guidance recommends prioritising volume over quality: many automated lower-signal test cases beat a handful of hand-graded ones. This is correct and counter-intuitive, and it follows directly from the interval math - a noisy grader on 500 cases tells you more than a perfect grader on 20, because the sampling error dominates the grading error at small n.
The limit: it holds for the capability suite, not the safety canary. A 200-case injection suite scored roughly is fine. The eight cases covering "the bot must never issue a refund without approval" are graded exactly, by a person, every time, and must pass at 100%. Volume for the trend, precision for the tripwire.
Always grade at the cheapest rung that works 6 min live
There is a strict ordering of graders by cost, speed and trustworthiness, and it runs in the opposite direction to how most teams reach for them. Reach for the LLM judge last, not first - it is the most expensive, slowest and least reliable rung, and half the cases you would point it at can be settled by running the output instead.
The rung-4 habit costs real money. A common pattern in text-to-query evals is to ask a judge "is this SQL correct?" - a question that is slow, expensive, non-deterministic, and strictly worse than running both queries and comparing the rows. Execution accuracy became the field standard on the BIRD benchmark precisely because exact-string matching rejected valid alternative SQL while a judge was unreliable. The machine-checkable question is almost always available and almost always better; you just have to notice it is there.
Capability and regression are different animals 4 min live
One golden set cannot serve both purposes, and merging them produces a suite that is simultaneously too easy to drive improvement and too flaky to block a deploy. Split them on day one.
| Capability suite | Regression suite | |
|---|---|---|
| Question it answers | How good is this bot, and is it getting better? | Did we break something we already fixed? |
| Expected pass rate | Starts low. If it is at 100%, it is too easy to be informative | Near 100%, always. Anything else is a bug, not a score |
| Size | Large - hundreds, because you are estimating a rate | Small and growing - one case per past incident |
| Where cases come from | Sampled production traffic + an adversarial slice | Every bug you ever fixed, frozen forever |
| Grading | Mixed rungs; a noisy judge is acceptable at volume | Deterministic rungs wherever possible, so the gate is sharp |
| In CI | Reported, trended, not blocking | Blocking. A red regression suite stops the deploy |
| Runs | Nightly or per release candidate - it is expensive | Every pull request - it must be cheap enough to |
Beacon's golden set skeleton ★ 14 min · everyone builds
Not a toy exercise. Build the sheet you will actually use for the rest of this course, on a bot you actually own. If you do not have one yet, use Beacon's support bot - the category shares are given below.
Open a sheet with these seven columns: id · category · input · expected behaviour · grading rung · required capability · source. The last column records where the case came from - a ticket number, a log line, a person. Invented cases get marked "invented" and you will see how few of them you trust in a month.
Write your category list first, before any cases. For a support bot, Beacon uses five: straightforward help-centre lookup (30%), answer that must cite policy exactly (20%), multi-turn referring to earlier turns (20%), outside the knowledge base and must escalate (15%), adversarial or injected input (15%). Your percentages should come from a traffic sample, not from a guess.
Now fill fifteen rows - three per category. Take them from real logs if you have them. For each row, write the expected behaviour as an observable outcome ("escalates and does not state a policy") rather than a quality adjective ("handles it well"). If you cannot phrase it observably, you cannot grade it.
For every row, assign the grading rung from Part 2 - and push each one as far down the ladder as it will go. Watch how many rows you initially wrote as "judge" collapse to a rule once you phrase the expectation observably. That collapse is the point of the exercise.
Mark which rows would go in the blocking regression suite versus the capability suite. On a fresh bot this will be almost none - the regression suite is grown from incidents, not written up front. Leave the column; it fills itself in over the next six months.
The first look at the gate you are building toward ★ 8 min · explore
This is the scorecard b9 builds properly. Open it now, once, so the shape of the destination is in your head for the next eight sessions. Switch on Golden set sampled from real traffic and watch what happens to the confidence figure - that single toggle is Part 1 of the anatomy, and it is worth more than every other lever on the board.
Press Ship anyway right now, before touching anything. Read the incident list. Every item in it is a failure mode this course teaches you to catch before it happens rather than after.
Now switch on Golden set sampled from real traffic alone. The confidence figure falls hard. Nothing about the bot changed - you only stopped grading it on questions chosen because it answers them well.
Add Adversarial + unanswerable slice and Judge calibrated vs human labels. It falls further. You now have an honest number for the first time, and it is bad. This is the normal and correct experience of building a first real eval.
Try it yourself - this week ◐ 40-50 min total
- Grow your golden set from 15 rows to 40, and pull at least half of the new rows from production logs or support tickets rather than from your head. Keep the "source" column honest - you will want to know later which rows you invented.
- Go through your existing evals, if you have any, and count how many are graded by an LLM judge that could be graded by a rule or by execution instead. Move two of them down the ladder this week and note what it does to your eval runtime and cost.
- Find the last three production incidents your bot caused, and write one frozen regression case for each. That is your regression suite. It is allowed to have three rows; it is not allowed to have zero.
- Write down the threshold you would ship at, today, before you have measured anything. Seal it. In b9 you will find out whether you would have moved it after seeing the result - almost everybody does, which is exactly why part 3 of the anatomy says decide it first.
Official sources covered
Every framework and figure on this page traces to a primary source - vendor evaluation documentation, published benchmarks, and peer-reviewed papers catalogued in this course's source map. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Anthropic's guidance says to start with 20-50 tasks drawn from real failures. Why is that not the number you launch on?
The two activities have different statistical requirements. Early iteration produces huge effect sizes that are visible at tiny n, which is exactly why starting small is good advice. A launch decision compares your result to a threshold and needs the interval around that result to be narrow enough to clear it - a completely different problem, and the reason 19 of 20 proves only 76%.
2 · Your text-to-query bot's eval asks an LLM judge "is this SQL correct?" What should you do?
This is the rung-4 habit. SQL is runnable, so execution grading is available - and execution grading is free of judge bias, reproducible run to run, and settles the question the judge was only estimating. Execution accuracy displaced both exact-string matching and judging on the field's main text-to-SQL benchmarks for exactly this reason.
3 · Why must the blocking regression suite be graded deterministically rather than by a judge?
You cannot buy sharpness with volume in CI - a per-pull-request suite has to stay fast and cheap, which caps n. So you buy it by eliminating variance in the grader instead. Deterministic rungs mean a red suite is a real regression and never a judge having an off day, which is what makes the gate safe to block on.