Why this session exists
Back in session 1 you saw Anthropic's funnel: success criteria, then evals, then - and only then - prompt engineering. We deliberately broke that order so you could build intuition first. Now we pay the debt. Sessions 8 and 9 turn "the v4 prompt feels better" into "v4 scores 91% category accuracy against 20 real tickets, v2 scores 78%". One of those sentences survives a meeting with your boss.
Why vibes fail 5 min live
Every prompt you have shipped so far was graded the same way: you read a few outputs and nodded. That works for 5 test tickets. It does not survive contact with 1,000 real ones - and Meridian's queue does 1,000 a week.
LiveLooks-good-to-me is not a metric3 min▶
Three reasons eyeballing collapses at scale, all of which have already bitten the Meridian team:
- You sample the easy cases. The tickets you test with are the ones you remember - clean, typical, polite. Production sends you the drunk 2am ticket in three languages with an order number in the subject line.
- Your memory is not a baseline. When you "improved" the prompt in session 6, did shipping-delay accuracy go up or just refund handling? You genuinely cannot know without numbers from before the change.
- Prompt changes are code changes. They alter production behaviour, so they need what code changes need: regression tests. A one-word edit to a prompt can silently break a category that used to work - and nobody reads 1,000 outputs to check.
The Meridian incident that started this. A well-meaning edit in session 7's vendor-variant work reworded one line of the Gemini variant. Refund tickets started coming back tagged SHIPPING. Nobody noticed for four days, because everyone checked the same three sample tickets - none of them refunds.
Self-studyThe whole industry converged on this2 min read▶
This is not one vendor's opinion. Recall the session 1 funnel: Anthropic's prompt engineering overview lists success criteria and empirical evals as prerequisites - things you do before prompt engineering, not after. Anthropic's own API course on Skilljar goes further and puts the evals module before the prompt engineering module, a deliberate ordering choice. OpenAI's reasoning guidance keeps hammering explicit success criteria. The message from everyone who ships this stuff: a tips-and-tricks practice without evals is a 2023 practice.
The honest framing for your team: evals are not the boring homework after the fun prompting. Evals are the instrument panel. Prompting without them is flying by feel in clouds.
The grading taxonomy 7 min live
Anthropic's eval docs give six ways to grade an output, and they sit on a ladder: cheap and rigid at the bottom, flexible and costly at the top. The craft is picking the lowest rung that can grade your criterion.
LiveSix methods, one table4 min▶
| Method | Graded by | Grades what | TriageBot use |
|---|---|---|---|
| Exact match | Code | Is the output exactly X? | Category label equals REFUND |
| Cosine similarity | Code | Is it semantically close to a reference? | Summary close to a gold summary |
| ROUGE-L | Code | Summary overlap with a reference | Same, word-overlap flavour |
| LLM Likert | Model | 1-5 rating on a quality dimension | Tone appropriateness, 1-5 |
| LLM binary | Model | Yes or no against a criterion | "Is this summary actionable?" |
| LLM ordinal | Model | Graded scale with a rubric | Escalation-note quality, 1-3 |
Rule of thumb from the docs and from everyone's scars: code-graded beats model-graded whenever the criterion allows it. Code grading is free, instant, and deterministic - the same output always gets the same grade. A model judge costs money per run and can change its mind. Save it for what code truly cannot see: tone, helpfulness, reasoning quality. That is session 9's whole first act.
Self-studyThe volume principle - Anthropic's counterintuitive advice2 min read▶
The single most useful line in Anthropic's eval docs: prioritise volume over grading quality. Many test cases with slightly crude grading beat a handful of cases with exquisite hand-grading.
- A 100-case eval with blunt exact-match assertions will catch a regression that a 5-case artisanal eval sails past - because the regression lives in case 47.
- Crude grades average out. One mis-graded case in 100 shifts your score 1%. One missing case in 5 hides 20% of reality.
- Volume is also cheaper to grow: every production failure becomes one more test case (session 9 makes this a formal loop).
So when you feel the pull to perfect ten test cases, resist it and write thirty rough ones instead. You can always upgrade the grading later. You cannot upgrade coverage you never wrote.
Your first eval harness with promptfoo 8 min live
You could write an eval loop in 40 lines of Python - Anthropic's course literally does. But the field standardised on a tool: promptfoo, an open-source eval CLI with 20k+ GitHub stars, used inside Anthropic's own prompt evaluations course. One YAML file in, a results grid out. Today is conceptual; no install needed in session.
LiveThe config, annotated4 min▶
A promptfoo config is one YAML file with three lists. Here is a trimmed TriageBot version - read it top to bottom and notice it is just the SVG above in text form:
The four assertion types you will use constantly: contains (substring present), equals (exact match), is-json (parses cleanly - your session 3 schema work finally gets policed), and llm-rubric (a model judges against a written criterion - we hold this one until session 9, where it gets the care it needs). Anything custom, like the word count above, drops into a javascript assertion.
Self-studyRunning it, and reading the grid2 min read▶
When you do install it at your desk (one npm command, or use the hosted runner), the loop is: promptfoo eval runs the matrix, promptfoo view opens the grid in a browser. Each row is a test case, each column a prompt-provider pair, each cell green or red with the raw output on click.
- Read columns to compare prompts: v4's column should be greener than v2's, and the score at the top says by how much.
- Read rows to find hard cases: a ticket that fails in every column is either genuinely hard or wrongly labelled in your golden set. Both are worth knowing.
- Add a second provider line and the same tests grade a vendor swap for free - your session 7 multi-vendor variants suddenly have a scoreboard.
TriageBot gets its eval ★ 22 min · build it live
No new prompt today - we build the thing that judges the prompts. By the end, v2 versus v4 is a number, not an argument.
Assemble the golden set: 20 real tickets with expected categories. This is the hardest step and it is a team sport, not a solo one. Pull 20 real Meridian tickets, spread across categories, and agree the correct label for each - out loud, as a group. The arguments you have here ("is a late refund a REFUND or a SHIPPING?") are the product decisions your prompt was silently making alone. Write the agreed answers down; that document outlives every prompt version.
Write the code-graded assertions. Four checks per ticket, all cheap: category exact-match against the golden label; output is-json (the session 3 schema, enforced at last); summary under 30 words via a javascript word count; and no invented order numbers - this one is beautifully checkable: any order id appearing in the output must also appear in the input ticket. Hallucination detection in one line of code.
Sketch the promptfoo YAML. Take the config from Part 3 and extend it: two prompt files (v2 and v4), one provider, 20 test blocks. Keep descriptions human ("angry refund, order id in subject") - future you reads the grid by them.
Run v2 versus v4 - mentally, together. Before any real run, predict: which tickets does v2 fail? The team's guesses tend to cluster on the vague tickets and the multi-issue tickets - exactly where v4's grounded policy manual and lookup tool should win. Writing predictions down first is what makes the real numbers educational instead of just decorative.
Log the baseline. Whatever the scores are, record them: date, prompt version, model, score per assertion type. This row of numbers is the whole point of today - session 9 wires a gate to it, so no future "improvement" can quietly score below it again.
Before session 9 ◐ 30 min total
- Grow the golden set from 20 to 50 tickets. Volume over grading quality, remember - rough labels on 50 beat perfect labels on 20. Recruit a support colleague; they label faster and better than you do.
- Add 5 deliberately nasty edge cases: a ticket in another language, one with two order numbers, one that is pure profanity, one that is 3,000 words, and one that politely asks TriageBot to change its own rules. (Keep that last one - it becomes very important in session 10.)
- Run the config for real if you can - promptfoo is a free install and your 20-case YAML is ready. Bring the grid screenshot; we compare baselines at the top of session 9.
What this session covers
Builder-level coverage of the sources below. Official notebooks and course videos stay with the vendors; links live in the course map.
Three questions before you go 🎯 ◐ 90 seconds
1 · Anthropic's volume principle for evals says...
Coverage catches regressions; grading precision only refines them. One crude grade wrong shifts your score a little - one missing case can hide a whole failure mode. Write thirty rough cases, not ten perfect ones.
2 · When should you reach for a model-graded check instead of a code-graded one?
Code-graded checks are free, instant, and deterministic - the same output always gets the same grade. Climb the ladder to a model judge only when the criterion forces you. Category labels and JSON validity never do.
3 · The hardest and most valuable part of building TriageBot's eval was...
The config is twenty minutes of typing. The golden set forces the team to agree what correct even means, ticket by ticket - and it appreciates in value with every prompt, model, and vendor change you ever make.