The rung you cannot avoid and cannot trust yet
Session b1 put the LLM judge at rung 4 of the grading ladder, second from the bottom in trustworthiness, and told you to reach for it last. That advice stands. But "last" is not "never" - there are questions no rule and no compiler can settle, and every one of them lands on a judge. The question this session answers is not whether to use one. It is what has to be true before a judge's output is allowed into a launch gate.
Three questions nothing cheaper can answer 5 min live
Before defending the judge, be precise about what it is for. A judge is not a general-purpose grader you point at everything. It is the grader of record for a small, specific class of question - the ones where correctness is a property of meaning rather than of a string, a return code or a result set.
LiveThe test for "does this actually need a judge"3 min▶
Ask one question of every case in your golden set: is there any machine-checkable consequence of this answer being right? If yes, that consequence is your grader and the judge is a detour.
- "Is this SQL correct?" has a consequence - the rows it returns. Grade at rung 2, not rung 4.
- "Did it escalate?" has a consequence - a handoff event in the trace. Grade at rung 1.
- "Was the refund policy quoted accurately?" has no machine-checkable consequence. The passage says "14 days from delivery" and the answer says "two weeks after it arrives". Rung 4, and no lower.
In practice, running this test over a real golden set moves somewhere between a third and a half of the rows down the ladder, and it should be the first thing you do to any eval you inherit. What survives the test is the set the rest of this session is about.
What the judge prefers, for reasons unrelated to quality 7 min live
Zheng et al. measured judge behaviour while building MT-Bench and named four failure modes. The useful thing about that paper is not the list - lists of model weaknesses age badly. It is that the same paper proposed a mitigation for each one, and every mitigation is cheap, mechanical, and something you can wire in this afternoon.
Verbosity bias is the one that quietly reverses your roadmap. Session b3 makes conciseness an operational metric: shorter answers are faster to read, cheaper to generate and usually better UX. Now grade those answers with an uncalibrated judge. The trimmed answer loses to the padded one, the quality dashboard dips, and the team concludes the conciseness work made the bot worse. Nothing about the bot got worse. The grader had a preference and nobody had measured it. This is the specific mechanism by which an unmeasured judge does not merely add noise - it points your team in a direction.
Self-studyPosition swapping in practice, and what to do with the leftovers3 min read▶
The mitigation is mechanical: present the pair twice, once in each order, and keep only the verdicts that survive the swap. The implementation is about ten lines.
The interesting output is not the verdicts. It is the no_verdict pile. That pile is your position-bias rate, measured on your own data rather than quoted from a paper, and it is the single cheapest diagnostic in this session. A large pile means your rubric is not discriminating and no amount of averaging will fix it; those cases need a sharper scale, not more runs.
Do not silently drop them either. Report them: "n = 400, of which 37 had no consistent verdict" is an honest line in a launch document, and it is the kind of line that stops a room from over-reading a pass rate.
The step that turns an opinion into an instrument 6 min live
Every instrument in a lab has a known error against a reference. That is the entire difference between an instrument and a device that displays numbers. A judge becomes an instrument at exactly the moment you can state its agreement with a human-labelled slice of your own traffic - and not one minute before.
Three numbers recur across the practitioner literature and they are worth memorising as a set, because each one guards a different mistake:
| The number | What it is for | The mistake it prevents |
|---|---|---|
| 100+ traces | Read them by hand before you write any rubric. This is error analysis, not grading | Writing a rubric for the failures you imagined instead of the ones you have |
| 100+ labelled examples | The slice you validate the judge against, drawn from the same traffic | Quoting a judge's pass rate with no idea whether the judge is right |
| Not 100% pass | A suite everything passes is measuring nothing. Make it harder | Mistaking an easy eval for a good bot, which no gate downstream can catch |
LiveWhat the vendors converged on, independently4 min▶
The interesting part of the vendor landscape is not any single product. It is that three organisations designing judge tooling separately arrived at the same three ideas: a bounded scale, an explicit threshold, and a documented notion of how good the grader itself is.
| Pattern | Where it shows up | What to copy |
|---|---|---|
| Bounded scale, thresholded to pass/fail | Azure AI Foundry agent evaluators (intent resolution, task adherence, tool-call accuracy, task navigation efficiency) score on 1-5 with a default threshold of 3 | A graded scale for trending plus a binary for the gate. Do not let a room debate a 3.4 |
| A reason returned with the score | The same evaluators return the rationale alongside the number | Store the reason with every verdict. Step 4 of calibration is impossible without it |
| Autorater calibration against human raters | Vertex's rubric-based evaluation frames the grader as an autorater whose agreement with human raters is itself checked | The vocabulary. "Autorater" makes it obvious the grader is a component with an error rate |
| Published benchmark targets per eval | Arize Phoenix ships pre-tested eval templates with precision and F1 figures published for each one | Never adopt a template without asking what it scored on a labelled set. If nobody knows, you are back to step 1 |
Anthropic's own eval-design guidance lands in the same place from the other direction: use a different model as the grader, define every scale point explicitly, force predictable output so parsing is not a second source of error, and quantify even fuzzy criteria rather than leaving them to the model's taste.
The honest edge: Phoenix ships under Elastic License 2.0 rather than plain open source, and the Azure and Vertex patterns are documented product behaviour rather than peer-reviewed findings. Copy the shape of the design, not the marketing around it.
Write Beacon's groundedness rubric ★ 10 min · everyone builds
Beacon's Support Bot answers from the help centre. The one thing that must never happen is a fluent, confident, wrong version of the refund window. That is a groundedness failure, it cannot be executed, and it is what a judge is for. Write the rubric now, before you have seen a single score.
Read 20 real traces first, silently, before writing anything. You are looking for the shape of the failures, not counting them. Every rubric written without this step grades the failures its author imagined.
Pick one criterion for this rubric. Groundedness only. The instinct to score helpfulness, tone and grounding on one scale produces a number nobody can act on - if it drops, you cannot tell which of the three moved.
Write out every scale point in observable language, with money and timing called out explicitly, because those are the claims that cost Beacon something. A scale point you cannot apply to a trace in five seconds is not defined yet.
Set the pass threshold inside the rubric, not in the analysis afterwards. Then force the output format so the parsing step cannot introduce its own errors.
Choose a grader model that is not the model under test. Beacon's bot runs on one family, so the judge runs on another. This costs a procurement conversation and removes an entire bias.
Calibrate it, then fix what the disagreements tell you ★ 12 min · everyone builds
This is the loop from Part 3, run once, on paper or in code. It takes an afternoon the first time and twenty minutes every time after. The output is a single percentage that you are then allowed to put next to the word "calibrated" in a launch document.
Sample 100+ traces from the same traffic distribution the judge will grade in production. Not the demo set, not the hardest cases, not last week's incidents. The slice has to look like the job.
Label them by hand against your rubric, two people independently, then reconcile the disagreements out loud. Human-human disagreement is your ceiling: a judge cannot beat the consistency of the people who defined the target.
Run the judge on the same slice with the swap harness. Record three things per trace: the score, the returned reason, and whether the verdict survived the swap.
Compute agreement and, more importantly, read the disagreements as a group. They cluster. Sort them by answer length, by category and by whether the answer was a refusal, and the systematic skew becomes visible in about a minute.
Fix the rubric against the cluster you found - usually by adding two few-shot examples that sit exactly on the disputed line - then re-run and re-measure. If agreement did not move, the rubric was not the problem and your labels probably disagree with each other.
What a first calibration run usually looks like. Agreement comes back somewhere in the sixties, which feels like a disaster and is completely normal. The disagreements sort into two piles: long answers the judge passed and the humans failed (verbosity bias, straight out of the paper), and polite refusals the judge failed and the humans passed (the rubric never said what an abstention scores, so the judge invented a rule). Neither pile needs a better model. The first needs a few-shot example of a long unsupported answer scoring 2; the second needs one more line in the rubric. Second run lands in the low eighties, and that number - not the bot's pass rate - is the one that unlocks the quality gate in b9.
Try it yourself - this week ◐ 50-60 min total
- Take any judge you are already running in an eval and answer one question in writing: what is its agreement with a human on your data? If the honest answer is "we never measured", you have found this week's most valuable hour of work, and it is an hour, not a sprint.
- Run the swap harness on 50 existing judged cases and count the no-verdict pile. Report that count next to your pass rate in your next eval summary. Watch how differently the room reads the pass rate once it is standing beside a position-bias number.
- Audit your rubrics for the single-criterion rule. Any rubric scoring more than one thing on one scale gets split this week - a combined score cannot tell you which component moved, which makes it useless for the only job an eval has.
- Find the capability suite in your project that scores 100% and make it harder until it does not. Bring the three new failing cases to your next standup; they are worth more than the previous 100% ever was.
Official sources covered
Every framework and figure on this page traces to a primary source - peer-reviewed papers, vendor evaluation documentation and the practitioner canon catalogued in this course's source map. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your bot and your judge run on the same model family, and the judge reports a high pass rate. What is the specific problem?
Judges have a documented preference for outputs from their own model, so a same-family setup mixes an authorship effect into every score. It is unrecoverable after the run - there is no correction factor to apply, because you never measured how large the effect is on your data. The mitigation is structural and cheap: grade with a different model from the one under test, which is also what Anthropic's eval guidance recommends.
2 · Your judged capability suite has scored 100% for three releases running. How should you read that?
A capability suite exists to find headroom, so a perfect score is a statement about the suite rather than about the bot. It will keep reading 100% straight through a regression, because none of its cases were ever hard enough to fail. The regression suite is the one that is supposed to sit near 100%, since every case in it is a bug you already fixed - which is exactly why b1 insisted the two suites stay separate.
3 · You run every judged comparison in both answer orders. Thirty-seven of 400 cases give a different verdict depending on order. What do you do with them?
A verdict that flips when you swap the order was decided by position, not by the answers, so keeping it in the numerator manufactures signal from noise. But quietly dropping it is worse than useless - the size of that pile is a free, honest diagnostic measured on your own data, and reporting "400 cases, 37 with no consistent verdict" is the line that stops a room over-reading the headline. A large pile means the rubric is not discriminating, which is a rubric fix, not a sampling fix.