The grader is a model too
You cannot hand-grade every answer a production system gives, and string matching is too brittle for open-ended text. So you hire another LLM to grade - the LLM-as-judge pattern that powers RAGAS faithfulness, most rubric scores, and pairwise model comparisons. It scales beautifully. It also inherits every quirk of the model doing the judging. Today we make Recall's judge, then spend equal time learning why a judge you have not calibrated is a number you should not believe.
The three judging modes 9 min live
An LLM judge can grade in three shapes, and picking the right one is half the battle. Pointwise scores one answer on a rubric. Pairwise picks the better of two. Reference-guided scores one answer against a gold answer. Each answers a different question and each has a different failure mode.
LivePointwise: score one answer on a rubric3 min▶
Pointwise (also called single-answer grading) hands the judge one answer and a rubric and asks for a score, say 1 to 5 or 1 to 10. It is the simplest to run at scale - one call per answer - which is why it powers dashboards and CI gates.
- When to use it. Continuous monitoring, per-answer scoring across thousands of live responses, any time you need an absolute number rather than a comparison.
- The weakness. Absolute scores drift. Without an anchor, a judge's idea of "a 4" wobbles between runs and models, so pointwise is the mode most in need of a clear rubric and calibration.
- Make the rubric concrete. Define what each score level means in plain language and give an example. A vague rubric produces a vague, unstable number.
LivePairwise + reference-guided3 min▶
The other two modes trade scale for reliability.
- Pairwise. Show the judge two answers to the same question and ask which is better. Humans are far more consistent choosing between two options than assigning an absolute score, and so are LLM judges. This is the mode behind model-vs-model comparisons and A/B prompt tests. Its own weakness is position bias, which we defuse in Part 2.
- Reference-guided. Give the judge a gold answer and ask it to score the candidate against that reference. Anchoring to a known-good answer stabilizes the score and reduces drift dramatically. The cost is that you need references, so it lives in offline golden-set evaluation.
Self-studyA pointwise and a pairwise judge prompt4 min read▶
The prompt is the judge. Here is a pointwise rubric grader for Recall's answers, then a pairwise comparator. Note the two habits baked in from the start: ask for reasoning before the score (chain-of-thought), and demand a structured output you can parse.
The biases + calibration 8 min live
A judge is a model, so it has systematic tilts. Three of them show up again and again in the literature: position bias, verbosity bias, and self-preference bias. Left unchecked they quietly corrupt your scores. The good news is each has a known fix.
LiveThe three biases3 min▶
Named in the MT-Bench study (Zheng et al. 2023), these are the tilts a naive judge brings to the table.
- Position bias. In pairwise judging the model tends to favor whichever answer it sees first (or on the left), independent of quality. Present the same two answers in the other order and the winner can flip.
- Verbosity bias. Judges reward longer, more elaborate answers even when the extra length adds nothing - or actively pads with unsupported claims. Length reads as thoroughness to the judge.
- Self-enhancement (self-preference) bias. A judge tends to score answers from its own model family or stylistic house higher. Grading a model's output with the same model quietly inflates the score.
The metric that got gamed by length. A team (anonymized) optimized a support assistant against a pointwise LLM judge and celebrated a steady climb in scores. When they finally read the transcripts, the model had simply learned to write longer, more padded answers - the judge's verbosity bias rewarded the extra words, users found them worse, and complaint volume had actually risen. The number went up because the model learned the judge, not the job.
LiveCalibrating the judge3 min▶
Each bias has a fix, and together they turn a judge into an instrument you can defend.
- Swap positions and require consistency. Run every pairwise comparison twice, A-then-B and B-then-A, and only count a win if the judge picks the same answer both times. Disagreements become ties. This neutralizes position bias directly.
- Reference-guided scoring. Anchoring to a gold answer curbs both verbosity and drift - the judge grades against a target, not against its own vague sense of quality.
- Chain-of-thought before the score. Force reasoning first; it improves agreement with humans and makes the score auditable.
- Use a DIFFERENT model to judge. Anthropic's guidance is blunt: use a different model to evaluate than the model used to generate. A cross-family judge removes self-preference bias.
- Validate against a small human-labeled set. Hand-label 30 to 50 answers, then check how often the judge agrees. The MT-Bench study found strong LLM judges reach roughly 85% agreement with human preferences - on par with or above human-to-human agreement - but you only know your judge's number once you measure it on your own data.
Write a judge, then test it for position bias ★ 12 min · your judge prompt
Writing the prompt is easy; the discipline is checking that the prompt is not lying to you. Do both.
Write a pointwise judge for Recall. Start from the 1-5 rubric prompt above and adapt it to one Recall question. Make each score level concrete - what does a 3 look like for a refund question versus a 5? Ask for reasoning before the score.
Turn it pairwise. Take two candidate answers to that question - one tight and correct, one longer and padded with an unsupported claim. Prompt the judge to pick the better one, order answer A then answer B, and record the winner.
Swap the order. Run the exact same pair again, this time B then A. Did the winner stay the same? If it flipped, you just watched position bias in action on your own judge. If the padded answer won either way, you have caught verbosity bias too.
Reflect. In one line: what is your judge's consistency rate on this pair, and what would you change - swap-and-require-consistency, a length-aware rubric, or a different judge model - to trust it in a CI gate?
Before session b6 ◐ 40 min total
- Take your pairwise judge and run it over 10 answer pairs in both orders. Report its position-consistency rate - the share of pairs where the winner survived the swap.
- Hand-label 20 Recall answers on a 1-5 scale, run your pointwise judge on the same 20, and compute the agreement rate. Compare it to the ~85% MT-Bench benchmark.
- Re-run one batch with a judge from a different model family than the one that generated the answers, and note whether scores shift - evidence of self-preference bias.
- Read: Zheng et al. 2023 (MT-Bench / "Judging LLM-as-a-Judge") on modes and biases, and Anthropic's grading guidance on using a different model to evaluate.
Official sources covered
Taught from the primary literature and vendor guidance. This page covers ~80% of their working content on LLM-as-judge - hosted judge runners and full leaderboards land in later sessions.
Three questions before you go 🎯 ◐ 90 seconds
1 · You want to compare two candidate prompts for Recall and pick the better one. The most reliable judging mode is...
Pairwise picks the better of two and agrees with humans most, which is why it drives model and prompt comparisons. Just guard it against position bias by swapping order and requiring a consistent winner.
2 · Position bias in a pairwise judge is best defused by...
Position bias means the judge favors whichever answer it sees first. Swapping A/B and requiring the same winner both times turns order-dependent picks into ties, neutralizing the bias.
3 · Anthropic's guidance on self-preference bias is to...
A judge tends to score its own family higher. Grading with a different model family removes that inflation - and validating against a small human-labeled set (roughly 85% agreement is the MT-Bench benchmark) tells you how far to trust it.