The grader problem

Anthropic's develop-tests guidance is blunt: "prioritize volume over quality" - a larger set of questions with automated grading beats a small set graded by hand. The reasoning is statistical: eval scores are noisy estimates, and 500 automated verdicts pin down a model's behavior better than 30 artisanal ones, even if each automated verdict is individually less trustworthy.

That doctrine only works if the automated grader is trustworthy ON AVERAGE - which is exactly why the human budget goes to calibration. The division of labor:

• Code graders run on every commit - thousands of checks, zero marginal cost.
• The LLM judge grades the open-ended bulk - hundreds of answers per run.
• Humans label a small fixed anchor set once, and the judge is measured against it. The demo below makes this concrete.

OpenAI's graders guide names the same families from the other direction: string check, text similarity (BLEU / ROUGE / METEOR / cosine and friends), score model (the judge), and python (arbitrary code). Two vendors, one shape: deterministic checks, model judgment, custom code, humans above them all.

Zheng et al. also measured the ceiling: GPT-4-class judges reach "over 80% agreement" with human preferences. Two readings of that number, one wrong and one right:

• Wrong reading: "the judge is 20% wrong - humans would do better."
• Right reading: 80% EQUALS the agreement rate between two humans. The judge has hit the ceiling of the thing it approximates - a judge cannot beat the humans it is calibrated against, because human preference IS the target, disagreements and all.

The habit this buys you: demand the agreement number with every judge score. "Faithfulness 0.92" alone is a model output. "Faithfulness 0.92, and this judge agrees with our human labels 85% of the time on the anchor set" is a measurement.

The loop from the demo, as the code you would actually run. No framework required - agreement is a mean of matches:

import json

anchors = json.load(open("anchor_set.json"))   # ~25 human-labeled answers

def agreement(judge_fn, anchors):
    verdicts = [judge_fn(a["question"], a["answer"]) for a in anchors]
    hits = [v == a["label"] for v, a in zip(verdicts, anchors)]
    misses = [a for a, h in zip(anchors, hits) if not h]
    return sum(hits) / len(hits), misses

score, misses = agreement(raw_judge, anchors)
print(f"agreement: {score:.0%}")               # 58% -> now READ the misses
for m in misses:
    print(len(m["answer"].split()), m["label"])  # every miss: long + bad

The last loop is the part teams skip: the misses are where the systematic bias lives. In the demo every miss was long-and-bad - once you SEE that, the fix names itself. The number tells you whether to trust the judge; the misses tell you how to improve it.

• Rubric-anchored scales. Never ask for "a score 1-5" bare - define what a 2 and a 4 MEAN, with a short example of each. An unanchored scale drifts with the judge's mood; an anchored one is a checklist in disguise.
• Pairwise over pointwise where possible. "Which of these two is better?" is a more reliable judgment than "score this one" - comparison is easier than absolute calibration, for models and humans alike.
• Temperature 0. A grader that gives different verdicts on identical input is measurement noise you chose to add. Determinism first.
• Version-pin the judge model. A judge upgrade silently moves every score on your dashboard - pin the exact version, treat changes as breaking.
• Re-calibrate when it changes. New judge model, new rubric, new task distribution: re-run the anchor set and re-measure agreement before trusting a single new number. The ~25 labels keep paying rent.

Where this connects: learn-ai-evals-with-phoebe owns the harness that wires judges into pipelines and CI; tonight you learned what must be true about the judge before any of that machinery is worth running.