Agent evals

Every grader in existence is one of three kinds. The essay's advice is not to pick one - it is to combine them and know each one's failure mode:

• Code-based graders - assert the answer contains "150", check a row landed in the database. Fast, objective, cheap to run thousands of times - and brittle: a correct answer phrased as "$150.00" can fail a naive string check, and a lucky wrong answer can pass one.
• LLM-judge graders - a model grades the transcript against a written rubric. Handles open-ended answers code cannot - but judges need calibration against human labels before you trust them, and every rubric needs an "Unknown" escape clause so the judge can abstain instead of guessing when the transcript is ambiguous.
• Human graders - the gold standard. Too slow and expensive to run on every trial, so their real job is to calibrate the other two: label a sample by hand, then check your code graders and LLM-judges agree with the humans.

Agents are non-deterministic twice over: the model samples its tokens, and the environment (tool latency, data state, ordering) shifts under it. That is why the unit of evidence is the trial, and why every score in this course is reported over N trials, never one. The gap between pass@k and pass^k on the same task is a direct reading of that variance - in the homework you will measure it on your own agent and treat it as a debt to pay down, not a dice roll to accept.

The agentevals package (LangSmith's trajectory-evals toolkit) ships four match modes plus an LLM judge, each a different strictness dial against a reference trajectory:

• strict - same tool calls, same order. The most brittle; reach for it only when order is the requirement (approve before send).
• unordered - same tool calls, any order. Right for "it must have checked both months" tasks.
• subset - the agent called no tools outside the reference set. Catches rogue extra calls.
• superset - the agent called at least the reference tools. Catches skipped required steps.
• create_trajectory_llm_as_judge - a rubric-driven judge over the whole trajectory, for the qualities no matcher can express ("did it re-check before acting?").

Arize's course teaches the same idea as decomposition: evaluate the router (did it pick the right tool?), each skill (did the tool step succeed?), and the trajectory separately - then improve the LLM-judge itself against human labels.

From the Anthropic essay's field notes - check your harness against all five before trusting any number it prints:

• Eval saturation - your suite hits 100% and stops teaching you anything. Add harder tasks from fresh failures; a saturated eval is a retired eval.
• Brittle grading - the string check that fails "$150.00", the judge that punishes formatting. Calibrate graders against transcripts you have read yourself.
• Shared state between trials - trial 3 passes because trial 2 already wrote the row. Reset the environment between trials or your pass^k is fiction.
• Ambiguous task specs - if two reasonable humans disagree on what "done" means, your grader is guessing. Tighten the success condition, not the agent.
• Positive-only test sets - every task has a happy path. Add tasks the agent SHOULD refuse (b7's injection trap belongs in the golden set too).

Everything from Parts 1-2 in one file. run_agent is your b2 loop unchanged - the harness wraps it, never reaches inside it (grade outcomes, remember):

import anthropic

client = anthropic.Anthropic()
K = 5  # trials per task - one trial proves nothing

def run_agent(prompt: str) -> str:
    """Your b2 loop: model + Daybreak tools, returns the final answer."""
    ...  # unchanged from b2 - the harness only sees what comes out

# --- graders ------------------------------------------------------
def grade_march(answer: str) -> bool:      # code-based: fast, brittle
    return "150" in answer                 # March 2026 revenue is $150

RUBRIC = (
    "PASS if the answer names BOTH the March refund and the cancelled "
    "subscription as causes of the March dip. FAIL if it invents causes. "
    "UNKNOWN if the answer is too ambiguous to grade - never guess."
)

def grade_dip(answer: str) -> bool:        # LLM-judge: rubric + escape
    verdict = client.messages.create(
        model="claude-opus-5", max_tokens=5,
        messages=[{"role": "user",
                   "content": RUBRIC + "\n\nAnswer to grade:\n" + answer}],
    ).content[0].text.strip()
    return verdict == "PASS"   # UNKNOWNs get queued for human review

# --- golden set ---------------------------------------------------
TASKS = [
    {"prompt": "What was March 2026 revenue?",  "grader": grade_march},
    {"prompt": "Why did revenue dip in March?", "grader": grade_dip},
    # ... grow toward the full 12, then past it with real failures
]

for task in TASKS:
    wins = [task["grader"](run_agent(task["prompt"])) for _ in range(K)]
    print(task["prompt"], "| pass@5:", any(wins), "| pass^5:", all(wins))

any(wins) is pass@k, all(wins) is pass^k - the whole metric split is two builtins. The work is in the graders, which is exactly where the essay says the work should be.