Two things go wrong that a final-answer eval cannot see
A bot answers. An agent decides, calls, reads, decides again, and eventually answers. Every eval you have built so far grades the last box in that chain, which means two entire failure families are invisible to it: an agent that reached the right answer down a route you would never approve, and an agent that reaches the right answer only sometimes. Both ship happily through a green dashboard.
Trajectory matching, and its five settings 7 min live
A trajectory is the ordered list of tool calls an agent made. Grading it means comparing that list to a reference list a human wrote, and the entire question is how strictly. Google's Vertex agent evaluation names the modes precisely; treat those names as the vocabulary, because the semantics are what you are actually choosing between.
LivePrecision and recall are a pair of dials, not a score4 min▶
The binary modes give you a verdict. Trajectory precision and recall give you a diagnosis, and the diagnosis is usually more useful in the week before launch:
- Recall high, precision low. The agent does everything it should, plus a pile of things it should not. It is exploring. This costs latency and tokens, and every extra call is an extra chance to touch something it should not have touched.
- Precision high, recall low. Everything it did was legitimate, but it skipped steps. This is the dangerous one: the skipped step is often the verification call, the policy lookup, or the human approval.
- Both low. The agent is not doing the task. Stop tuning the eval and go look at ten traces.
Azure reports exactly this shape under Task Navigation Efficiency: the three match modes plus precision, recall and F1 over the tool sequence. Report the pair, not the F1, at least until launch - the F1 hides which of the two failure shapes you have, and they have completely different fixes.
Self-studyThe reference trajectory is a design artefact, and it will be wrong3 min read▶
Writing a reference trajectory forces a decision most teams have never made explicitly: what is the approved way to do this task? That is a policy question wearing an engineering costume. The first version of your reference set will encode how you happened to build the agent last month rather than what the business requires, and you will discover this the first time a genuinely better path fails exact match.
The correct reaction is not to loosen every case to any-order. It is to sort the reference calls into two kinds: required calls, which carry policy or safety weight and belong in an in-order match, and incidental calls, which are one valid implementation among several and should not be graded at all. Most reference trajectories shrink by half when you do this, and get sharper as they shrink.
Tool-call metrics, with the formulas they are actually computing 6 min live
Four libraries, five metric names, and only three underlying questions: were the right tools called, were they called with the right arguments, and did the task get done. Knowing which question a metric answers is the whole skill - the names do not tell you.
Task Completion is seductive and it is a judge. It needs no reference trajectory at all - hand it the task and the trace, and it returns an alignment score against a 0.5 default threshold. That convenience is exactly why it spreads through a codebase: it is the only agent metric you can add without doing the work of writing references. But referenceless means there is nothing to disagree with the model, so it inherits every bias from b4 and reports them as task success. Use it as a trend alongside deterministic tool metrics, never as the blocking gate, and calibrate it against human labels before you quote its number to anyone.
pass@k and pass^k pull in opposite directions 7 min live
These two look like variations on one idea and they are opposites. One asks whether the agent can. The other asks whether it does, every time. Research optimises the first because it measures headroom. Production runs on the second, because production is not a single run.
LiveWhy tau-bench grades the database and not the sentence3 min▶
tau-bench made one design choice that is worth stealing outright: a task succeeds when the final database state matches the goal state. No judge, no reference answer, no string comparison. Just: did the world end up the way it was supposed to?
That is the execution rung from b1, applied to agents. It is deterministic, it is free of judge bias, and it is immune to an agent that describes doing the right thing while doing the wrong thing - a failure mode that is otherwise very hard to catch, because the narration is generated by the same model that took the action. If your agent writes to anything at all, you have a state to compare, and you should be grading it there rather than asking a judge whether the response sounded correct.
Self-studyThe agent benchmark landscape, in one paragraph each4 min read▶
WebArena puts agents in realistic self-hosted web environments. The best GPT-4 agent scored 14.41% against a human rate of 78.24%. That gap is the honest starting point for anybody about to promise a browsing agent.
GAIA asks questions that are easy for people and hard for assistants: humans reach 92%, GPT-4 with plugins reached 15%. It is the clearest available demonstration that "hard for a model" and "hard for a person" are unrelated axes.
SWE-bench contributes the validation pattern most worth copying. Each task carries FAIL_TO_PASS tests, which must go from failing to passing, and PASS_TO_PASS tests, which must stay passing. That second set is the regression suite in miniature, and it is what stops an agent from "fixing" the bug by breaking the rest of the repository. If your agent produces anything verifiable, build both halves.
BFCL checks tool calls deterministically by parsing them into an abstract syntax tree and comparing structure rather than strings, which is why it can grade function calls without a judge. It is now at V4 with holistic agentic evaluation. Its lesson for you: a tool call is structured data, so grade it as structured data.
Write Beacon's trajectory spec ★ 11 min · everyone builds
Beacon's support agent can issue refund credits, which is what made it T3 in b9. That capability is a sequence of tool calls, and this is where you decide which parts of that sequence are policy and which are implementation detail.
Take five real traces from your agent - actual traces, not imagined ones - and write down the tool calls each one made, in order. You now have five candidate reference trajectories and, almost certainly, five different paths for tasks you thought were the same.
For each call in each trace, mark it required or incidental. Required means policy, safety or correctness depends on it: the approval call, the policy lookup, the permission check. Incidental means it is one valid way among several. Delete every incidental call from the reference - grading it only punishes better implementations later.
Now choose the match mode per case, not per suite. Cases where a required call must happen before another get in-order match. Cases where two lookups are independent get any-order match. Reserve exact match for the handful of cases where an extra call is itself a violation.
Write one row of the spec in a format you can execute. Keep the required calls, the mode and a comment saying why that mode - the comment is what stops somebody loosening it in six months without knowing what they gave up.
Compute pass^k on a task you already ship ★ 9 min · everyone runs
This takes fifteen lines and one afternoon of compute, and it is the single most uncomfortable number you can put in front of a launch meeting. Do it before somebody else does.
Pick one task your agent already handles. Run it n = 10 times from an identical starting state, with whatever temperature and tooling production uses. Do not lower the temperature for the experiment - you are measuring production, not a laboratory.
Score each run at the execution rung if you can: did the final state match the goal state? Count the successes as c. Resist the urge to grade partial credit; pass^k is a consistency measure and partial credit dissolves it.
Plug n and c into the estimator and read pass^k for k = 1, 3 and 5. Then look at how many steps your real workflow chains together and read the number for that k. That is the number your users meet.
Repeat for your five highest-traffic tasks and put the pass^3 column next to the pass@1 column in one table. The gap between those two columns is the gap between your demo and your support queue.
Try it yourself - this week ◐ 50-60 min total
- Grow the trajectory spec from four rows to fifteen, and mark every required call that carries policy weight. Count how many of those required calls your current eval would notice if the agent silently stopped making them. For most teams the honest answer is zero.
- Run the pass^k measurement on your three highest-traffic tasks and put pass@1 and pass^3 side by side in one table. Take that table to whoever owns the roadmap. It is a more persuasive artefact than any eval report you will write this quarter.
- Find one step in your longest agent workflow that could be deterministic code instead of a model call, and move it. Re-run pass^3. The improvement is usually larger than a model upgrade would have bought you, and it is permanent.
- Copy the SWE-bench pattern: for one verifiable task, write the tests that must go from failing to passing and the tests that must stay passing. The second list is the part everyone forgets, and it is the part that catches an agent that fixes one thing by breaking three.
Official sources covered
Every framework and figure on this page traces to a primary source - vendor evaluation documentation, published benchmarks, and peer-reviewed papers catalogued in this course's source map. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your agent's trajectory scores high recall and low precision against the reference. What is it doing?
Recall is overlap over the reference calls, so a high recall means nothing required was skipped. Precision is overlap over the predicted calls, so a low precision means the agent added calls of its own. That is the flailing shape: correct but wasteful, and every unnecessary call is another chance to touch something it should not have touched. The opposite pattern - high precision, low recall - is the more dangerous one, because the skipped call is usually the verification or the approval.
2 · A task your agent passes 75% of the time is used three times in one workflow. What is the end-to-end success rate?
This is the pass^k arithmetic and it is the reason capability numbers flatter multi-step agents so badly. Three green 75% steps in a design review look acceptable; the workflow they compose fails more often than it succeeds. tau-bench showed the same shape on real agents: GPT-4o-class systems under 50% at pass^1 fall under 25% at pass^8 on retail, with no change to the agent or the tasks. The fixes are architectural - shorten the chain, make steps deterministic, verify and retry - not prompt-level.
3 · Why is DeepEval's Task Completion metric a poor choice for a blocking gate, despite being the easiest agent metric to adopt?
Referencelessness is exactly what makes it convenient and exactly what makes it untrustworthy as a gate. It needs the full trace and the task, and it returns a judged score, so it sits on rung 4 of the grading ladder with all the position, verbosity and self-enhancement biases that implies. Run it as a trend beside deterministic tool metrics, calibrate it against human labels before quoting it, and block on the rules instead.