One scorecard, six lights, one signature
A launch decision fails in a predictable way: somebody presents a single accuracy number, the room debates whether it feels high enough, and the loudest opinion wins. The fix is not a better number - it is six numbers that cannot be traded against each other, each with a threshold written down before anyone ran anything.
Six gates that do not trade against each other 7 min live
The central design rule: gates are conjunctive. All green means go. Any red means no-go, regardless of how green the others are. A bot that is brilliant and unaffordable does not ship; a bot that is fast, cheap and ungrounded does not ship either. The moment you let a strong gate compensate for a weak one, you are back to debating a single number.
LiveWhy the quality gate needs a risk tier attached3 min▶
There is no universal launch bar, and any course that gives you one is selling something. The bar comes from the blast radius of a wrong answer, which is what the tier encodes (session a5 builds the table):
- T1 - low (read-only, informational, reversible): bar around 80%. A wrong answer costs a user thirty seconds.
- T2 - medium (writes data, sends messages, makes commitments): bar around 85%, plus an adversarial slice and shadow mode before canary.
- T3 - high (financial or irreversible actions, regulated domains): bar around 90%, human approval on the action itself, kill switch drilled.
Beacon's support bot can issue a refund credit. That single capability moves it from T2 to T3, raises its bar by five points, and forces the approval gate. Notice what a cheap product decision it would be to remove that capability - and how rarely anyone weighs it that way. Scoping the action space down is the fastest way to pass a launch gate, and it is almost never on the table because it feels like giving up.
Self-studyReliability is not capability: pass^k3 min read▶
Everything on this scorecard measures a single run. Production is not a single run - the same user asks the same thing on Tuesday. tau-bench introduced pass^k, the probability that all k independent trials of a task succeed, as against pass@k, the probability that at least one does. They pull in opposite directions: pass@k rewards a model that occasionally gets there, pass^k rewards one that always does.
The arithmetic is unforgiving. A task the bot passes 75% of the time succeeds three times running only 42% of the time. On tau-bench's retail domain, GPT-4o-class agents scored under 50% at pass^1 and under 25% at pass^8 - the same agent, the same tasks, just asked to be consistent. If your bot handles an eight-step workflow, that is the number that matters, and it is not the one on your dashboard. Session b8 measures it properly.
The climb: six rungs from applause to signature ★ 10 min · watch, then argue
Step through these in order. Beacon's support bot is T3, so the bar is 90%. Read each caption before you look at the meter - the story is in what changed between rungs, and twice it is nothing about the bot at all.
Three bots, three different gates ★ 12 min · everyone builds
Same company, same discipline, three different answers. Work each board until all six gates are green, then compare what it took. The levers that mattered are different every time, and two levers are marked "not a gate for this bot" - read why.
Beacon Support Bot - the judged one
No ground truth. Correctness is an opinion until a human or a calibrated judge supplies one. Groundedness carries the weight here, and the escalation path is a gate, not a nicety.
Beacon Analyst Bot - the executable one
Every answer is runnable, so most of this eval belongs on the execution rung rather than the judge rung. Watch what output verification does here that it could not do for the chatbot - and notice that grounding is dimmed, because there is no passage to cite when correctness is settled by rows.
Beacon Insight Bot - the compound one
The hardest of the three, because it is the other two stacked: an executable inner stage and a judged outer narrative, with errors compounding between them. This is the only bot that needs all five quality levers before it clears the bar.
Take each board to all-green and write down the minimum lever set that got you there. They differ, and the differences are the argument for archetype-specific criteria rather than one company-wide eval template.
On any board, switch every quality lever on but leave latency + cost engineering off. Performance and cost go red together. That is the real trade every eval-focused team discovers late: quality levers are bought with milliseconds and cents, and something has to pay for them.
Now drop the analytics bot to n=100 with everything else on. It clears a T2 bar of 85% - and would miss a T3 bar of 90%. Same evidence, different tier, different answer. The tier is a product decision that lands as a statistics requirement.
Write the sign-off sentence ★ 5 min · everyone writes
A launch decision that cannot be written in one paragraph has not been made. Fill this in for your own bot, from your own scorecard, and notice every place you have to write "we did not measure that".
On [n] cases sampled from [which traffic, which period], including a [x]% adversarial and unanswerable slice, graded by [rungs used] with the judge calibrated against [m] human labels, [bot] passed [s]. True task success is therefore at least [CI lower bound] with 95% confidence, against a [tier] bar of [bar]. Latency is [p95] against a [SLO] SLO and cost is [cost] per [unit] against a [budget] budget. The safety canary passes at 100%, irreversible actions require human approval, the regression suite is wired to CI, and rollback was last rehearsed on [date]. Rollback triggers are [list]. I recommend launch at [x]% canary.
The last two blanks are the ones that get skipped, and they are the ones Google's own data flags. The ML Test Score rubric lists "models can be quickly and safely rolled back" as a production-readiness test and adds that rolling back is an emergency procedure operators should practice. In Google's internal survey across teams, no single test in that rubric was implemented by more than 80% of teams - and the scoring rule is deliberately harsh: your final score is the minimum across the four sections, not the average. Strong evals do not compensate for an unrehearsed rollback, in exactly the way a green quality gate does not compensate for a red ops gate here.
Try it yourself - this week ◐ 60 min total
- Fill in the sign-off paragraph for a bot you actually own, leaving genuine blanks where you have no evidence. The blanks are your backlog, in priority order, and they are more useful than any roadmap you would have written.
- Assign your bot a tier by writing down the worst thing it can do unattended. If that sentence involves money, a message a customer receives, or a decision about a person, you are at T3 and your bar just moved.
- Compute the Wilson interval on your last real eval run. If the lower bound is below your bar, you have two options and you should cost both: improve the bot, or label more examples. Teams reflexively pick the first and it is often the more expensive one.
- Rehearse a rollback. Not a discussion about rollback - an actual timed revert of the prompt or model version in a staging environment, with somebody holding a stopwatch. Write the number down. It belongs in the sign-off.
Official sources covered
Every figure on this page traces to a primary source catalogued in this course's source map. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Between ladder rung 5 and rung 6 the pass rate stays at 94.7% but the confidence figure climbs from 88.8% to 92.5%. What changed?
This is the rung teams skip because it improves nothing you can demo. The interval width shrinks roughly with the square root of n, so five times the cases buys you a little over half the width. The bot was already good enough at rung 5; it just was not yet provable against a 90% bar. Improving and proving are separate pieces of work with separate costs.
2 · Your bot scores brilliantly on quality and safety but is 60% over its cost budget. The quality gate is very green. Do you ship?
The entire value of a six-gate scorecard is that gates cannot be traded against each other. The moment a strong gate is allowed to compensate for a weak one, you are back to arguing about a single blended number and the loudest opinion wins. Cost being red has a real fix - route easy turns to a smaller model, cache, trim context - and that fix belongs before launch, not after finance sees the bill.
3 · Beacon's support bot moves from T2 to T3 purely because it can issue a refund credit. What is the cheapest way to pass its launch gate?
The tier is set by blast radius, and blast radius is a product decision, not a fact of nature. Removing one capability can drop the bar five points and remove the human-approval requirement entirely - far cheaper than the labelling and engineering needed to earn a T3 bar. It is almost never proposed, because cutting scope feels like failure while spending three weeks on evals feels like rigour. Option C is the genuinely dangerous answer: moving a bar after seeing the result is exactly what setting it in advance was for.