The first of three archetype gates
Sessions b5, b6 and b7 gate the same company's three bots, one at a time, with the same discipline and three completely different answers. This is the first, and it is the hardest to measure, because it is the only one of the three where nothing can be run. The Analyst Bot's SQL executes. The Insight Bot's chart renders. A support answer just sits there, fluent and confident, being either true or not.
Judged quality answers a different question from outcome 5 min live
The mistake that makes chatbot evals useless is picking one family. Quality-only teams ship a bot that is grounded, polite and does not actually get anyone's problem solved. Outcome-only teams ship a bot with a beautiful containment rate that contains people inside a conversation they eventually abandon. You need both, and they fail in opposite directions.
LiveEscalation correctness is a gate, not a courtesy3 min▶
Most teams treat escalation as the sad path and never grade it. For a T3 bot it is the safety valve, and a safety valve is graded like a safety valve - both directions, separately, because the two failures cost completely different things.
- Missed escalation - the bot should have handed off and did not. It invented a policy, or attempted an action outside its remit. This is the expensive direction, and it is the failure the unanswerable slice of your golden set exists to catch.
- Over-escalation - the bot handed off something it could have handled. Cheap per incident, but it is the direction a nervous team over-corrects into, and it destroys containment quietly while every quality metric stays green.
- Handoff quality - did the human inherit a usable summary and the conversation so far, or start cold while the customer repeats themselves? This one is gradeable at rung 1: either a summary was attached or it was not.
Grade all three. A single "escalation rate" number averages the expensive failure with the cheap one and hides both.
Containment is not resolution, and resolution is whatever you wrote down 7 min live
These two words get used interchangeably in vendor decks and they measure different things. Containment asks whether a human was involved. Resolution asks whether the customer's problem went away. A bot can contain a conversation perfectly and resolve nothing, and the gap between those two numbers is where every uncomfortable support conversation lives.
The published numbers, with their source status attached. Nothing here is a target for your bot - they are anchors for what a normal range looks like, and every one of them depends on a definition you did not write.
| Figure | What it refers to | Source status |
|---|---|---|
| 20-40% | Containment at the start of a deployment | Gartner-attributed · SECONDARY |
| 70-90% | Containment for a mature deployment | Gartner-attributed · SECONDARY |
| 40-65% | Containment for a well-configured RAG chatbot | Gartner-attributed · SECONDARY |
| 38-72% | Intercom Fin's reported production resolution range | Vendor-reported · SECONDARY |
| ~67% | Fin's reported average across 7,000+ customers and 40M+ conversations | Vendor-reported · SECONDARY |
| $0.99 | Price per resolution under that definition | Vendor documentation |
Self-studyWhy resolution is the stricter number, and what that costs you3 min read▶
Containment is easy to instrument: did a human agent touch this session? One boolean, available from your routing logs today. Resolution needs a judgement about a customer's state of mind, which is why the strict definitions reach for proxies - an explicit confirmation, or an exit followed by silence.
The consequence is that containment is always the larger number, and a team under pressure will drift toward reporting it. Watch for the drift. A rising containment rate with a flat resolution rate means the bot is getting better at keeping people in the channel and no better at helping them - and the first place that shows up is not your dashboard, it is your repeat-contact rate.
The reversal clause is the honest part of the strict definition and the part you should copy first. A resolution that gets taken back when the customer returns is the only version of the metric that cannot be gamed by a confident wrong answer, because a confident wrong answer produces exactly that return.
What a single-turn eval never sees 6 min live
Almost every golden set is a list of questions with a list of expected answers. Almost every real support session is six turns long. The gap between those two facts is the largest measured blind spot in chatbot evaluation, and the numbers are worse than most teams guess.
This is why the Beacon golden set puts 20% of its weight on multi-turn cases. Session b1 set that share deliberately: multi-turn questions that refer back to earlier turns are the second-largest category in the set, ahead of the unanswerable and adversarial slices. It looked like an odd allocation at the time. It is the direct consequence of a measured average 39% drop between single-turn and multi-turn performance - if a fifth of your traffic is conversational and your suite has none of it, your headline number is describing a bot nobody uses.
LiveGrading the session, not the turn3 min▶
Once a case is a conversation rather than a question, the unit of grading has to change with it. Three rules that make multi-turn cases workable:
- The case passes only if the whole session passes. A six-turn session where turn 4 contradicts turn 2 is a failed case, even though five turns were individually fine. Averaging turn scores is how the collapse stays invisible.
- Script the user side. Fixed customer turns, so the only thing varying between runs is the bot. A conversation driven by an improvising simulated user gives you a different case every run, which is a grader problem wearing a case's clothes.
- Include one instruction that must survive. A currency, a timezone, an account, a constraint stated in turn 2 that turn 6 depends on. That single planted fact turns a pleasant chat into a test.
These sessions are expensive to run - six model calls per case rather than one - which is a real argument for keeping the multi-turn slice proportionate rather than dominant, and for grading it nightly instead of per pull request.
Self-studySatisfaction proxies, safety and tone, and the latency number4 min read▶
The satisfaction proxy. Human CSAT surveys have two problems for a launch gate: only a small and self-selecting minority of customers ever answer one, and the answers arrive after the launch decision. The working pattern is an AI-estimated satisfaction score computed over all traffic, validated against the survey responses you do have. Treat it exactly like the judge in b4 - it is a rung-4 grader, and it needs the same calibration against a human-labelled slice before it is allowed near a gate.
Safety and tone red-teaming. The adversarial slice in your golden set is a fixed list, and a fixed list is a floor rather than a test. Automated red-teaming is what raises it: Microsoft's PyRIT generates and auto-scores thousands of adversarial prompts, taking the work "in hours instead of weeks". For a T3 customer-facing bot the target is not a score, it is coverage - injection attempts hidden in pasted ticket text, attempts to talk the bot into a refund, and tone failures on customers who are already angry. Every genuine finding becomes a permanent regression case, per b1's split.
Latency. Chat is the one archetype where the p95 latency number is a product requirement rather than an efficiency concern - a customer watching a blank box for two seconds has already formed an opinion. The common interactive target is p95 time-to-first-token under 500ms, with 200-500ms considered well-tuned; a looser starting SLO used by teams still tuning is p95 TTFT under 1s with p95 inter-token latency under 50ms. Beacon's board below is set at 800ms, which sits deliberately between the two while the bot is in canary. Percentiles, never averages - session b3 explains why the average is the number that hides your worst conversations.
Take Beacon's support bot to all-green ★ 12 min · everyone builds
This is the same board b9 uses to run all three bots, pinned here to the chatbot at tier T3. Work it in the order below rather than switching everything on at once - the order is where the lesson is, and two of the moves make the number worse on purpose.
Touch nothing and press Ship anyway. Read the incident list. Every entry is a failure this session has a metric for, and none of them are visible from the pass rate you just shipped on.
Now switch on the three measurement levers only - golden set sampled from real traffic, adversarial and unanswerable slice, judge calibrated vs human labels. The number falls hard. Nothing about the bot changed; you stopped flattering it and you plugged in the b4 calibration.
Add cite-or-abstain grounding and retrieval / conversation context. These are the two biggest categories in a support set - the policy-citation cases and the multi-turn cases from Part 3 - and they move together.
Add abstention policy and action rails. Abstention is escalation correctness in lever form; rails are what stops an instruction pasted inside a customer ticket from becoming an action. Notice that output verification stays dimmed for this bot, and read why - there is nothing to execute, which is the whole premise of this session.
Bring the last gates green: human approval on irreversible actions (mandatory at T3, because of the refund credit), latency and cost engineering to pay for everything you just switched on, and the regression suite and canary in CI. Then push n to 500 and watch the confidence figure, not the pass rate.
Write down your minimum lever set - the smallest combination that holds all six gates green. Keep it. In b6 and b7 you will build the same list for the other two bots, and the three lists barely overlap. That difference is the entire argument for gating per archetype instead of running one company-wide eval template.
Write Beacon's resolution definition, then price the clauses ★ 10 min · everyone writes
The artifact this session is really about. Six sentences, written before anyone measures anything, and every one of them moves the headline number. Write yours now, on the bot you own, and then find out what each clause costs.
Write the hard clause first, because it is the only one that is not an inference. If your product has no way for a customer to confirm anything, you have just discovered a product gap that is cheaper to fix than any eval on this page.
Decide whether soft resolutions count, and set the window explicitly. Then run the number both ways on last month's logs. The gap between the two is the size of the argument you will otherwise have in a launch meeting.
Write the reversal clause and set its window. This is the clause that costs you the most and protects you the most - it is the only rule that stops a confident wrong answer from counting as a success, because a confident wrong answer produces exactly the return it detects.
List what never counts. Greeting-only sessions first, then anything a human touched. Both are places where a resolution rate quietly inflates while nobody is being dishonest.
Now build the multi-turn slice this definition needs: take five real sessions from your logs, script the customer turns verbatim, plant one instruction in turn 2 that turn 5 depends on, and grade each session as a single pass or fail. Five cases is enough to see the Part 3 collapse in your own product.
Finally, price the clauses. Recompute your resolution rate under strict rules and under loose ones, and put both numbers on the launch slide with the definitions underneath. The room will trust the strict number far more than it would have trusted a single unexplained one.
Try it yourself - this week ◐ 50-60 min total
- Write your resolution definition and circulate it to support ops before you circulate any number. If the two teams disagree about what resolved means, you have found the reason your dashboards have never agreed with each other.
- Compute containment and resolution separately on the same month of traffic. If containment is far ahead of resolution, that gap is your actual backlog - the bot is keeping people in the channel without helping them, and no quality metric on your board will say so.
- Build five multi-turn cases from real sessions, with a planted instruction in turn 2 and a whole-session pass rule. Run them. Nearly every team finds at least one failure the single-turn suite has been passing for months.
- Grade escalation in both directions for a week. Count missed escalations and over-escalations separately, and put the two counts side by side. Whichever number is larger tells you which way your bot is nervous, and they need opposite fixes.
- Take one adversarial pattern - an instruction pasted inside customer ticket text - and generate twenty variants of it rather than testing the one you thought of. Every variant that lands becomes a permanent regression case.
Official sources covered
Every figure on this page traces to a source catalogued in this course's source map, with secondary claims labelled as secondary rather than presented as measurement. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A vendor quotes a 67% resolution rate and asks why yours is 44%. What is the first thing to establish?
Resolution is not a standard quantity, it is whatever the definition says. Counting soft resolutions, dropping the reversal clause, or measuring only bot-routed traffic each move the number by a large margin, so two rates computed under different definitions are not comparable at all. The published range for that same vendor spans 38-72% across customers, which tells you the deployment matters as much as the definition - and both of those figures are vendor-reported, secondary claims rather than independent measurement.
2 · Your single-turn golden set scores 94%. What should you expect from real six-turn sessions, and why?
The drop is measured and it is large, and the mechanism matters more than the size: models do not become less capable across turns, they become less reliable - forgetting a constraint stated earlier, contradicting themselves, re-asking for something already supplied. That diagnosis points at conversation memory, state handling and multi-turn cases graded on the whole session. It does not point at a model upgrade, which is the expensive thing teams reach for first.
3 · Beacon's support bot can issue refund credits. What does that one capability do to this gate?
Tier follows blast radius, and blast radius follows the action space rather than the answer quality. A bot that can move money has an irreversible failure mode, which is what T3 encodes: a higher bar, a person committing the action, and a rehearsed kill switch. It is also the reason session b9 asks the uncomfortable question about whether the refund capability needs to exist at all - removing it drops the tier, the bar and the approval requirement in one move, and it is almost never proposed.