learn-agent-launch-with-phoebe / Leader session 2 of 6
Learn Agent Launch with Phoebe · Leader track · Session 2 of 6

Three bot archetypes: why one eval template cannot serve all three

A chatbot, a text-to-query bot and an analytics bot look like three instances of the same thing on an org chart. They are not. They differ in whether ground truth exists at all, which means they differ in what can be measured, what counts as good, how they fail, and what has to be true before they ship. This is the session where you stop asking "how accurate is it" and start asking "accurate at what, graded how, against whose bar".

🟢 Leader track Heads of product · eng leaders · risk & compliance 45 min
0-3 · Welcome 3-25 · The contrast, the ceiling, the three bots 25-40 · Two decisions, out loud 40-45 · Q&A
Part 0

The one distinction everything else follows from

Ask of any bot: can a machine tell whether the answer was right, without asking a person? For a text-to-query bot, yes - run the query, compare the rows. For a support chatbot, no - correctness is an opinion until a human or a calibrated judge supplies one. For an analytics bot, half and half, and the half that cannot be checked sits on top of the half that can. Everything on this page is a consequence of that one question.

Live - presented in session Self-study - read after class Official sources covered
★ What you walk out with today One slide that separates the three archetypes on five axes, an honest number for what "good" looks like in each - including why 40-70% is a normal support-bot result and 99% is a definition problem - the three converging reasons 80-90% is the real text-to-query ceiling, and the three Beacon bots you will be gating for the rest of this course.
Part 1 · the one slide

Five axes, three answers, no overlap 8 min live

This is the slide to carry into every launch review for the next year. If your organisation has one eval checklist, it was written for one of these three columns and it is blind to the other two.

Chatbot Text-to-query Analytics bot Ground truth None. Correctness is an opinion until a human or judge supplies one. Executable. Run both queries, compare the rows. Mixed. The numbers are checkable, the narrative around them is not. Headline metric Resolution rate - and the definition of resolution is the whole argument. Execution accuracy (EX). Exact-string match is deprecated. Task pass rate through a cascade: valid, then legal, then faithful. Realistic good 40-70% resolution. A well-configured RAG bot sits around 40-65%. 80-90% EX ceiling on clean, well-modelled schemas. ~75% chart pass rate for a GPT-4-class model on matplotlib. Signature failure Ungrounded answer: an invented policy stated with total confidence. Schema linking - 37% of errors on BIRD, up to 68% across many DBs. A legal-looking wrong chart. 21.44% illegal rate even on matplotlib. Unique gate Safety red team. It talks to customers, so somebody will try to break it. Read-only rails plus abstention - it must be able to refuse. Numeric reproducibility: the same question twice must give the same number. Read down the columns, not across the rows Nothing in this table transfers. A resolution-rate template applied to a text-to-query bot measures the wrong thing with the wrong grader against the wrong bar - and it will pass, because the bar was set for a different problem. One eval template for the whole company is the most expensive shortcut in this course. The bars are honest numbers from published benchmarks, not guarantees. They tell you what to expect before you measure, and what to be suspicious of after. A support bot claiming 99% is defining it wrong.
🔍 Click to zoom - five axes, three archetypes, and not one shared answer
LiveWhy 40-70% is a good support-bot number and 99% is a definition problem4 min▶

Resolution rate has no natural definition, so the number moves enormously with the choice - which makes it the single most gameable figure in this whole course. Intercom publishes theirs, and reading it is instructive precisely because most vendors do not:

  • Hard resolution means the customer explicitly confirmed they were helped. Soft resolution means the customer left and did not come back within 24 hours.
  • A soft resolution is reversed if the customer returns to the same thread. A greeting-only conversation never counts at all.
  • Across their reported base of 7,000+ customers and 40M+ conversations, production resolution runs 38-72% with an average around 67%. That average is a secondary claim reaching us through third parties, so treat the range as the more useful figure.

Separately, containment rate - the share of conversations never handed to a human - is commonly cited as 20-40% early, 70-90% at maturity, with a well-configured RAG chatbot around 40-65%. Those figures are Gartner-attributed and also secondary. Both metrics point the same way: somewhere between four and seven in ten is what a working support bot looks like. When somebody presents 99%, they have not built a better bot; they have counted greetings, or they have counted "the bot replied" as a resolution.

Self-studyThe analytics column is the other two stacked3 min read▶

An analytics bot runs a query and then writes a sentence about the result. The query half has executable ground truth. The sentence half does not. Errors compound across the join: a correct number described wrongly and a wrong number described convincingly both leave the building as an insight.

The published cascade for chart generation makes this concrete. In Microsoft's VisEval work, GPT-4 produced invalid code 3.29% of the time with matplotlib and 25.41% with seaborn - so validity itself depends on which library you asked for. Past validity comes legality: an illegal rate of 21.44% even on matplotlib, meaning code that runs cleanly and produces a chart that does not answer the question asked. Overall pass rate landed at 75.27% for GPT-4, 61.79% for GPT-3.5 and 51.59% for Gemini-Pro, with readability around 3.80 out of 5. A chart that renders is not a chart that is right, and a chart that is right is not an insight that follows.

Part 2 · the ceiling

Why 80-90% is the honest text-to-query number 8 min live

Of the three archetypes this one has real, machine-checkable ground truth, which should make it the easy case. It is instead the archetype where the loudest accuracy claims are made, because a runnable answer is easy to score and easy to score generously. Three independent pieces of evidence converge on the same ceiling.

1 The benchmark itself says humans are still better On BIRD, the standard text-to-SQL benchmark, human performance is 92.96% and state of the art sits around 81.95% on the test split. The gap has been the story of the field: the last ten points are the hard ones. 2 The gold answers themselves are wrong often enough to move rankings An audit of gold annotations found error rates of 52.8% on BIRD Mini-Dev and 62.8% on Spider 2.0-Snow. Re-scoring 16 agents on corrected data moved scores by -7 to +31 points and rankings by up to nine places. 3 The vendor number that beats the ceiling needed a lot of help to get there Snowflake reports 90%+ on an internal 150-question set, with a semantic model, agentic retries and lenient multi-gold scoring. Single-shot GPT-4o scored 51% on the same set. Internal, not independently verified. What the three together actually license you to say 80-90% execution accuracy on a clean, well-modelled schema is a good result. Above it, ask what was added: a semantic layer, retries, a lenient grader, or an easier set. Below it, on messy enterprise schemas, is normal - on Spider 2.0's enterprise workflows o1-preview scored 17.0%, against 91.2% on the older Spider 1.0. None of the three is an opinion. Two are published audits and one is a vendor disclosing its own method. That is the standard to hold any accuracy claim to, including the ones your own team brings you.
🔍 Click to zoom - three independent lines of evidence, one ceiling
Real world

The Snowflake disclosure is the most useful of the three, and it is the one people skip. A vendor reporting 90%+ could have stopped at the headline. Instead the method is published: an internal 150-question benchmark, a hand-built semantic model over the schema, agentic retries, and lenient scoring that accepts more than one correct answer. The same blog reports that single-shot GPT-4o managed 51% on that identical set. Read those two numbers together and you learn the thing that matters for your own project - most of the distance between 51% and 90% was bought with engineering around the model and generosity in the grader, not with the model itself. That is a genuinely encouraging finding, and it is also the reason you cannot take 90% into your own launch review as an expectation. It is internal and not independently verified; say so when you cite it.

Why schema linking is the failure to ask about When a text-to-query bot is wrong, it is usually not because it wrote bad SQL. It is because it picked the wrong table or the wrong column - 37% of BIRD errors by one analysis, with JOIN errors adding another 36%, and above 60% in large multi-database settings. That failure is invisible in the output: the query runs, returns rows, and the rows are wrong. It is also the failure your data model can fix without touching the model at all, which makes it the cheapest lever available to you as a leader. Session b6 measures it.
Part 3 · the running case

Meet the three Beacon bots 5 min live

Beacon is shipping all three archetypes this quarter, out of one engineering organisation, under one set of standards. Every remaining session in this course returns to these three, and the contrast between them is the spine of the whole thing.

Beacon Support Bot customer-facing · tier T3 Archetype Chatbot. No ground truth - correctness is an opinion. A wrong answer It issues a refund credit that cannot be undone. The gate that decides it Groundedness and a safety red team at a T3 bar, with human approval on refunds. Beacon Analyst Bot internal · tier T1-T2 Archetype Text-to-query. Executable ground truth - run the SQL. A wrong answer A plausible number in a board deck, unnoticed. The gate that decides it Execution accuracy on a held-out set, read-only rails, and abstention. Beacon Insight Bot exec-facing · tier T2 Archetype Analytics. Executable inner stage, judged narrative. A wrong answer A confident insight that the data does not support. The gate that decides it Validity, then legality, then faithfulness - plus numeric reproducibility. Same company, same discipline, three different gates One eval team, one set of standards, and three completely different definitions of ready. The support bot has the lowest honest accuracy of the three and the highest bar, because it is the only one that can move money. If your company has one launch checklist for all bots, it fits exactly one of them and is blind to the rest. Sessions a3 to a5 build the version that flexes by archetype and by risk tier.
🔍 Click to zoom - three bots, one quarter, three definitions of ready
The counter-intuitive line to remember Beacon's support bot will report the lowest number of the three and carry the highest bar. That is not a contradiction - accuracy and stakes are set by different things. Accuracy is set by whether ground truth exists; the bar is set by what happens when the answer is wrong. Any framework that ties them together will either over-gate the analyst bot or under-gate the support bot, and in practice it does both at once.
Decision 1 of 2

Classify your own portfolio ★ 8 min · everyone writes

List every bot, assistant or agent live or planned in your organisation. Then run each one down the five axes. The exercise takes eight minutes and usually produces two surprises.

Assign an archetype to each using only the ground-truth question: can a machine tell whether the answer was right without asking a person? Yes means text-to-query. No means chatbot. Partly means analytics, and partly is more common than people expect.

Write down the headline metric each one currently reports. Then check it against the second row of the table. A support bot reporting execution accuracy, or a query bot reporting a judge score, has borrowed the wrong template - and that borrowing is almost always invisible until you write the two side by side.

Write down what each one currently claims as its accuracy. Compare against row three. Anything meaningfully above the realistic band is a definition question, not an achievement, and you now know exactly which definition to ask about.

Circle the unique gate each one is missing. Most organisations run one gate - usually a quality number - across the whole portfolio. The safety red team, the read-only rail and the reproducibility check tend to be missing entirely, because they belong to only one column each.

Find the mismatch. Two surprises show up reliably: a bot everyone calls a chatbot that is really a text-to-query bot with a chat window on it, and a bot everyone calls an analytics tool that is generating narrative nobody is grading. Name both out loud.

The hybrid nobody classifies A support bot that can look up your order status is two archetypes in one conversation: a chatbot wrapped around a query. The retrieval half has executable ground truth and should be graded there; the conversational half does not and cannot. Teams grade the whole thing with a judge because the outside is a chat window - and throw away the cheapest, most truthful signal they have. Ask where the executable half of your chatbot is.
Decision 2 of 2

The 96% in the room ★ 7 min · everyone discusses

A vendor presents a text-to-query product at 96% accuracy. Your own team's internal prototype is at 74% and everybody in the room has just become quiet. Work out, in pairs, what you would ask - then compare against the list below.

★ The four questions, in the order that saves the most time

1 · On whose data? A published leaderboard number, an internal set, or your schema? Enterprise workflows crater results: 17.0% on Spider 2.0 against 91.2% on the older, cleaner Spider 1.0, from the same class of model.

2 · Who wrote the correct answers, and were they audited? Gold-annotation error rates of 52.8% and 62.8% have been measured on two widely used splits, and correcting them shifted rankings by up to nine positions. A number scored against unaudited gold answers is a number with a wide unlabelled error bar.

3 · What was added around the model? A semantic layer, retries, multi-gold lenient scoring - all legitimate engineering, and all of it yours to build and maintain. Single-shot with none of it can be half the headline figure.

4 · What does it do when it does not know? A model with no abstention path can score worse than useless once you penalise confident wrong answers, which is exactly what a reliability-scored benchmark does. Ask to see a question it refuses.

What you are actually deciding Not whether 96% is true - it may well be, on the set it was measured on. You are deciding whether that measurement transfers to your schema, your questions and your grader. The honest answer is usually "some of it does", and the four questions tell you which part. The version of this conversation that goes badly is the one where nobody asks, the number is believed, and the gap is discovered in production by an analyst who trusted a plausible-looking figure.
Homework

Try it yourself - this week ◐ 40-50 min total

Source material

Official sources covered

Every figure on this page traces to a primary source catalogued in this course's source map, and secondary or internal claims are labelled as such on the page rather than only here. This page covers:

✓Chatbot reality band: Fin production resolution 38-72% across 7,000+ customers and 40M+ conversations, ~67% average; containment 20-40% early, 70-90% mature, 40-65% for a well-configured RAG botPart 1 · the 67% average and the containment figures are secondary and are marked so on the page; the resolution definition itself is vendor-published
✓Text-to-query ceiling: BIRD human performance 92.96% vs SOTA ~81.95%; Spider 2.0 enterprise workflows at 17.0% vs 91.2% on Spider 1.0Part 1 and Part 2 · execution accuracy as the metric, exact-set-match deprecated
✓Gold-annotation audit: 52.8% error rate on BIRD Mini-Dev, 62.8% on Spider 2.0-Snow; re-scoring 16 agents moved results -7 to +31 points and rankings by up to nine positionsPart 2 evidence 2 · the reason a leaderboard number carries an unlabelled error bar
✓Snowflake Cortex Analyst 90%+ on an internal 150-question set with semantic model, agentic retries and lenient multi-gold scoring; single-shot GPT-4o 51% on the same setPart 2 evidence 3 · internal and not independently verified, stated as such on the page
✓Schema linking as the dominant failure: 37% of BIRD errors plus 36% JOIN errors; above 60% in large multi-database settingsPart 2 callout · measured in b6
✓VisEval cascade: GPT-4 invalid code 3.29% matplotlib vs 25.41% seaborn, illegal rate 21.44%, overall pass 75.27% (GPT-3.5 61.79%, Gemini-Pro 51.59%), readability 3.80/5Part 1 self-study and the contrast table · full cascade in b7
◐Abstention and reliability scoring, where a model without a refusal path can score negativeNamed in Decision 2 · measured in b6
◐Risk tiers T1-T4 and the launch bar attached to eachUsed loosely on the Beacon cards · defined properly in a5
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · A team reports 99% resolution on your customer support bot. What have you most likely been shown?

Resolution has no natural definition, which makes it the most gameable number in the course. Intercom's published rules are worth borrowing precisely because they are strict: hard resolution needs explicit confirmation, soft resolution needs the customer to stay away for 24 hours, a soft resolution is reversed if they come back, and greeting-only conversations never count. Under rules like those, production runs 38-72%. Ask for the definition before you argue about the number.

2 · Why is 80-90% treated as the honest ceiling for text-to-query accuracy rather than a target to be beaten?

The ceiling is empirical, not theoretical. Each of the three pieces would be arguable alone; together they say the same thing from a benchmark's own human baseline, from an independent audit of the answer key, and from a vendor's own account of its method. Note what the third one actually shows: most of the distance from 51% to 90% was bought with engineering around the model and generosity in the grader, and both of those are things you would have to build and maintain yourself.

3 · Your organisation has one eval checklist used for all three bots. What goes wrong?

The dangerous outcome is not a false alarm, it is a silent pass. Every axis differs: ground truth exists for one and not the others, the headline metric is different, the honest band for good is different, the signature failure is different, and each has a unique gate the others do not need. A checklist built around resolution rate will never ask a text-to-query bot whether it can refuse, and will never ask an analytics bot whether the same question twice gives the same number.

Leader session 2 cheat sheet · pin this

The one sorting questionCan a machine tell whether the answer was right without asking a person? No = chatbot. Yes = text-to-query. Partly = analytics.
Five axes, no overlapGround truth · headline metric · realistic good · signature failure · unique gate. Read down the columns, never across.
40-70% is a working support botProduction resolution runs 38-72%, average around 67% (secondary). A 99% claim is a definition, not an achievement.
80-90% is the query ceilingBIRD human 92.96% vs SOTA ~81.95%; audits found 52.8% and 62.8% bad gold answers; the 90%+ vendor claim is internal and heavily assisted.
~75% is a good chart botGPT-4-class pass rate 75.27% on VisEval, with a 21.44% illegal rate even on matplotlib. A chart that renders is not a chart that is right.
Accuracy and stakes are set separatelyGround truth sets how accurate you can be. Blast radius sets how accurate you must be. Beacon's support bot is lowest on one and highest on the other.
Schema linking is the cheap lever37% of BIRD errors, above 60% across many databases. Fixed by the data model, not the model - and it is your call, not the vendor's.
The four questions for any claimOn whose data · who wrote the gold answers · what was added around the model · what does it do when it does not know.