The archetype where the machine can settle the argument
Session b5 gated a bot with no ground truth, where correctness stayed an opinion until a human or a calibrated judge supplied one. Beacon's Analyst Bot is the opposite animal: every answer it produces is a query, and a query can be executed. That is the cheapest, sharpest grading rung on the ladder from b1, and this archetype lives on it.
Execution accuracy, and the ceiling nobody puts on the slide 7 min live
Two correct SQL queries almost never look alike. They join in a different order, alias differently, use a different date predicate, and return exactly the same rows. Any metric that compares the text rejects one of them, which is why the field moved to comparing what the queries do.
LiveWhy the ceiling matters to your launch, not just to a leaderboard4 min▶
Three numbers, three different warnings, and they only make sense together.
- BIRD: SOTA around 81.95%, human 92.96%. Humans do not reach 100% on this task either. When somebody asks your bot for 95% accuracy on natural-language queries, the honest answer is that measured human performance on the field's main benchmark is 92.96%.
- Spider 2.0: o1-preview at 17.0%. The same class of model scores 91.2% on the older Spider 1.0. Nothing about the model collapsed - the schemas got real. Enterprise warehouses have hundreds of tables, dialect quirks, and multi-step workflows, and academic benchmark scores do not transfer to them.
- The annotation audit: 52.8% and 62.8% gold-error rates. Re-evaluating 16 agents on corrected data shifted scores by -7% to +31% and moved rankings by up to nine positions. That is not a rounding correction; it is the leaderboard being partly a measurement of the labels.
The practical consequence for your own set is uncomfortable and worth saying out loud: above roughly 80% on any hand-annotated text-to-SQL set, some of what you are measuring is your annotators. That is not a reason to stop measuring. It is a reason to audit your own gold queries before you audit your model, and to stop treating the last few points as engineering progress.
Self-studyReading a vendor accuracy claim, using Snowflake's as the worked example3 min read▶
Snowflake published a 90%+ accuracy figure for Cortex Analyst. It is a useful number to read carefully, because every clause in it is doing work. The claim is measured on an internal 150-question benchmark, it is not independently verified, and it required three things at once: a curated semantic model over the tables, agentic retries rather than a single attempt, and lenient multi-gold scoring that accepts several correct queries per question. On the same internal set, single-shot GPT-4o scored 51%.
None of that makes the number dishonest. It makes it a number about a system on a specific set, and the 51% tells you how much of the system is doing the lifting. The transferable lesson is the shape of the question you ask any vendor, including yourself: whose questions, whose gold, how many attempts, how lenient the scoring, and who checked it. Session a3 turns those five into a grilling script.
The aggregate hides which way it failed 6 min live
Two bots reporting 78% execution accuracy can be completely different products. One fails loudly, in a way the analyst notices within a second. The other fails silently, in a way that reaches a planning meeting. Report the taxonomy, not the headline.
The 3.1x revenue bug. Beacon's Analyst Bot joined orders to customers on the wrong key and reported 3.1 times the real revenue for a region. The query was syntactically perfect, ran in 400ms, returned a tidy grouped table, and nobody questioned it until the finance close two weeks later. That single incident is bucket 3 in the taxonomy and a schema-linking failure in the error class - the two most common findings in the published error analyses meeting in one production bug. The fix was not a bigger model. It was column descriptions and a curated set of joinable keys handed to the bot as context.
Self-studyWhy schema linking is the lever, and what "context" concretely means3 min read▶
Schema linking is the step where a natural-language phrase gets bound to an actual table and column. "Revenue" could be orders.amount, orders.gross, invoice_lines.net or a view somebody built in 2023. The bot cannot guess, and published error analyses keep landing in the same place: schema linking accounts for roughly 37% of BIRD errors with a further 36% attributable to JOIN mistakes, and in large multi-database settings the schema-linking share rises above 60%, measured up to 68.3%.
In practice, "give it schema context" means four unglamorous artifacts: column descriptions written by somebody who knows the business, an explicit list of joinable key pairs, a shortlist of the canonical tables for each common question type, and a set of worked examples of correct queries. None of that is modelling work. All of it is data-product work, which is precisely why it tends to be nobody's job and stays undone while the team argues about models.
Three criteria execution accuracy will never see 5 min live
A perfect EX score still leaves three launch questions unanswered: what the bot does when the question has no answer, what happens when a question has several right answers, and what stops it doing something destructive. Only one of those is a metric. The other two are a scoring choice and a rail.
| Criterion | How you score it | What it catches that EX cannot |
|---|---|---|
| Execution accuracy | Run predicted and gold, compare result multisets | The core correctness question, deterministically |
| Failure taxonomy | Split every miss into syntax, empty, wrong-non-empty | Whether your misses are loud or silent |
| Abstention | Reliability Score with a penalty for confident wrong answers on infeasible questions | A bot that never says "that cannot be answered from this warehouse" |
| Multiple gold queries | Several accepted golds per question, lenient column-level precision and recall | Valid alternative queries being marked wrong by a single-gold set |
| Efficiency | R-VES: execution accuracy weighted by runtime ratio against gold | A correct answer that scans 4.2TB and costs more than the analyst's afternoon |
| Rails | Not scored - enforced. Read-only role, allowed verbs, row limit | Nothing. It removes the failure class rather than measuring it |
Build the Analyst Bot's execution harness ★ 12 min · everyone builds
Take the golden-set skeleton from b1 and give it the one thing this archetype allows: a runner. Twenty minutes of harness work replaces every judge call you were about to make.
For each case, freeze three things: the natural-language question, the gold SQL, and the gold result multiset stored as sorted rows. The stored rows are the ground truth. The gold SQL is only how you produced them, and it is allowed to be rewritten later without invalidating the case.
Write the runner: execute the predicted query against a frozen snapshot of the warehouse, execute or load the gold, and compare sorted multisets. Classify every miss into syntax · empty · wrong-non-empty and report the three rates alongside execution accuracy. If your report has one number on it, you built half a harness.
Set the category shares from a real query log, not from imagination. Beacon's Analyst Bot uses six: single-table filter and aggregate 25%, multi-table join needing schema linking 25%, under-specified question 15%, silently-wrong-filter 10%, correct-via-full-table-scan 10%, and an adversarial slice of questions engineered to emit a DROP at 15%.
Add the cases the golden-set method structurally omits: questions the warehouse cannot answer, and questions where "best" or "top" is undefined. Their expected behaviour is an abstention with a named reason, and they are scored right or wrong like any other case. This is the slice that decides whether your bot is honest.
Record runtime / gold_runtime next to every pass, and put the rails in the database rather than the prompt. A read-only role and a row limit are enforcement; "please do not drop tables" is a wish.
Take the Analyst Bot to all green ★ 10 min · everyone builds
The same board b9 uses, mounted on this archetype at tier T2. Two levers are dimmed as "not a gate for this bot" - work out why before you hover them.
Press Ship anyway at the default settings and read the incident list. Every line in it is one of the four taxonomy buckets wearing production clothes, and the 3.1x revenue bug is right there in the middle.
Switch on Retrieval / schema context alone and watch the multi-table join category recover. That single lever is the answer to the largest published error class in this archetype, and it costs you column descriptions rather than a model upgrade.
Hover the two dimmed levers. Cite-or-abstain grounding is dimmed because there is no passage to cite: correctness here is settled by executing the query, not by tracing prose back to a source. Human approval on irreversible actions is dimmed because a read-only query is reversible by definition - and it stops being dimmed the moment somebody lets this bot write.
Now switch on Action rails and Output verification together, then take the board to all six gates green. Note which lever fixed the full-table-scan category and which one fixed the silently-wrong filter. They are different levers and they defend different buckets of the taxonomy.
Try it yourself - this week ◐ 50-60 min total
- Run your existing text-to-query eval and split the misses into syntax, empty and wrong-but-non-empty. Report the three rates to your team next to the aggregate. The reaction to the third number is usually the most productive meeting you will have this quarter.
- Take twenty of your gold queries and have somebody else re-derive the expected result independently. Count the disagreements. That count is your annotation error rate, and it is the ceiling on what your eval can prove.
- Add ten cases whose correct answer is an abstention: five infeasible, five under-specified. Score them. If your bot answers all ten confidently, you have found the failure that no accuracy number on your dashboard was ever going to show you.
- Check what database role your bot actually connects as. If it can write, you are measuring a failure class you could have deleted. Move it to a read-only role with a row limit this week, and then delete the corresponding eval cases, because you no longer need them.
- Log
runtime / gold_runtimefor one week of production queries. Find the correct answers that cost the most. Efficiency is a real gate on this archetype and almost nobody sets one until finance does it for them.
Official sources covered
Every figure on this page traces to a primary source - published benchmarks, peer-reviewed papers and vendor documentation catalogued in this course's source map, dated 2026-08-13. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Why did the field deprecate exact-set-match in favour of execution accuracy?
Exact-set-match punishes valid alternative SQL, which makes it a measurement of writing style as much as correctness. Execution accuracy asks the only question that matters to the analyst - did the right rows come back - and it settles it deterministically, with no judge, no opinion and no dialect argument. Option C is a real irritation but it is not why the metric was retired.
2 · Your bot misses 22% of cases. Which failure bucket should worry you most?
Syntax errors are caught by a dry-run and empty results are caught by the analyst within a second. Neither reaches a slide. A silently wrong result set does - Beacon's bot joined on the wrong key, over-reported regional revenue by 3.1x, and nobody noticed until the finance close. That bucket is also where schema-linking failures land, which is roughly 37% of BIRD errors plus a further 36% of JOIN errors, and it is the reason you split the taxonomy instead of reporting one aggregate.
3 · A vendor cites 90%+ text-to-SQL accuracy. What is the honest way to read it?
The number is not dishonest and option C throws away real information. It is a claim about a whole system on one specific internal set, and the 51% single-shot figure tells you how much of the result comes from the semantic model, the retries and the scoring leniency rather than the model. It is also not comparable to BIRD, where SOTA sits around 81.95% against human performance of 92.96% - and where an audit found 52.8% gold-annotation errors in one split, which is the reason to be sceptical of everybody's high numbers, vendors included.