The number that sets expectations 8 min live
One model, two benchmarks. GPT-4o scores 86.6% on Spider 1.0 and 10.1% on Spider 2.0. Nothing about the model changed. What changed was the conditions: Spider 2.0 uses 632 real enterprise workflow problems on databases with over 1,000 columns - the largest exceeding 3,000 - across multiple SQL dialects, with solutions that often run past 100 lines. That collapse is the precise distance between the demo you were shown and the warehouse you actually own.
LiveWhat the academic baselines actually say4 min▶
- BIRD (12,751 question-SQL pairs, 95 databases, 33.4 GB): human performance is 92.96%. Not 100 - people get this wrong too. The best leaderboard system reaches 81.95%. So the human-machine gap is about 11 points, and neither is close to perfect.
- LiveSQLBench, built specifically to resist contamination by pulling from live, regularly-updated databases: the leader sat at 48% as of March 2026, with most systems clustering at 35-40%.
- Spider 2.0-Snow leaderboards show entries above 96%. Hold that thought - Part 3 explains why that number is not what it appears to be.
The honest framing for a stakeholder: on genuinely enterprise-shaped problems, contamination-controlled measurement puts the state of the art below half. Everything above that number depends on work you do, not on the model.
Self-studyWho publishes accuracy figures, and who does not3 min read▶
Worth knowing before any procurement conversation:
- Snowflake Cortex Analyst is the only vendor publishing a number: "90%+ SQL accuracy." To their credit they published methodology - and the methodology is an internal, unpublished 150-question suite. They also published the baseline, which is the more useful figure: GPT-4o single-shot scored 51% on that same internal eval despite 86.6% on public Spider 1.0. And a semantic model lifted the same LLM from 57% to 78% on BIRD.
- Databricks, ThoughtSpot and Microsoft publish no accuracy figure at all for their conversational analytics products.
Treat that absence as informative. Nobody in this category wants to be measured on a public enterprise benchmark - which tells you roughly where they expect to land.
The bottleneck is metadata, and it was measured 8 min live
Microsoft's engineering team ran Databricks Genie against LiveSQLBench in three configurations, publishing their methodology. Same product, same benchmark, same questions. Only the semantic context around the data changed - and accuracy moved by a factor of seven. This is the strongest evidence in the whole course for why the semantic layer exists, and it did not come from a company selling semantic layers.
| Configuration | Accuracy | What changed |
|---|---|---|
| Genie, empty configuration | 9.50% | the product, pointed at raw tables |
| Genie + column descriptions | 69.23% | somebody wrote down what the columns mean |
| Genie + feedback mechanism (3 iterations) | 88.50% | the system learns from corrections |
| GitHub Copilot CLI, metadata only | 66.70% | (comparison point) |
| GitHub Copilot CLI + domain hints | 80.77% | (comparison point) |
LiveWhat you are actually buying4 min▶
Read those first two rows again: 9.50% to 69.23%, a 7x improvement, from column descriptions. Not a better model. Not a bigger context window. Somebody documenting what the columns mean.
Which reframes the purchase entirely. You are not buying accuracy. You are buying a tool that will be exactly as accurate as the metadata work underneath it. Two consequences for how you plan:
- The licence cost is the small number. The metadata work is the real project, and it is the one nobody scopes.
- A pilot on a well-documented data mart tells you almost nothing about performance on the undocumented warehouse - which is exactly why pilots succeed and rollouts disappoint.
LiveThe vendors' own fine print4 min▶
Two product pages say the quiet part, and both are worth reading aloud in a planning meeting.
Databricks' best-practices page states that "Quality table and column descriptions in Unity Catalog are critical for Genie accuracy," that "Genie Agents support up to 30 tables or views," and recommends: "Aim for five or fewer tables. The more focused your selection, the better."
Five tables is a curated data mart. Set that against any framing of "ask any question of all your data," including from the same vendor. The tool is real and genuinely useful within its scope; the marketing frame is not the product spec.
Microsoft's Power BI Copilot documentation is the most candid vendor statement in this category:
"You need to prepare data to work with Copilot. Model owners need to invest in prepping their data for AI to ensure Copilot understands the unique business context... Without this prep, Copilot can struggle to interpret data correctly - leading to generic, inaccurate, or even misleading outputs."
A vendor stating that its own product produces misleading outputs without upstream investment is more useful than any benchmark, and it is the right sentence to put in front of an executive before a rollout, not after.
One more from the same docs, easy to miss and consequential: when a question cannot be answered from the semantic model, Copilot "answers from the large language model's general knowledge." The system does not fail closed. It fails plausibly.
The benchmarks are broken too 6 min live
Here is the move that separates a sophisticated evaluation from a credulous one. Vendor claims are unaudited - fine, we knew that. But the independent benchmarks we would use to audit them turn out to be substantially mis-annotated. A peer-reviewed CIDR 2026 paper from the University of Illinois examined the ground truth itself.
In BIRD Mini-Dev, 52.8% of problems contain annotation errors - 16.7 points higher than the previously reported rate.
In Spider 2.0-Snow, among the 121 problems with open-sourced gold SQL, the annotation error rate is 66.1%. Two thirds.
Re-evaluating five leading agents after corrections moved scores by up to 19 points and rankings by up to 3 places. One agent, CHESS, went from 62% to 81% - and from 4th place to 1st.
The authors' conclusion: "annotation errors remain a barrier to robust benchmarking in text-to-SQL, with direct implications for leaderboard validity."
LiveSo what about those 96% leaderboard scores?4 min▶
Spider 2.0-Snow's leaderboard shows entries above 96%. Four things to know before quoting them:
- They are commercial vendor submissions of their own products, with no evidence of independent validation by the benchmark maintainers.
- The benchmark being topped has a 66.1% annotation error rate on its open-gold subset. A near-perfect score on a two-thirds mis-annotated benchmark measures something, but not capability.
- The maintainers themselves warn against fine-tuning on the released gold SQL "as it may affect the fairness of evaluation." Public gold plus commercial incentive is a well-known saturation path.
- Contamination-controlled LiveSQLBench tells a different story: 48%.
The honest position to hold: nobody currently has a trustworthy public measurement of enterprise text-to-SQL accuracy. Any single number - 90% or 10% - is a claim about a specific setup, not a property of the technology. That is not cynicism; it is the accurate state of the evidence, and it is a far more defensible thing to tell your executives than a vendor's percentage.
Self-studyThe failure mode that actually matters3 min read▶
Microsoft's evaluation found something more important than the headline numbers: the majority of remaining failures across every system were business logic errors - incomplete aggregations, incorrect metric calculations, ambiguous interpretations needing domain expertise - not broken SQL.
Sit with the implication. A query that errors out is safe: someone notices. A syntactically valid query that returns a confidently wrong number is dangerous, because nothing flags it. It goes in a deck. It informs a decision. The failure surfaces months later, if ever.
This is precisely why metric definitions live in a semantic layer rather than in each question. The layer does not make the model smarter - it removes the ambiguity the model would otherwise resolve by guessing.
The trap, in your platform ★ 12 min · everyone runs it
Back to the simulator, with all eight layers on. Switch off the semantic layer and watch the business-client card. They stay unblocked - the tool keeps answering - and turn red. That single state change is this entire session, rendered in your own architecture.
All eight layers on: the client card is green, "asks questions and self-builds dashboards on governed metrics." Five of five personas working.
Switch Semantic layer + BI off. The count drops to 3 of 5, and the client goes red: "Answering confidently from ungoverned metrics - plausible SQL, wrong business logic." They are not blocked. They are being answered, wrongly, with confidence.
Note which other persona went red-adjacent: your analyst is now blocked too, because they needed that layer as well. The semantic layer serves both - which is the argument for funding it that actually lands.
Switch it back on. That toggle, in a real organization, is somewhere between three months and a year of metric definition work. Price the toggle accordingly when you build the roadmap in b10.
Try it yourself - this week ◐ 30-40 min total
- Pick the five tables your business asks about most. Check honestly: do their columns have descriptions? That audit is the difference between 9.50% and 69.23%, and it takes an afternoon to assess.
- Write the one-paragraph promise you would make to an executive about self-serve at your organization, using only claims the evidence in this session supports. Bring it to b10.
- If you already run a conversational analytics tool, sample twenty real questions and check the generated SQL against what an analyst would have written. Count business-logic errors separately from failures - the ratio is the interesting number.
- Optional read: the CIDR 2026 paper on text-to-SQL annotation errors. It is short, and it will permanently change how you read a leaderboard.
Official sources covered
This session is deliberately structured as vendor claim versus independent evidence, because the gap between them is the lesson. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Genie scored 9.50% with an empty configuration and 69.23% with column descriptions. What does that prove?
Same product, same benchmark, same questions - only the surrounding metadata changed, and accuracy moved 7x. The comparison systems showed the same pattern (66.70% to 80.77% with domain hints). The licence is the small cost; the metadata work is the real project.
2 · A vendor cites a 96% score on the Spider 2.0-Snow leaderboard. What is the strongest objection?
The peer-reviewed CIDR 2026 audit found corrections moved one agent from 62% to 81% and from 4th to 1st place. A near-perfect score on a two-thirds mis-annotated benchmark is not a capability measurement.
3 · Why is a confidently wrong answer worse than a blocked user?
Microsoft's evaluation found most remaining failures across all systems were business logic errors, not SQL failures. A query that errors out is safe because someone notices. That asymmetry is why self-serve sits above the semantic layer rather than beside it.