The claim this session has to earn
Every vendor demo of "ask your data a question" works. Every real deployment of it disappoints, and the postmortem is almost never about the language model. It is about a schema that never wrote down what one row means, a "revenue" column that three teams define differently, and two fact tables joined on order_id because the join ran without error.
So the claim is this: an AI analyst is only as accurate as the model underneath it, and closing the gap is modeling work, not prompt work. By the end of this session you will have measured that gap yourself, on the model you built in b6-b8.
sql/40_agent_views.sql, semantic/schema_card.md, semantic/contract.yaml, semantic/golden_questions.jsonl - plus three runnable scripts and a scorecard that proves they work.
The four ways a model breaks an agent 7 min live
These are not hypothetical. Each one is a real query the lab below runs against Bazaar, with the real wrong number it produces. Every one is fixed by publishing something the model already knows.
LiveNames are the prompt3 min▶
The schema is the largest single thing an agent reads before it writes a query. Every unclear name spends tokens and buys a guess, so naming stops being a style preference and becomes an accuracy lever:
net_revenue, notamt2. A name that needs a glossary needs the glossary in the context window.is_declined, notflag3. A boolean named after its meaning can be summed into a numerator. A boolean named after its position cannot.- Grain in the view name.
v_sales_linesays one row is one line.v_saleswould not. - Different facts, different names.
list_priceon the product andunit_priceon the line - the naming is what stops an agent treating them as the same number (session b2's point, now load-bearing).
gross_revenue, net_revenue, commission, aov. Writing that mapping down (it is a dozen lines in semantic/contract.yaml) removes an entire class of guess for zero engineering cost.
Self-studyRemove the footgun, do not prompt around it3 min read▶
You can tell an agent "never join the two fact tables". It will mostly comply. Or you can build the access layer so that no view exposes both facts, and the join becomes unwritable - which is what sql/40_agent_views.sql does.
Prefer the second every time. The general principle: constraints beat instructions. An instruction is probabilistic and competes with everything else in the context window; a schema that lacks the dangerous path is deterministic. Applied to Bazaar:
- Read-only credentials on views only. No base tables, no writes.
python/agent_text_to_sql.pyalso enforces this in code, twice - an allowlist regex and a read-only SQLite connection. - No fact-to-fact path. The two facts are never both reachable in one view, so the double-count query cannot be expressed.
- Pre-split measures.
approved_amountanddeclined_amountexist so that no ratio has to be stored, so no average of averages is available to compute. - Unknown members instead of NULLs. A NULL dimension key silently drops rows from an inner join; a
-1unknown row keeps them countable (session b6).
Ask the agent, with and without the contract ★ real SQL, real execution
Pick a question. The lab shows you the SQL an agent writes for it, runs that SQL against the real Bazaar star in your browser, and tells you whether the number is right. Then flip the contract lever off and watch the same question produce a confident wrong answer. Finish with Score all 20 questions for the whole scorecard.
Start with the contract ON. Read what the agent can see: five views with stated grains, thirteen metric definitions, the join rules. Ask a few questions and check the numbers.
Flip the lever off. Same question, same database - now the agent has table names and nothing else. Read the SQL it writes and the wrong number it produces.
Try q02, q04 and q20 specifically. Those are the grain error, the averaged average and the fact-to-fact double count - the three failures that cost the most in production.
Score all 20. 0/20 without, 20/20 with. That delta is what "agent-ready" means, and it is entirely your modeling work.
Execution match is the honest metric. The scorecard does not compare query text - two correct queries can look nothing alike, and a wrong query can look reasonable. It runs both and compares the result sets, to the cent. That is the same grading approach published text-to-SQL benchmarks use, and it is the only one that cannot be gamed by writing plausible SQL. python/eval_golden_questions.py runs the identical grading locally, against a live model if you want.
The four artifacts 6 min live
"Agent-ready" is not a setting. It is four files, each of which is useful to humans on its own - which is the tell that this is real modeling work rather than AI theatre.
| Artifact | What it is | What it prevents |
|---|---|---|
sql/40_agent_views.sql | five flat, self-describing views + a metric table, read-only | the fact-to-fact join, base-table access, cryptic column names |
semantic/schema_card.md | the prompt: every view, its grain sentence, six hard rules | grain errors, invented thresholds, rates with no denominator |
semantic/contract.yaml | owner, SLO, allowed and forbidden joins, metrics, synonyms, known limitations | silent breaking changes, definition drift, undocumented modeling choices |
semantic/golden_questions.jsonl | 20 questions with the grounded SQL and the recorded ungrounded failure | regressions - it is a test suite for your model's answerability |
LiveMetrics as data, not documentation3 min▶
The highest-leverage single object in the whole model is a table of metric definitions inside the database. Six columns: name, definition, SQL expression, source view, grain, caveat. Thirteen rows for Bazaar.
SELECT metric_name, sql_expression, grain, caveat
FROM v_metric_definitions
WHERE metric_name IN ('net_revenue', 'orders', 'aov', 'decline_rate');
Note the caveat column - it is the one people leave out and the one that saves the most damage. "Never average an AOV." "Below ~30 attempts it is noise." "Carts, not sessions." Those sentences are the difference between a metric an agent can use and a metric an agent can misuse fluently.
LiveQuery-to-text: narrate from metadata, not from vibes3 min▶
The return trip matters as much. A fluent sentence around a wrong number is worse than a table, because it removes the reader's last chance to notice. python/agent_query_to_text.py narrates deterministically from the model's own metadata: it classifies each column from its name, demands a denominator for anything that looks like a rate, flags small samples against the 30-observation floor, and appends the contract's caveat for any metric the result names.
SELECT payment_method_label,
COUNT(*) AS attempts,
SUM(is_declined) AS declines,
ROUND(100.0 * SUM(is_declined) / COUNT(*), 2) AS decline_rate_pct
FROM v_payment_attempt
GROUP BY payment_method_label
HAVING COUNT(*) >= 30
ORDER BY decline_rate_pct DESC;
Drop the HAVING clause and re-run. A method with a handful of attempts jumps to the top of the ranking. The model cannot stop someone asking for that; the caveat and the visible denominator are what stop them acting on it.
Self-studyThe three scripts, and how to run them4 min read▶
All three live in python/ and all three run offline, with no API key, so nothing here is a demo you cannot reproduce.
agent_text_to_sql.py- the full loop against the live Claude API: schema card as system prompt, one strictrun_sqltool, a read-only allowlist guard, results fed back until it answers.--offlineuses the recorded golden queries instead;--no-contractwithholds the schema card so you can watch it fail.agent_query_to_text.py- narration.--golden q14narrates a golden question;--llmadds a Claude-written version grounded on the deterministic facts, so the prose cannot contradict the metadata.eval_golden_questions.py- the scorecard.--mode groundedscores 20/20,--mode naivescores 0/20,--mode livecalls a real model with or without the contract.
Try it yourself - this week ◐ 30-40 min total
- Write a schema card for one model you own. One view per section, a grain sentence each, and the three rules someone would otherwise get wrong. Keep it under 100 lines - it is a prompt, not documentation.
- Create a metric-definitions table in your own warehouse. Five metrics is enough to start. Include the caveat column, and fill it in honestly.
- Write five golden questions for your model, with the SQL you believe is correct. Run them. If any of the five is hard to write, that is a modeling gap, not a SQL problem.
- Take one metric everyone in your organisation uses and ask three people to define it precisely. Compare. That gap is what the contract file closes.
- Run
python python/eval_golden_questions.py --mode naivelocally and read every failure explanation. Each one is a modeling decision you now know to make. - Bring one subject area your model does not cover to the capstone - b10 takes returns and refunds all the way down the ladder.
Sources covered
Full source map in materials/official-course-map.md. This page covers:
agent_text_to_sql.pylearn-prompt-engineering, learn-rag, learn-text-to-sqlThree questions before you go 🎯 ◐ 90 seconds
1 · The agent joins v_sales_line to v_payment_attempt on order_id and reports revenue of 121,003 instead of 111,906. What is the fix?
Constraints beat instructions. An instruction competes with everything else in the context window; a schema that lacks the dangerous path is deterministic. Remove the footgun rather than asking the agent not to pull the trigger.
2 · Why does the lab grade by comparing result sets rather than the SQL text?
Execution match is the honest metric. Text similarity rewards plausible-looking SQL, which is exactly the failure mode you are trying to detect. Run both queries, compare the numbers to the cent.
3 · Which single artifact does the most to prevent an agent inventing its own metric definitions?
Metrics as data beats metrics as documentation: the definition cannot drift out of sync with what is queryable, an agent can look it up mid-query, and a human gets an answer instead of a Slack thread. Thirteen rows did most of the work in this session.