learn-data-modeling-with-phoebe / Builder session 9 of 10
Learn Data Modeling with Phoebe · Builder track · Session 9 of 10

Agent-ready modeling

Text-to-SQL fails for boring reasons, and almost all of them are modeling reasons: cryptic names, two tables that look joinable but are not, metrics with no single definition, grains nobody wrote down. This session publishes Bazaar's model so an AI analyst can query it accurately - views, a schema card, a metric table, a data contract - and then measures the difference on twenty real questions. Without the contract the agent scores 0 out of 20. With it, 20 out of 20. Same database, same model, same questions.

🔴 Builder track Analytics engineers · DE · DS · AI engineers Signature lab · real SQL execution 45 min
0-3 · Welcome 3-15 · Why agents fail on good data 15-40 · The lab + the four artifacts you ship 40-45 · Q&A
Part 0

The claim this session has to earn

Every vendor demo of "ask your data a question" works. Every real deployment of it disappoints, and the postmortem is almost never about the language model. It is about a schema that never wrote down what one row means, a "revenue" column that three teams define differently, and two fact tables joined on order_id because the join ran without error.

So the claim is this: an AI analyst is only as accurate as the model underneath it, and closing the gap is modeling work, not prompt work. By the end of this session you will have measured that gap yourself, on the model you built in b6-b8.

Live - presented in session Self-study - read after class ▶ Live lab - real SQL execution Sources covered
★ What you walk out with today Four shipped artifacts - sql/40_agent_views.sql, semantic/schema_card.md, semantic/contract.yaml, semantic/golden_questions.jsonl - plus three runnable scripts and a scorecard that proves they work.
Part 1 · covers text-to-SQL benchmark practice, dbt model contracts

The four ways a model breaks an agent 7 min live

These are not hypothetical. Each one is a real query the lab below runs against Bazaar, with the real wrong number it produces. Every one is fixed by publishing something the model already knows.

1 · Wrong grain COUNT(*) on order lines 481 not 327 orders Fix: publish the grain sentence per view 2 · Averaged average AVG(line revenue) as AOV 80.68 not 116.93 Fix: metric table with the exact expression 3 · Fact-to-fact join sales joined to payments 121,003 not 111,906 Fix: never expose both facts in one view 4 · Rate, no base 18.2% decline rate... ...on 11 attempts Fix: ship numerator and denominator None of these four is a language problem. Each one is something the model knew and never wrote down. Publish the grain, the metric definitions, the permitted joins and the denominators, and all four disappear. The uncomfortable part: every one of these wrong numbers is a query that RAN. No error, no warning, no NULL. A confident wrong answer is the failure mode of an unmodeled warehouse, and it is worse than an error, because an error stops someone and a number does not.
🔍 Click to zoom - four failures, four real wrong numbers, four modeling fixes
LiveNames are the prompt3 min

The schema is the largest single thing an agent reads before it writes a query. Every unclear name spends tokens and buys a guess, so naming stops being a style preference and becomes an accuracy lever:

  • net_revenue, not amt2. A name that needs a glossary needs the glossary in the context window.
  • is_declined, not flag3. A boolean named after its meaning can be summed into a numerator. A boolean named after its position cannot.
  • Grain in the view name. v_sales_line says one row is one line. v_sales would not.
  • Different facts, different names. list_price on the product and unit_price on the line - the naming is what stops an agent treating them as the same number (session b2's point, now load-bearing).
The cheapest accuracy win in this whole course A synonym map. The business says "GMV", "topline", "take rate", "basket size"; your model has gross_revenue, net_revenue, commission, aov. Writing that mapping down (it is a dozen lines in semantic/contract.yaml) removes an entire class of guess for zero engineering cost.
Self-studyRemove the footgun, do not prompt around it3 min read

You can tell an agent "never join the two fact tables". It will mostly comply. Or you can build the access layer so that no view exposes both facts, and the join becomes unwritable - which is what sql/40_agent_views.sql does.

Prefer the second every time. The general principle: constraints beat instructions. An instruction is probabilistic and competes with everything else in the context window; a schema that lacks the dangerous path is deterministic. Applied to Bazaar:

  • Read-only credentials on views only. No base tables, no writes. python/agent_text_to_sql.py also enforces this in code, twice - an allowlist regex and a read-only SQLite connection.
  • No fact-to-fact path. The two facts are never both reachable in one view, so the double-count query cannot be expressed.
  • Pre-split measures. approved_amount and declined_amount exist so that no ratio has to be stored, so no average of averages is available to compute.
  • Unknown members instead of NULLs. A NULL dimension key silently drops rows from an inner join; a -1 unknown row keeps them countable (session b6).
Signature lab · 15 min · everyone builds

Ask the agent, with and without the contract ★ real SQL, real execution

Pick a question. The lab shows you the SQL an agent writes for it, runs that SQL against the real Bazaar star in your browser, and tells you whether the number is right. Then flip the contract lever off and watch the same question produce a confident wrong answer. Finish with Score all 20 questions for the whole scorecard.

Start with the contract ON. Read what the agent can see: five views with stated grains, thirteen metric definitions, the join rules. Ask a few questions and check the numbers.

The size of the whole contract Governed read-only views 5 views Metric definitions 13 definitions Golden questions 20 questions Five views, thirteen metric definitions, twenty golden questions - the whole semantic contract.
🔍 Click to zoom - the whole contract is five views, thirteen metrics, twenty questions

Flip the lever off. Same question, same database - now the agent has table names and nothing else. Read the SQL it writes and the wrong number it produces.

Try q02, q04 and q20 specifically. Those are the grain error, the averaged average and the fact-to-fact double count - the three failures that cost the most in production.

Score all 20. 0/20 without, 20/20 with. That delta is what "agent-ready" means, and it is entirely your modeling work.

Real world

Execution match is the honest metric. The scorecard does not compare query text - two correct queries can look nothing alike, and a wrong query can look reasonable. It runs both and compares the result sets, to the cent. That is the same grading approach published text-to-SQL benchmarks use, and it is the only one that cannot be gamed by writing plausible SQL. python/eval_golden_questions.py runs the identical grading locally, against a live model if you want.

Part 2 · what you ship

The four artifacts 6 min live

"Agent-ready" is not a setting. It is four files, each of which is useful to humans on its own - which is the tell that this is real modeling work rather than AI theatre.

agent views schema card the contract golden questions AGENT-READY MODEL 20 of 20 questions answered right PLAIN SCHEMA table names only, agent guesses = an agent you can trust Four small files, each useful to humans alone - that is the tell this is modeling work, not AI theatre.
🔍 Click to zoom - four small files turn a guessing agent into a checked one
ArtifactWhat it isWhat it prevents
sql/40_agent_views.sqlfive flat, self-describing views + a metric table, read-onlythe fact-to-fact join, base-table access, cryptic column names
semantic/schema_card.mdthe prompt: every view, its grain sentence, six hard rulesgrain errors, invented thresholds, rates with no denominator
semantic/contract.yamlowner, SLO, allowed and forbidden joins, metrics, synonyms, known limitationssilent breaking changes, definition drift, undocumented modeling choices
semantic/golden_questions.jsonl20 questions with the grounded SQL and the recorded ungrounded failureregressions - it is a test suite for your model's answerability
LiveMetrics as data, not documentation3 min

The highest-leverage single object in the whole model is a table of metric definitions inside the database. Six columns: name, definition, SQL expression, source view, grain, caveat. Thirteen rows for Bazaar.

SELECT metric_name, sql_expression, grain, caveat
FROM v_metric_definitions
WHERE metric_name IN ('net_revenue', 'orders', 'aov', 'decline_rate');

Note the caveat column - it is the one people leave out and the one that saves the most damage. "Never average an AOV." "Below ~30 attempts it is noise." "Carts, not sessions." Those sentences are the difference between a metric an agent can use and a metric an agent can misuse fluently.

LiveQuery-to-text: narrate from metadata, not from vibes3 min

The return trip matters as much. A fluent sentence around a wrong number is worse than a table, because it removes the reader's last chance to notice. python/agent_query_to_text.py narrates deterministically from the model's own metadata: it classifies each column from its name, demands a denominator for anything that looks like a rate, flags small samples against the 30-observation floor, and appends the contract's caveat for any metric the result names.

SELECT payment_method_label,
       COUNT(*)          AS attempts,
       SUM(is_declined)  AS declines,
       ROUND(100.0 * SUM(is_declined) / COUNT(*), 2) AS decline_rate_pct
FROM v_payment_attempt
GROUP BY payment_method_label
HAVING COUNT(*) >= 30
ORDER BY decline_rate_pct DESC;

Drop the HAVING clause and re-run. A method with a handful of attempts jumps to the top of the ranking. The model cannot stop someone asking for that; the caveat and the visible denominator are what stop them acting on it.

Self-studyThe three scripts, and how to run them4 min read

All three live in python/ and all three run offline, with no API key, so nothing here is a demo you cannot reproduce.

  • agent_text_to_sql.py - the full loop against the live Claude API: schema card as system prompt, one strict run_sql tool, a read-only allowlist guard, results fed back until it answers. --offline uses the recorded golden queries instead; --no-contract withholds the schema card so you can watch it fail.
  • agent_query_to_text.py - narration. --golden q14 narrates a golden question; --llm adds a Claude-written version grounded on the deterministic facts, so the prose cannot contradict the metadata.
  • eval_golden_questions.py - the scorecard. --mode grounded scores 20/20, --mode naive scores 0/20, --mode live calls a real model with or without the contract.
Two guardrails worth copying verbatim Read-only credentials on views only, and always show the generated SQL beside the number. The second one is not a nicety: the SQL is the only part of an agent's answer a human can actually check.
Homework

Try it yourself - this week ◐ 30-40 min total

Source material

Sources covered

Full source map in materials/official-course-map.md. This page covers:

Published text-to-SQL benchmark practice - execution-match gradingThe lab · result sets compared to the cent, not query text
dbt model contracts and the semantic-layer patternPart 2 · contract.yaml, metrics as data, breaking-change policy
Anthropic tool-use documentation - tool definitions, strict schemas, read-only guardrailsSelf-study · the shape of agent_text_to_sql.py
Dehghani, Data Mesh - the contract as the product interfacePart 2 · ownership and SLO fields; the org argument is session a3/a5
Prompt engineering and retrieval techniqueOut of scope by design - learn-prompt-engineering, learn-rag, learn-text-to-sql
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · The agent joins v_sales_line to v_payment_attempt on order_id and reports revenue of 121,003 instead of 111,906. What is the fix?

Constraints beat instructions. An instruction competes with everything else in the context window; a schema that lacks the dangerous path is deterministic. Remove the footgun rather than asking the agent not to pull the trigger.

2 · Why does the lab grade by comparing result sets rather than the SQL text?

Execution match is the honest metric. Text similarity rewards plausible-looking SQL, which is exactly the failure mode you are trying to detect. Run both queries, compare the numbers to the cent.

3 · Which single artifact does the most to prevent an agent inventing its own metric definitions?

Metrics as data beats metrics as documentation: the definition cannot drift out of sync with what is queryable, an agent can look it up mid-query, and a human gets an answer instead of a Slack thread. Thirteen rows did most of the work in this session.

Builder session 9 cheat sheet · pin this

The claimAn AI analyst is only as accurate as the model. 0/20 to 20/20 on the same database, same questions.
Four failuresWrong grain · averaged average · fact-to-fact join · rate with no denominator.
Names are the promptnet_revenue not amt2. Grain in the view name. Different facts, different names.
Constraints beat instructionsDo not prompt around a footgun - build the layer so the bad query cannot be written.
Metrics as dataA table with expression, grain and caveat. The caveat column is the one that saves you.
Exec matchGrade result sets, never query text. Plausible SQL is the failure you are hunting.
Two guardrailsRead-only on views only. Always show the SQL beside the number.
Shipssql/40 · schema_card.md · contract.yaml · golden_questions.jsonl + 3 scripts. Next: b10 capstone.