The claim, and who owns the budget
The claim of this session is narrow and testable: an AI analyst is only as accurate as the model underneath it, and closing that gap is modeling work rather than prompt work. Which means most of the spend on "AI for analytics" belongs in the data team, doing things that are useful even if the agent is never switched on.
The numbers on this page come from the course's own marketplace dataset - the same model the builder track designs - so the arithmetic is checkable rather than rhetorical. Builder session b9 builds the artifacts and runs the measurement live; this session decides what to pay for and how to tell whether it worked.
Four failure modes, four real wrong numbers 8 min live
None of these is a language problem. Each is something the model knew and never published - and each produces a query that runs cleanly and returns a number that is wrong.
| Failure | What a leader sees | What actually fixes it |
|---|---|---|
| Wrong grain | the agent's volume numbers do not match the finance pack | publish "one row is one ___" for every table the agent can reach |
| Averaged average | a ratio that disagrees with the same ratio computed from totals | a metric definition holding the exact expression, so it is looked up rather than invented |
| Joining two facts | revenue that reads high and cannot be reconciled to cash | a view layer that does not expose both tables together, so the query is unwritable |
| Rate with no base | confident rankings that reverse next week | metrics that ship numerator and denominator, and a stated minimum sample |
LiveWhy "constraints beat instructions" is the budget argument4 min▶
You can tell an agent not to join those two tables. It will mostly comply. Or you can build the access layer so the dangerous join does not exist, and compliance becomes structural. The second costs data-team effort once; the first costs vigilance forever, and vigilance is not a line item you can staff.
- An instruction is probabilistic and competes with everything else the model is reading. A missing table is deterministic.
- This is why the spend is mostly modeling. Prompt work is cheap, visible and does not accumulate. Model work is less visible and compounds - the same views, definitions and tests make your human analysts faster too.
- And it gives you a test for any proposal: does this remove the possibility of the error, or does it ask something to remember not to make it? Fund the first.
Self-studyThe measurement, and why it is honest3 min read▶
The builder track measures this rather than asserting it. Twenty standard business questions, the same database, the same model, the same AI analyst - once with the semantic contract published and once without. Grading compares the result sets, to the cent, not the text of the query.
- Without the contract: 0 of 20 correct. Not because the questions are hard - because grain, definitions and permitted joins were never published, so the agent guessed and guessed plausibly.
- With it: 20 of 20. The contract is 13 metric definitions, five governed read-only views and a short schema card. It is small, and it is the whole difference.
- Why result-set grading matters: comparing query text rewards SQL that looks reasonable, which is exactly the failure you are trying to detect. Two correct queries can look nothing alike.
That is also the shape of the honest pilot: pick twenty questions your business actually asks, agree the right answer with a human first, then measure.
What to fund, in what order 7 min live
The order matters more than the total. Each rung makes the next one cheaper, and every rung pays for itself in human productivity before an agent touches it.
| # | Fund this | Why here in the order |
|---|---|---|
| 1 | published grains: one sentence per table, in the catalog | cheapest, fastest, and it removes the single most common error. Nothing later works without it |
| 2 | metric definitions as data: name, expression, grain, owner, caveat | ends the definition argument for humans and machines at once; the caveat field is the part that prevents damage |
| 3 | a governed read-only view layer | makes the dangerous joins unwritable and gives you an access boundary you needed anyway |
| 4 | a golden question set with agreed answers | your regression test. Without it you cannot tell improvement from luck, or detect the day a change breaks accuracy |
| 5 | the agent itself | last, and the smallest line. By now it is a configuration exercise rather than a research project |
LiveHow to judge a vendor claim in one question4 min▶
Ask for execution-match accuracy on your questions, against your model. Twenty questions your business actually asks, answers agreed by a human first, results compared to the cent. Then listen to how the answer arrives:
- A benchmark score instead of your questions tells you the tool works on somebody else's schema. Your schema is the variable under test.
- "It learns your schema automatically" is a claim about reading column names. It cannot recover a grain nobody wrote down or a metric nobody defined - that information is not in the schema to be read.
- A refusal to be measured on failures is the tell. The number that matters is not how many it got right; it is whether it ever answered confidently and wrongly, because that is the mode that damages trust.
- A good vendor will ask for your metric definitions in the first meeting. That is a sign they know where accuracy comes from - and if you do not have them, you have just learned what to fund first.
Self-studyWho owns what, and the 30-day plan3 min read▶
Ownership: the data team owns grains, definitions and the view layer. The business owns the golden questions and the agreed answers - and that is not a formality, because a question nobody in the business will vouch for cannot be used to measure anything. The AI or platform team owns the tool.
The 30 days:
- Week 1: collect twenty questions your business asks repeatedly. Agree the correct answer for each with a named human. Expect to discover two or three that nobody can agree on - those are findings, not delays.
- Week 2: write the grain sentence for every table those questions touch. Publish where consumers meet it.
- Week 3: define the metrics those questions use, as data, with the caveat field filled in honestly.
- Week 4: measure. Run whatever tool you already have against the twenty, grade by result, and report the number. That number, not a vendor's, is your baseline.
None of that is an AI project. All of it is what makes an AI project work, and all of it survives if the AI project is cancelled.
Write your first five golden questions ◆ run this in your own meeting
This is the single highest-value hour a leadership team can spend on AI readiness, and it needs no technology at all.
Each person names one question they ask of the data at least monthly. Real phrasing, not a cleaned-up version.
For each, agree the correct answer for last month - the number, and the definition behind it. Take the disagreements seriously; they are the actual output of this exercise.
Mark each question with whether the room could state the grain and the metric definition it depends on.
Hand the marked list to the data team as the readiness backlog, in that order. You have just written a specification and a test suite in fifteen minutes.
LiveWhat a good answer sounds like, and what to listen for4 min▶
Good: "Which merchant lost the most revenue last fortnight? Revenue means settled net revenue after discounts, at order-line grain, excluding failed payments. Last fortnight the answer was one merchant down 85%, and here is the number."
Listen for these failure modes:
- "It depends what you mean by revenue." Stop and write down the variants. You have found why the agent will be wrong, and also why two teams disagree in meetings today.
- "Just ask the tool and see what it says." Then you have no way to grade it. The agreed answer must exist before the tool runs, or the exercise measures nothing.
- Only vague questions get offered. "How is the business doing" cannot be graded. Push for questions with a number as the answer - that constraint is what makes the list a test.
The pilot that was never going to work
The pattern is consistent. A team buys a natural-language analytics tool, the demo on sample data is excellent, and week two produces order counts that disagree with finance. Everyone concludes the AI is not ready. What was not ready was the model: nobody had written down that one row of the sales table is a shipment, so the tool counted shipments and called them orders - exactly as a new analyst would have. The organisation that fixes the grain and the definitions gets a working agent from the same tool a quarter later, and gets faster human analysts on the way there. The one that concludes "AI does not work for us" gets neither.
Questions to ask your data team 5 min live
- "If we handed our data to an AI analyst tomorrow, what would it get wrong, specifically?" A specific list means the team understands their own model. A confident "nothing" means nobody has tested it.
- "Where would it read what one row of our sales table means?" If the answer is a person, the agent cannot read it - and neither can a new hire.
- "Which of our metrics have a written definition with a caveat?" The caveat is the part that stops fluent misuse. Ask to see three.
- "Which two of our tables look joinable but are not?" Every warehouse has a pair. A team that can name theirs has thought about this; a team that cannot has not looked.
- "What is our execution-match accuracy on twenty of our own questions?" If the answer is "we have not measured", that is week 4 of the plan, and it is the cheapest de-risking available.
- "Does the tool show the query beside the number?" Non-negotiable. It is the only checkable part of the answer.
- "What of this plan would we still want if we never deployed an agent?" The honest answer is nearly all of it, and that is the point.
Your first 30 days ◐ 40 min of conversations
- Collect the twenty questions. Agree the answers with named humans, and keep the list of ones nobody could agree on - it is your definition backlog.
- Ask for grain sentences on the five tables those questions touch. One sentence each, published where a consumer meets it.
- Pick your three most-used metrics and require a caveat for each. If a metric has no caveat, ask whether anyone has looked hard enough.
- Before any tool procurement, send the vendor five of your questions and ask for execution-match accuracy against your model, plus the failure list.
- Book the measurement for week 4 now, and put the number in the same pack you report other baselines in. A readiness number that nobody sees does not change behaviour.
Sources covered
Full source map in materials/official-course-map.md. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · An AI analyst pilot returns order counts that disagree with the finance pack. What is the most likely cause?
This is the most common failure in practice, and it is a modeling failure. A schema exposes column names, not the sentence "one row is one shipment" - so if nobody wrote that down, no amount of prompt or model work recovers it.
2 · Why is "fund the agent last" the right order?
The rungs make the agent a configuration exercise rather than a research project, and they pay for themselves in human analyst productivity first. That is also the sentence that gets the plan funded: cancel the AI programme and you still want all of this.
3 · A vendor offers a benchmark accuracy score. What should you ask for instead?
A benchmark says the tool works on somebody else's schema; yours is the variable under test. And insist on the failures, not just the wins - a tool that answers confidently and wrongly does more damage than one that declines, because a number does not stop anybody.