Where this session sits
Sessions 1-4 made you personally effective with AI: delegate well, ask well, think with it, judge what comes back. From here the track turns outward. Most of the AI value in your organisation will not come from your own chats - it will come from ideas you sponsor and a data team builds. This session teaches you to hand over an idea the way a good product manager would: specified, measurable, and testable. Session 6 adds the risk and strategy layer.
Specify like a PM 8 min live
Anthropic's own guidance to builders starts with a funnel, and the first stage is not a prompt. It is a definition of success: decide what good looks like, in measurable terms, before anyone writes a single line. Teams that skip this stage do not know when they are done - or worse, they think they are done when they are not.
LiveDana's idea meets the data team3 min▶
Four sessions in, Dana is dangerous in the best way. Her complaint-summary prompt saves her an hour a week, so she walks into the data team's Monday stand-up with the obvious next step: "Let's use AI for complaint handling. Can you build that?"
The team lead does not open a laptop. She asks three questions:
- "When it drafts a reply to an angry customer, what makes the draft good? Good enough to send?"
- "On how many real complaints should we test it before a customer ever sees it?"
- "When it gets one wrong - and it will - who catches it?"
Dana cannot answer any of them yet. That is not a failure; it is the actual work. "AI for complaint handling" is a wish. "Drafts that agents approve without edits 80% of the time, tested on 100 real tickets, agent always reviews before send" is a specification. This session is about learning to bring the second thing.
The pattern in every company: AI projects rarely die because the model was too weak. They die because nobody defined success, so the pilot drifted for six months and was quietly cancelled. The fix costs one meeting - held at the start.
LiveWhat good success criteria look like3 min▶
Anthropic's documentation asks builders for criteria that are specific, measurable, and achievable - and it names the dimensions worth measuring. None of them require code to understand:
| Dimension | Plain-language question | Example bar |
|---|---|---|
| Task fidelity | Does it actually do the job right? | "Correct complaint category on 95 of 100 tickets" |
| Consistency | Does the same input get the same quality every time? | "Same ticket, three runs, same category all three" |
| Tone | Does it sound like us? | "9 in 10 drafts approved by a senior agent without tone edits" |
| Latency | Is it fast enough for the real workflow? | "Draft ready in under 10 seconds" |
| Cost | What does each answer cost at real volume? | "Under 4p per ticket at 3,000 tickets a week" |
Notice the shape: every bar is a number a non-technical person can check. "Make it good" is not a criterion. "80% approved without edits" is - and it turns every later argument into a measurement.
Self-studyWeak wish vs real specification - side by side2 min read▶
| The wish (what teams usually get) | The specification (what they can build) |
|---|---|
| "Use AI to handle complaints" | "Draft replies to written complaints for agent review" |
| "It should be accurate" | "Right complaint category on 95% of a 100-ticket test set" |
| "It should sound professional" | "Matches these 5 example replies our best agent wrote" |
| "It should not make things up" | "Never promises a refund; escalates to a human instead" |
| "Soon, ideally" | "Pilot on one store's tickets by end of quarter, then review" |
The right column is the same skill you learned in Session 1 - the anatomy of a good ask - applied at project scale. Goal, context, constraints, format, example. A project brief is a prompt for your organisation.
Evals: how you know it works 8 min live
An eval is the least glamorous and most important word in applied AI. In business language: an eval is a set of real examples plus a way to grade the answers. It is the acceptance test of an AI feature - and whether your team has one is the single fastest tell of whether they will ship a product or just a demo.
LiveVibes vs numbers4 min▶
Every AI demo you will ever see was graded one of two ways. Learn to hear the difference:
| Vibes | Numbers |
|---|---|
| "We tried it on a few examples - looks good to me" | "Graded on 100 real tickets from last quarter" |
| Tested on the happy cases someone picked | Tested on real cases, including the ugly ones |
| "It feels much better after the last change" | "The change moved approval from 71% to 84%" |
| Quality argument = whoever is most senior wins | Quality argument = the score wins |
| No way to notice quiet degradation later | Re-run the eval any time; regressions show up as a number |
The grading itself can be simple: humans scoring answers, exact-match checks, or another AI grading against a rubric. Your team will pick the mechanics - the builder track spends two whole sessions there. What leaders must insist on is that grading exists, uses real examples, and produces a number that can be tracked over time.
Self-studyWhy demos are easy and products are hard3 min read▶
A demo needs to work once, on an example the presenter chose. A product needs to work on Tuesday's weird ticket, the complaint written in all caps, the one that mentions a lawyer. That gap is where AI projects go to die, and evals are the only honest bridge across it:
- Evals force realism. Building the test set means reading 100 real tickets - which surfaces the messy cases the idea never considered.
- Evals de-risk change. New model version? Prompt tweak? Re-run the eval, compare numbers. Without one, every change is a leap of faith. (Session 6 turns this into a governance habit.)
- Evals end quality theatre. "I do not like the tone" becomes a rubric item with a score, not a taste war between departments.
The wider industry teaches the same lesson - courses on evaluating AI agents now grade not just final answers but each step an AI takes along the way. Your team will meet that depth in the builder track. The leader-level takeaway is one line: no eval, no launch.
The five questions to ask your team 6 min live
You do not need to review prompts or read code. You need five questions - the same five, every time a team proposes or demos an AI feature. They are answerable in plain language, and together they cover context, quality, failure, data, and money.
LiveThe five, and why each matters6 min▶
| # | The question | Why it matters |
|---|---|---|
| 1 | What context does it see - and how does that stay current? | From Session 1: context beats phrasing. An assistant answering from last year's price list fails politely and constantly. "Kept current how, by whom?" is a budget question wearing a technical costume. |
| 2 | How do we measure it is good - on how many real examples? | The eval question. A number means a product; a story means a demo. Also sets the launch bar everyone signs up to in advance. |
| 3 | What happens when it is wrong - who catches it? | It will be wrong sometimes. The design question is whether a human reviews before impact, or the error reaches a customer. "Agent approves each draft" and "auto-send" are different projects with different risk. |
| 4 | What data goes in - and are we allowed? | Customer complaints contain names, orders, sometimes health or payment details. Which system does it flow to, under which agreement? Cheaper to answer now than in a regulator's letter. |
| 5 | What does it cost per answer at scale? | Pennies per answer times 3,000 tickets a week times 52 weeks is a real line item. Latency and cost per answer decide whether the pilot economics survive contact with real volume. |
Back at Meridian: Dana brings the five questions to the complaint-handling kickoff. Question 3 changes the project on the spot - the team had assumed auto-send; the room chooses agent-review-first. Question 5 kills a rival idea the same week: a chatbot whose per-answer cost at real volume was triple the support line it replaced.
Self-studyHow to ask them without micromanaging2 min read▶
These questions are guardrails, not steering wheels. Three habits keep them useful:
- Ask for the answers, not the mechanism. "How do we measure quality?" is your question. Which grading tool they use is theirs. Judge the bar, not the plumbing.
- Accept "not yet" as a good answer. "We have not built the eval set yet - that is next sprint" is a healthy team being honest. "We do not need one" is the answer that should worry you.
- Ask them of vendors too. An AI vendor who cannot answer question 2 about their own product is selling you a demo. The five questions work unchanged in procurement.
Work it live: your one-page AI project brief ★ 15 min · work it live
Take the AI idea you brought - the one you have mentioned in meetings but never written down. In fifteen minutes you will turn it into a one-page brief a data team could actually pick up. The trick: do not write the brief yourself. Make the AI interview you first, like the data team lead did to Dana.
Pick one real idea. The one you would actually sponsor. If you have three, pick the one closest to customers - the stakes make the answers honest.
Run the interview prompt. Answer the questions one at a time, out loud if you are in the room. Notice which question you cannot answer - that is your homework, not a reason to stop.
Check the criteria are numbers. Scan the drafted brief's success criteria. Every one should be checkable by a non-technical person: a percentage, a count, a time, a cost.
Run the five questions against it. Context, measurement, failure, data, cost. The brief should answer all five. If one is missing, add it to the open questions.
Golden-rule it. Show the page to the person next to you, cold. If they can say what is being built, how success is measured, and who catches errors - it is ready for a real data team.
Three things before Session 6 ◐ 25 min total
- Finish your one-page brief - close the open questions the interview flagged, then share it with someone on your data or tech team and ask one thing: "What is missing before you could scope this?"
- Ask the eval question once for real. Find one AI feature in flight or in a vendor pitch this week and ask: "How many real examples has this been graded on?" Bring the answer - number or story - to Session 6.
- Spot the missing owner. Pick one AI thing your organisation already uses and find out who owns its quality after launch. If the answer is "nobody, it just runs", you have found Session 6's subject in the wild.
What this session covers
Leader-level coverage of the sources below - working ideas in plain language, honestly flagged where the depth lives in the builder track. Official certificates and videos stay with the vendors; links are in the course map.
Three questions before you go 🎯 ◐ 90 seconds
1 · According to Anthropic's own guidance, the first step of an AI project is...
The funnel runs criteria, then evals, then the prompt. Most teams start at the prompt - which is why most pilots cannot say when they are done. Steps 1 and 2 are leadership work.
2 · In business language, an eval is...
Real examples, graded, giving a score you can track over time. It is the acceptance test of AI: re-run it after every change, and quality arguments become measurements.
3 · A team demos an AI feature and says "we tried it on a few examples - looks good to us". The best leader response is...
Vibes vs numbers. You do not need to read prompts - you need to know the quality bar exists, uses real examples, and produces a trackable number. Teams that skip evals ship demos, not products.