learn-ai-evals-with-phoebe / Leader session 2 of 6
Learn Evals with Phoebe · Leader track · Session 2 of 6

What "good" actually means before you can measure it

Session 1 made the case that you have to measure. This one gives you the words for what you are measuring. "Is it good?" is not a single question - for the most common kind of AI system, the one that looks things up before it answers, quality splits into three distinct pieces, and each catches a different way the system can fail. Learn this vocabulary and you can sit in any AI review and say exactly where quality broke, instead of waving at "the answer was bad".

🟢 easy Leaders: C-level · managers · curious minds No code, ever 45 min
0-3 · Welcome 3-20 · Concepts 20-40 · Exercises 40-45 · Q&A
Part 0

Why "good" needs three words, not one

Most business AI today is a lookup system: given a question, it first fetches relevant material - policy documents, past tickets, product specs - and then writes an answer from what it found. This is called retrieval-augmented generation, RAG for short, and you do not need the acronym, just the shape: find, then answer. The reason it matters for leaders is simple. When the answer is wrong, there are two very different places it could have gone wrong, and if you cannot name which one, you cannot fix it or even ask the right question. This hour gives you the naming.

Live - presented in session Self-study - read after class ★ Try it now prompt Sources covered at the end
★ What you walk out with today The RAG Triad - three named checks that together are strong evidence an answer is not hallucinated - the split between retrieval quality and answer quality, a one-line diagnostic for which half broke, and five questions to put to your data team this week.
Part 1 · covers the three checks of quality

The RAG Triad 9 min live

For a find-then-answer system, quality is not one thing - it is a triangle of three checks, and each corner catches a different failure. When all three pass together, you have strong evidence the answer is grounded in real material rather than invented. This is the single most useful mental model in the whole course.

Question what the user asked Context chunks we retrieved Answer what we replied Context Relevance Answer Relevance Groundedness right chunks found? answers the question? answer supported by context? All three corners pass = strong evidence the answer is not hallucinated.
🔍 Click to zoom - the RAG Triad: three checks that together catch hallucination
LiveContext Relevance - did we even find the right material?3 min

Context Relevance asks a question about the lookup step, before a single word of the answer is written: of the material the system pulled up to answer with, how much of it actually relates to what the user asked?

  • Plain English: "Did we fetch the right pages?" If someone asks about refund timing and the system retrieves the shipping policy, context relevance is low - the answer was doomed before it started.
  • The failure it catches: the system confidently answering from the wrong source material. A perfect writer cannot save a wrong document.
LiveGroundedness - is every claim actually backed by the material?3 min

Groundedness is the anti-hallucination check. It asks: is each statement in the answer actually supported by the material that was retrieved, or did the model add things from thin air?

  • How it is scored, in plain terms: break the answer into its individual claims, check each claim against the retrieved context, and score the fraction that are supported. Nine of ten claims backed by the documents is a groundedness of 0.9. This "break into claims, check each" method is worth remembering - it is exactly how the machine does it too.
  • The failure it catches: the fluent invention - a plausible detail, number, or policy that appears nowhere in the source. This is the failure that quietly misinforms customers.
LiveAnswer Relevance - did we actually answer the question asked?3 min

Answer Relevance closes the loop back to the user. An answer can be built from the right documents and be perfectly grounded in them, and still miss - because it answered a slightly different question, or buried the point, or trailed off into a related topic.

  • Plain English: "Was this a reply to what they actually asked?" Correct and grounded but off-target is still a bad answer.
  • The failure it catches: the technically-true non-answer - accurate, sourced, and unhelpful.
Why three, not one Each corner guards a different door: right material in, honest use of that material, on-target reply out. A single "is it good?" score hides which door was left open. The Triad names it.
Part 2 · covers where quality breaks

Retrieval quality vs answer quality 8 min live

Step back from the three corners and there are really two halves to a find-then-answer system: the half that finds the information, and the half that writes with it. Measuring both is not optional - because either half can pass while the other fails, and the two failures need completely different fixes.

RETRIEVAL metrics Did we find the right info? Context Relevance lives here. Wrong docs = doomed answer. GENERATION metrics Did we use it well? Groundedness + Answer Relevance live here. find then answer Good answer needs BOTH. Bad retrieval dooms a great writer; great retrieval cannot save a bad one. The diagnostic: low groundedness points at generation; missing info points at retrieval.
🔍 Click to zoom - two halves of quality: finding the info vs using it well
LiveWhy you must measure both halves3 min

It is tempting to measure just the final answer - that is what the user sees, after all. But a single answer score cannot tell you which half to fix, and worse, it can lull you.

  • Great retrieval + bad answer = still bad. The system found the perfect document and then wrote a muddled, off-target reply. The retrieval score looks great; the customer is unhappy.
  • Bad retrieval dooms even a great generator. The best writer in the world, handed the wrong documents, produces a confident wrong answer. No amount of writing skill recovers from being fed the wrong facts.
  • Two halves, two teams, two fixes. Fixing retrieval means better search and indexing. Fixing generation means better prompting and instructions. If you only know "the answer was bad", you do not know which team to send it to.
LiveThe one-line diagnostic every leader should keep3 min

Here is the takeaway to write on a sticky note. When an answer is bad, the Triad tells you where to look:

  • Low groundedness - the answer contains claims the documents do not support - means the generation half is the problem. The right info may have been there; the model invented or drifted. Fix the writing side.
  • Missing information - the right material was never retrieved, so low context relevance - means the retrieval half is the problem. Fix the search side; no prompt tweak will conjure a document that was never found.
Real world

The two-week wild goose chase. A team (anonymized) spent two weeks rewriting prompts to fix a bot that kept giving incomplete answers. Quality barely moved. When they finally looked at the retrieval half, they found the search was only ever pulling the first page of each policy - the answers were incomplete because the material was incomplete. The prompt was never the problem. The Triad would have pointed them at retrieval on day one.

Ask the two-part question Never accept "the answer was bad" as a diagnosis. Ask: "Was the right information found, and if so, was it used faithfully?" That single sentence routes the fix to the correct half.
Self-studyWhere the Triad comes from, and its cousins2 min read

The RAG Triad framing was popularized by DeepLearning.AI's short course on building and evaluating advanced RAG, taught with the open-source TruLens tool, which scores Context Relevance, Groundedness, and Answer Relevance on real traffic. A parallel open-source framework, RAGAS, offers a similar family of metrics under slightly different names - faithfulness for groundedness, and so on. You do not need to pick a tool today. You need the concept: three checks, two halves, and a diagnostic that tells you which half broke. The builder track puts real numbers on all three.

Exercise 1 of 2

Score a real answer on the Triad ★ 12 min · pen and paper

You do not need code to grade the Triad - just judgment. Here is one question, the material the system retrieved, and the answer it gave. Rate each corner pass or fail, and say which half broke.

Real world

Question: "How many vacation days do first-year employees get?"
Retrieved context: "The employee parking policy allows two reserved spaces per department. Badge access is granted on the first day."
Answer given: "First-year employees receive 15 vacation days per year."

Context Relevance - pass or fail? Does the retrieved material relate to the vacation question? Decide, and note what that tells you about the lookup step.

Groundedness - pass or fail? Is the "15 days" claim supported by the retrieved context? Check the answer's claim against the material in front of you.

Answer Relevance - pass or fail? Taken alone, does the answer address the question that was asked?

Which half broke? You should find context relevance failed (wrong material), groundedness failed (the number is invented - it appears nowhere), and answer relevance passed on its face. Wrong material plus a confident invented number - the failure started in retrieval and the model papered over it. This is the classic hallucination pattern.

★ Your Triad-scoring partner (paste into any chat AI)You are teaching me to grade the RAG Triad. I will give you a question, the retrieved context, and an answer. Score each of the three - Context Relevance (does the context relate to the question?), Groundedness (is every claim in the answer supported by the context?), and Answer Relevance (does the answer address the question?) - as pass or fail with one sentence each. Then tell me whether the failure, if any, is a retrieval problem or a generation problem. Here is the first case: [paste question, context, answer]
Exercise 2 of 2

Sort the complaints ★ 8 min · your next AI review

Real quality complaints arrive as vague grumbles. The skill is sorting each into "retrieval problem" or "answer problem" - because that routes it to the right fix. Sort these six.

"It made up a discount that does not exist." → Retrieval or answer? (The claim is unsupported - that is a groundedness failure, an answer problem.)

"It said it could not find anything about our return policy, but we have a whole page on it." → (The right document was never fetched - a retrieval problem.)

"Everything it said was true, but it never actually answered my question." → (Low answer relevance - an answer problem.)

"It answered about the wrong product entirely." → (Wrong material pulled up - a retrieval problem.)

"The facts were right but it added a warranty term that is not in our policy." → (An invented claim on top of good material - a groundedness failure, an answer problem.)

"It only ever quotes the summary and misses the detailed clauses." → (Incomplete material retrieved - a retrieval problem.)

The pattern to notice Invented, off-target, or misused facts point at the answer half. Missing, wrong, or incomplete material points at the retrieval half. Sorting complaints this way, in the meeting, is the fastest way to route a fix to the team that can actually make it.
Homework

Before session a3 ◐ 30 min total

★ Questions to ask your data team this week
  1. For our main AI system, do we measure retrieval quality and answer quality separately, or only the final answer?
  2. Do we track something like groundedness - the fraction of the answer's claims actually supported by the retrieved material?
  3. When an answer is wrong, can we tell whether the right information was found but misused, or never found at all?
  4. Do we check that the retrieved documents were even relevant to the question, before judging the answer?
  5. Would passing all three - context relevance, groundedness, answer relevance - give us confidence the answer is not hallucinated?
Source material

Official sources covered

The leader track teaches from published engineering guidance and course curricula - no vendor marketing. This page covers:

DeepLearning.AI · Building & Evaluating Advanced RAG (RAG Triad via TruLens)Part 1-2 · Context Relevance, Groundedness, Answer Relevance · all three pass = low hallucination
RAGAS · RAG evaluation metricsPart 2 · faithfulness = groundedness · a parallel metric family
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · The three corners of the RAG Triad are...

Context Relevance (right material found?), Groundedness (answer supported by that material?), and Answer Relevance (does it address the question?). All three passing is strong evidence the answer is not hallucinated.

2 · Groundedness is scored, in plain terms, by...

Break the answer into claims, check each against the context, and the score is the fraction supported. Nine of ten claims backed = groundedness of 0.9. It is the anti-hallucination check.

3 · An answer is wrong and its groundedness is low. The diagnostic says...

Low groundedness means claims are not supported by the retrieved context - the answer half misused or invented. Missing context, by contrast, points at the retrieval half.

Leader session 2 cheat sheet · pin this

Find, then answerMost business AI looks material up first, then writes from it. Quality can break in either step.
The RAG TriadContext Relevance, Groundedness, Answer Relevance - three checks, one per failure mode.
Context RelevanceDid we fetch the right material? A perfect writer cannot save a wrong document.
GroundednessIs every claim backed by the material? Break into claims, check each, score the fraction supported.
Answer RelevanceDid we reply to what was actually asked? Correct-but-off-target is still bad.
Two halvesRetrieval (found the info?) vs generation (used it well?). Measure both, not just the final answer.
The diagnosticLow groundedness → generation problem. Missing info → retrieval problem. Route the fix accordingly.
Three green = trustAll three corners passing is strong evidence the answer is grounded, not hallucinated.