Why "it looks right" is not good enough
A RAG answer can fail in three completely different places, and the polished paragraph on the screen hides all of them. Maybe the search step pulled the wrong passages, so the model was working from bad source material. Maybe it pulled the right passages but the model ignored them and made something up anyway. Maybe the answer is perfectly true but does not actually address what was asked. Each of these is a different bug with a different owner and a different fix - and if you cannot name which one you are looking at, you cannot ask your team to fix it. The RAG Triad gives you the three names.
The RAG Triad 9 min live
The RAG Triad, from DeepLearning.AI and TruLens, is the cleanest way a non-builder can reason about trust. Three questions, one per corner of a triangle. If all three are strong, the answer is trustworthy. If any one is weak, you know exactly where to look.
LiveCorner 1 · Context Relevance - did we retrieve the right passages?3 min▶
This corner judges the search step, before the model writes a word. When a question comes in, the retriever pulls a handful of passages. Context relevance asks: are those passages actually about the question, or did the search drag in noise?
- What it catches: a retriever that fetches plausible-looking but off-target passages. If the source material is wrong, even a perfect model can only give you a confident wrong answer.
- Plain-English test: read the passages the system pulled. Would a smart colleague say "yes, these are the right paragraphs to answer this"? If not, context relevance is your problem.
- Who owns the fix: the retrieval side - better chunking, a better embedding model, or re-ranking. Not the prompt.
LiveCorner 2 · Groundedness - is the answer supported by the passages?3 min▶
This is the corner leaders care about most, because it is the hallucination check. Groundedness asks: every claim in the answer - can you point to a retrieved passage that backs it up? If a sentence in the answer has no support in the source material, the model invented it.
- What it catches: confident hallucination. The passages were fine, but the model added, embellished, or drifted beyond what they actually say.
- Plain-English test: take each sentence of the answer and try to find the passage that supports it. A sentence with no home is an ungrounded claim.
- Why it is the trust metric: a grounded answer is one you can defend. An ungrounded one is a liability wearing a nice paragraph.
Self-studyCorner 3 · Answer Relevance - does it address the question?2 min read▶
The quietest failure. The answer is grounded, the passages were relevant, and yet the response does not actually answer what was asked - it wanders, over-explains, or answers a nearby question instead. Answer relevance closes the loop back to the user's real intent.
- What it catches: on-topic-but-off-target answers. Common when a question has two parts and the model answers only one, or when it buries the answer in caveats.
- How it is measured under the hood: a neat trick - the system asks the model to generate questions the answer would be a good response to, then measures how close those are to the real question. If the answer implies a different question, relevance is low.
Put the three together and you have a full trust picture: right sources (context relevance) → faithful to them (groundedness) → and it actually answers (answer relevance). Miss any one and the polished paragraph is hiding a fault.
Reading a scorecard without being a data scientist 8 min live
When your team runs an evaluation tool like RAGAS, it hands back a scorecard of numbers. You do not need to compute them - you need to read them. Every metric falls into one of two buckets: did we FIND the information (retrieval), or did we USE it well (generation). That split is the whole diagnostic.
LiveRetrieval metrics vs generation metrics3 min▶
Two buckets. Learn which side each number lives on and the scorecard stops being intimidating.
| Metric | Bucket | In plain English |
|---|---|---|
| Context Recall | Retrieval | Of everything needed to answer, how much did the search actually find? Needs a reference answer to check against. |
| Context Precision | Retrieval | Of what it found, were the good chunks ranked near the top rather than buried? |
| Faithfulness | Generation | Of the claims in the answer, how many are actually supported by the passages? The hallucination check. |
| Answer Relevancy | Generation | Does the answer stay on the question, without wandering or padding? |
Retrieval metrics judge the finding. Generation metrics judge the using. A single blended "accuracy" number tells you nothing actionable - the split is what makes a scorecard useful to a decision-maker.
LiveThe diagnostic rule that tells you who to call3 min▶
This is the one rule to remember from the whole session. When a RAG system is answering badly, do not guess - read which bucket is low:
- Low faithfulness or answer relevancy → a generation problem. The passages were fine; the model mishandled them. The fix is prompt design, a refusal instruction, or a stronger model - not the retriever.
- Low context recall → a retriever problem. The search never found the information, so the model had no chance. The fix is chunking, the embedding model, or how many passages you pull - not the prompt.
- Check end-to-end first, then component-wise. Look at whether the final answer is good, then drop into the corners to find where it broke. This is the sequence the DeepLearning.AI course teaches.
The team that rebuilt the wrong half. A support-bot team (anonymized) saw bad answers and spent six weeks re-engineering their retriever. When they finally ran a proper scorecard, context recall was 0.9 and faithfulness was 0.5 - it had been a generation problem all along. A one-line "answer only from the passages, and say you don't know if they don't cover it" prompt fixed in an afternoon what six weeks of retriever work had not.
Self-studyWhy you demand a golden reference set2 min read▶
Some of these metrics - context recall and answer correctness especially - cannot be measured against thin air. To know whether the retriever found everything needed, or whether the answer is factually right, you need a trusted answer to compare against. That trusted set of question-and-ideal-answer pairs is the golden set (or reference set), and it is the single most valuable asset in a RAG evaluation.
- Without it, scores are self-graded - the model checking its own homework - and context recall and answer correctness cannot be computed at all.
- With it, you get real numbers you can track over time, and you can prove an upgrade actually improved things rather than just feeling different.
Build your acceptance scorecard ★ 12 min · pen and paper
Before you trust a RAG system in production, decide - now, in calm daylight - the five numbers you will demand to see. No code, just judgment.
Pick five metrics you would require before signing off. A strong default: context recall, context precision, faithfulness, answer relevancy, and one business threshold of your own (for example, refusal rate on out-of-scope questions).
For each, write the bucket it lives in (retrieval or generation) and, in one line, what failure it catches. If you cannot name the failure, drop the metric - it is decoration.
Set a pass threshold for each. You do not need the "right" number - you need a line drawn before the demo dazzles you. Faithfulness especially: what is the lowest you would accept?
Add one line at the top: "scored against golden set of ___ examples." If that blank cannot be filled, the whole scorecard is provisional - and that is your finding.
Judge a real answer on the Triad ★ 8 min · scoring drill
Score one real answer by hand and the Triad becomes instinct. Here is a worked case; run it, then bring one of your own.
The question: "How long do I have to return an item?" The retrieved passage: "Refunds policy: items may be returned within 30 days of delivery for a full refund, provided they are unused." The answer: "You have 30 days from delivery to return an item for a full refund, as long as it is unused. Note that gift cards are non-refundable."
Context Relevance: is the passage the right source for this question? (Yes - it is the refunds policy, directly on point.)
Groundedness: is every claim in the answer supported by the passage? The 30-days and unused parts are. But "gift cards are non-refundable" appears nowhere in the passage - that is an ungrounded claim, even if it happens to be true. Groundedness fails here.
Answer Relevance: does it address the question asked? (Yes - it answers the return window directly.) Your verdict: strong on two corners, fails groundedness because of the invented gift-card line. That is exactly the kind of confident, plausible addition the metric exists to catch.
Before session a5 ◐ 30 min total
- Finish your five-metric acceptance scorecard and keep it - session a5 turns the refusal and citation lines into governance controls you can insist on.
- Run the Triad scoring drill on one real answer from a system you actually use (an internal assistant, a vendor demo). Note which corner is weakest.
- Find out whether your team has a golden reference set, how big it is, and who maintains it. If the answer is "no", that is your most important finding of the week.
- Optional: skim the builder track's evaluation session to see these same metrics computed in a tool - you do not need the code, but the numbers will feel more concrete.
- When we say an answer is "good", are we measuring groundedness, or just eyeballing it - and can you show me a faithfulness score?
- Do we have a golden reference set? How many examples, and who keeps it current?
- When answers are bad, do we check whether it is a retrieval problem or a generation problem before we start fixing?
- What are our current context recall and faithfulness numbers, and what thresholds must they clear before production?
- How does our system score on answer relevance - does it ever answer a nearby question instead of the one asked?
Official sources covered
The leader track teaches from published course curricula and evaluation frameworks - no vendor marketing. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · The three corners of the RAG Triad are...
Context relevance judges the retrieved passages, groundedness judges whether the answer is faithful to them, and answer relevance judges whether it addresses the question.
2 · Your scorecard shows context recall 0.9 but faithfulness 0.5. The bug is...
Strong recall means retrieval found the info. Low faithfulness means the model hallucinated beyond the passages - a generation problem. Do not touch the retriever.
3 · Why do you insist on a golden reference set before believing an "accuracy" number?
Context recall and answer correctness are measured against reference answers. No golden set means no trustworthy accuracy, only self-grading.