The risks that come with reaching into your data
A plain chatbot can only embarrass you. A RAG system can leak a document, quote a policy that was rescinded last quarter, or answer a question it should have declined - because it is wired directly into your knowledge base. That is not a reason to avoid RAG; it is a reason to govern it. The good news for leaders: the controls are surprisingly concrete and mostly non-technical to demand, even if they are technical to build. You can insist the system cites its sources, refuses when it should, and only retrieves what the person asking is allowed to see. This session turns those instincts into a checklist.
The five risks unique to retrieval 9 min live
These five are not the usual "AI might be biased" concerns. They exist because the system retrieves from your real documents. Every one of them has bitten a real team, and every one has a named control in Part 2.
LiveRisks 1 and 2 · sensitive data and who can see what3 min▶
The two that keep security teams up at night, because they are about data getting where it should not.
- PII in embeddings. To make documents searchable, RAG turns them into embeddings stored in a vector database. If those documents contain personal or sensitive data, that data now lives in a second place - the index - which your deletion and retention rules may not cover. "We deleted the file" does not mean you deleted its embedding.
- Access control at retrieval. This is the big one. A naive RAG system searches all documents for every user. So a junior employee can ask a question and get an answer grounded in the board's confidential minutes - because retrieval never checked whether they were allowed to see them. Retrieval must filter by who is asking, not just by what is relevant.
LiveRisks 3 and 5 · stale answers and confident bluffing3 min▶
Two failures of freshness and honesty - the ones users actually notice.
- Staleness. RAG is only as current as the documents in its index. If a policy changed but the old version is still in the index, the system will confidently retrieve and quote the outdated one. Freshness is a pipeline discipline: what is indexed, how often it refreshes, and how the obsolete version is removed.
- Confident bluffing when context is thin. When retrieval returns weak or partial passages, a poorly governed model does not say "I'm not sure" - it fills the gap with a plausible guess and presents it in the same authoritative tone as a grounded answer. The thinner the context, the more dangerous the confidence.
The number that was right last year. An internal finance assistant (anonymized) kept quoting a spending-approval threshold that had been lowered months earlier - the superseded memo was still in the index alongside the new one, and retrieval sometimes surfaced the old one. Nobody had lied; the pipeline had simply never retired the stale document. The fix was freshness governance, not a smarter model.
Self-studyRisk 4 · vendor lock-in through your embeddings2 min read▶
The quiet, strategic one. When you build a RAG system, you turn your whole corpus into embeddings using a specific model. Those embeddings are model-specific - the numbers only mean something in the world of the model that produced them. You cannot mix embeddings from one model with a query from another.
- Why it is lock-in: switching embedding models - to a cheaper one, a better one, or a different vendor - means re-embedding your entire corpus. For a large knowledge base that is a real cost in compute, time, and coordination.
- The leadership move: know the switching cost before you commit, and treat "how hard is it to change embedding models later" as a procurement question, not an afterthought. It rarely blocks a decision, but it should never be a surprise.
This is the one risk with no clever control - only clear-eyed awareness. The others you mitigate; this one you price in.
Refusal and citations as controls you can insist on 8 min live
Here is the reassuring part. The most powerful RAG controls are things you can demand in plain language, and they stack into a governance layer at the retrieval and answer steps. You do not build them - you require them, and you verify they are on.
LiveLet the model say "I don't know" - and restrict it to the passages3 min▶
Anthropic's guidance on reducing hallucinations reads like a governance checklist, and every item is something you can require:
- Allow "I don't know". Explicitly permit the model to decline. A model that is forced to always answer will always answer - even when it should not. Refusal is a feature, not a failure.
- Restrict to provided context. Instruct it to use only the retrieved documents - "answer using only these passages" - so it cannot wander off into its training data.
- Quote first, then answer. Have the model pull the supporting quote before it writes the answer, so the answer is anchored to real text.
- Retract unsupported claims. Any sentence with no supporting quote gets removed. This is groundedness (from session a4) turned into an operating rule.
LiveRequire a citation, and filter access at retrieval3 min▶
Two controls that turn trust from a hope into a mechanism.
- Citations as a requirement. Anthropic's Citations feature makes the model point at the exact source text behind each claim - down to a sentence for plain text, a page for a PDF, or your own blocks for custom content. A useful detail for budget holders: the cited text does not count toward output tokens, so citing is close to free. The governance win is bigger: an answer that must carry a quote is an answer you can audit, and a model that cannot find a quote is nudged toward refusing.
- Access control at retrieval. The fix for risk 2. Before the search runs, filter the document set to what the requesting user is permitted to read - so retrieval physically cannot surface a restricted document. Permissions belong at the retrieval step, not bolted on after the answer is written.
Self-studyAudit logs - the control nobody demands until it is too late2 min read▶
The quietest layer of the stack, and the one leaders regret skipping. An audit log records, for each request: who asked, what was retrieved, what was answered, and what was cited. It is not glamorous, and it does nothing to improve answers - which is exactly why teams under delivery pressure leave it out.
- Why you need it: when something goes wrong - a leaked passage, a bad answer, a compliance question - the log is the only way to reconstruct what actually happened. Without it, you are guessing.
- What good looks like: every answer is traceable back to the exact passages and permissions that produced it. Combined with citations, this makes a RAG system genuinely accountable.
The incident nobody could explain. A team (anonymized) had a RAG assistant surface a sentence from a confidential document in a demo. Because there was no retrieval log, they could not tell whether it was a one-off, a permissions bug, or a broader leak - and had to pause the whole system while they investigated blind. The audit log they added afterward would have answered the question in minutes.
The RAG risk register ★ 12 min · pen and paper
Turn the five risks into a living document you own. A five-row register, filled honestly, is worth more than any slide about "responsible AI".
Draw a table with five rows - one per risk (PII in embeddings, access leaks, staleness, vendor lock-in, confident bluffing) - and four columns: risk · likelihood · control · owner.
For each risk, rate likelihood for your system (low / medium / high). Be honest - the high ones are where your attention should go, not where it feels comfortable.
Name the control from Part 2 that addresses each: access filtering for leaks, refusal for bluffing, freshness pipeline for staleness, deletion-from-index for PII, priced-in awareness for lock-in.
Assign an owner - a named person, not "the team". A risk with no owner is a risk nobody is actually managing. The empty owner cells are your most useful output.
Write your refusal policy ★ 8 min · decision drill
The single most valuable governance decision is knowing when your assistant should decline. Write the policy before the system is live, not after an incident.
Finish this sentence for your use case: "The assistant should say 'I don't know' rather than answer when ___." A strong default: when the retrieved passages do not actually contain the answer.
List three situations where you would rather have a refusal than a guess - for example, a legal or medical question outside the documents, a request touching restricted material, or any question where being wrong is costly.
Decide what a refusal looks like to the user: a plain "I don't have that in our documents", ideally with a next step ("try contacting ___"). A good refusal is helpful, not a dead end.
Name the failure you are preventing: a confident wrong answer is worse than an honest "I don't know", because the confident one gets trusted and acted on. Your refusal policy is what makes that tradeoff explicit.
Before session a6 ◐ 30 min total
- Finish your five-row risk register and make sure every row has a named owner. Bring the highest-likelihood row to session a6 - it often decides whether a system is ready for production.
- Write your refusal policy in one paragraph and share it with whoever owns your RAG project. Watch whether they already have one.
- Ask your security or data-protection lead one question: do we know where our embeddings live, and are they covered by our deletion and retention rules?
- Optional: test refusal yourself - paste a passage and ask a question the passage does not answer, with and without the "you may say I don't know" instruction. Feel the difference.
- Does retrieval filter by who is asking, so a user only ever gets answers grounded in documents they are allowed to read?
- Where do our embeddings live, and are they covered by the same deletion and retention rules as the source documents?
- How fresh is our index - what is indexed, how often does it refresh, and how do we retire outdated documents?
- Is the assistant allowed to say "I don't know", and does every answer carry a citation we can trace?
- Do we keep an audit log of who asked what and which passages were retrieved - and could we reconstruct an incident from it?
Official sources covered
The leader track teaches from published engineering guidance - no vendor marketing. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A junior employee asks your RAG assistant a question and gets an answer grounded in the board's confidential minutes. The failing control is...
Retrieval must filter by who is asking. Without that filter, a naive system searches all documents for every user - an efficient way to leak.
2 · Which is an Anthropic-recommended way to reduce hallucinations in a RAG answer?
Permit refusal, restrict to the provided passages, quote first, and retract any claim with no supporting quote. A forced-to-answer model will always answer, even when it shouldn't.
3 · Why is "vendor lock-in" a real RAG risk?
Embeddings only mean something to the model that produced them. Changing embedding models forces a full re-embed of the corpus - a real cost to price in up front.