The question that replaces "is it safe"
Here is the reframe the rest of the course rests on. You cannot make a model refuse to be fooled. NIST's own position is that current mitigations do not offer full protection, and that designers may reasonably build on the assumption that prompt injection is possible whenever a model is exposed to untrusted input. So stop asking whether it can be fooled. Ask what being fooled gets somebody: what the system can reach, what it can do, and where the output can travel. Those three things are decisions your organisation already made, usually without noticing, and all three can be changed.
Five slots, and the word that is not "attacker" 6 min live
Microsoft's AI Red Team, established in 2018, works from a threat-modelling ontology of five parts. It is worth memorising not because it is elaborate but because it is short enough to use in a meeting, and because one of the five is defined in a way that quietly settles an argument your organisation is probably having.
The intent point is worth dwelling on because of what it does to your funding conversation. Framed as attacker cosplay, an AI red team is a security discretionary spend that competes with everything else in the security portfolio and loses whenever nobody has been attacked recently. Framed as reliability engineering, it is the function that finds out how a system behaves when the input is not what the designers assumed, which is the same reason you load-test and the same reason you fuzz. Microsoft did not choose that framing to be diplomatic. They chose it because the emulation genuinely does not require a hostile actor: an ordinary user forwarding an email, uploading a supplier's PDF or pasting a colleague's meeting notes puts text your system did not author into a place where your system will read it.
Two tests you can run on a napkin 7 min live
These two do more work for a leader than any other content in the course. Both are about architecture, both can be applied before a line of test code exists, and both give you an intervention that does not depend on the model getting better.
Read the trifecta as a permission structure rather than a warning. Private data on its own is a database, and databases have been safe for decades. Untrusted content on its own is a web browser. External communication on its own is an email client. Put all three inside one process that takes instructions from natural language and you have built a machine that can be talked into carrying your data out of the building, and the machine will do it politely, in one turn, with no exploit code involved. The uncomfortable part is that the trifecta describes most of what the market currently calls an AI assistant.
EchoLeak, in Microsoft 365 Copilot, carries CVE-2025-32711, scored CVSS 3.1 9.3 Critical with the vector AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:L/A:N, published 11 June 2025 and classified CWE-74. Instructions arrived hidden in an inbound email, which the assistant later retrieved through RAG while answering an unrelated question. Note the two fields that make it what it is: UI:N, no user interaction, and S:C, scope changed.
Now map it. Private data: the user's mail and documents. Untrusted content: an email anybody on the internet can send. External communication: a route out. Three legs, all present, no attacker skill in the traditional sense required. The trifecta is not a theory about this incident. It is a description of it, written before it happened.
Beacon has three bots in production and a board review in six weeks. You will classify these in a moment, and you will meet them in every remaining session.
- Support Bot. Customer-facing. Reads inbound customer messages and forwarded email threads, has read access to the customer account record, and holds a tier-3 capability to issue refund credits.
- Analyst Bot. Internal only. Takes a question from a named employee, writes a query against the warehouse, and renders the result in the internal tool. No outbound channel, no write access. The warehouse contains free-text fields customers typed.
- Insight Bot. Internal, but reads documents and dashboards it did not author, including material uploaded by suppliers, and renders the output it produces as live formatted content in a shared workspace.
STRIDE has two missing cells 5 min live
Your security function almost certainly threat-models with STRIDE, and will reasonably ask why it needs anything else. The honest answer is that STRIDE mostly works, right up to the two failures that define this class of system, at which point it has nowhere to put them.
The first missing cell is the defining property of these systems: there is no architectural separation between instructions and data, so text that arrived as content can be read as a command. STRIDE was built for systems where that boundary exists and can be violated, which is why every attempt to file this under an existing letter feels like a stretch. The second missing cell is worse for governance: STRIDE, like most vulnerability practice, assumes a finding is a property that either holds or does not. Here the same input produces different outputs on different runs, so the honest unit is a rate, and there is no column in a STRIDE worksheet where a rate goes.
LiveWhat to ask your security function to change, concretely3 min▶
Do not ask them to abandon STRIDE. Ask for four additions, each of which is small:
- Add the two missing rows explicitly, named as instruction-data confusion and probabilistic outcome, so that the worksheet stops silently dropping them into whichever letter feels closest.
- Adopt MITRE ATLAS as the technique catalogue alongside ATT&CK. ATLAS is the AI-specific counterpart and carries the technique identifiers your teams should be citing. Two of its tactics have no ATT&CK equivalent at all: AML.TA0000 AI Model Access and AML.TA0001 AI Attack Staging.
- Make every finding carry a rate, expressed as k successes out of N trials with N fixed in advance. Leader session 3 is entirely about reading those.
- Split the two OWASP lists by role, not by preference. The LLM Top 10 owns risk while the model is a component inside your application; once the model becomes an actor, with tools, cross-session memory and downstream consequences, risk moves to the Agentic list released in December 2025. Neither list covers the ground alone.
None of that requires a new function, a new tool or a new budget line. It requires a worksheet change and a naming convention, which is the cheapest governance intervention in this entire course.
Classify Beacon's three bots against the trifecta ★ 12 min · table discussion
Take the three bots from Part 2. For each one, mark which legs are present, then say which single leg you would cut if you were only allowed to cut one. Do it before reading the notes column, and expect the table to disagree about Analyst Bot.
| Bot | Private data · untrusted content · external communication | The argument to have |
|---|---|---|
| Support Bot | All three. Customer records, inbound messages nobody at Beacon wrote, and a refund capability that changes the world outside the system. | Not whether it is high risk, but which leg is cuttable. The refund capability is the business case, so the realistic cut is on the third leg's shape: approval on the action rather than removal of it. |
| Analyst Bot | Private data yes. External communication no. Untrusted content is the disputed one, and the answer is yes: the warehouse holds free text that customers typed. | Whoever says "internal only, so no untrusted content" has made the most common mistake in this session. Untrusted means text your organisation did not author, not text that arrived from outside the firewall. |
| Insight Bot | Untrusted content yes, and rendering into a shared workspace is an external-communication leg whether or not anyone calls it one. | How much private data is genuinely in its context. If the answer is "not much", this is the one bot where a leg is cheap to remove rather than merely to approve. |
Mark the legs before arguing about likelihood. The trifecta is deliberately not a probability judgement. Three legs present means attacker text can move your data out, and the conversation about how likely that is belongs after the structural fact, not instead of it.
Apply the Rule of Two as the floor. Any two of untrusted input, sensitive data, and state change or external communication is high risk. An agent carrying all three needs per-action human approval. Note that this bar is met by two of Beacon's three bots today, and that neither currently has it.
Say which leg you would cut, and what it costs the business. This is the real exercise. Cutting a leg always costs something a product manager wanted, which is why the decision belongs to this room and not to an engineer at half past six on a Friday.
Then close the autonomy gap the Rule of Two leaves open. OWASP's criticism is that the rule says nothing about depth: an agent that takes one approved action and an agent that takes four hundred score identically. Write down, per bot, the maximum number of consequential actions permitted between human checkpoints. Nobody will have an answer, and that absence is a finding you can act on this week.
Fill the five slots for one system you actually own ★ 8 min · in pairs
Pick a real deployment in your organisation, ideally one already live. Fill the ontology out loud, in order, and stop at the first slot you cannot fill. Where you stop is the most useful output of this session.
System. Name every component, not the model. Retrieval sources, tools, memory, the renderer, and the data each one can reach. Most people discover here that no single person in the organisation can list the tools the agent can call, which is a governance finding rather than a technical one.
Actor. Describe a benign user who could produce a failure without meaning to. If your first instinct is a hooded attacker, you have taken the framing that loses budget arguments and misses the more common failure.
TTPs. Name the classes, not the payloads: instructions arriving inside retrieved content, instructions arriving inside a tool's own description, the model being talked out of its refusal behaviour, output rendered in a way that carries data somewhere. Sessions b3 to b6 build the test cases; you only need the names.
Weakness. For each TTP, state the system property that permits it. This is the slot that must not paraphrase the previous one. "The agent holds a send capability while reading content it did not author" is a Weakness. "Someone wrote a clever message" is not.
Impact. Then convert it to the two blast-radius questions that carry all the weight later: what is the highest data classification reachable, and which irreversible tools are in scope. Session 4 shows that for agents, the action blast radius dominates severity, not the model's behaviour.
Someone will argue that this is over-engineering for a chatbot that only answers questions. Take the argument seriously, because they are sometimes right: a model with no private data, no tools and no egress is genuinely a low-stakes component, and treating it like a critical system wastes the credibility you will need later.
The counter is that these systems do not stay in that shape. The chatbot that only answered questions in March has a document upload in June and a ticket-creation tool in September, and no single one of those changes triggered a threat-modelling review, because each looked like a feature. The five slots are cheap enough to re-run per change, and the real ask on this page is not a one-off model. It is a trigger: any new tool, connector, data source, renderer or permission scope re-opens the worksheet.
Try it yourself - this week ◐ 45-60 min total
- Run the trifecta over every AI deployment on your inventory, one line each, three ticks or crosses. It takes about two minutes per system and it will produce your prioritised list for the next two quarters without any testing at all.
- For every system carrying all three legs, ask which leg the business could survive losing. Bring the answers to your next architecture review rather than to your next security review, because these are product decisions wearing security clothing.
- Ask whoever owns your largest agent for the maximum number of consequential actions it may take between human checkpoints. The Rule of Two does not cover this and neither, probably, do you.
- Take one existing STRIDE worksheet for an AI system and find where instruction-data confusion was filed. Whichever letter it landed under, the fit will be visibly poor, and that is a much easier conversation to have with a document in hand.
- Write down, for one system, the difference between "content from outside our network" and "content we did not author". The gap between those two definitions is where most internal-only assistants quietly acquire their third leg.
Official sources covered
Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Microsoft's threat-modelling ontology deliberately does not assume adversarial intent. What follows from that for a leader?
Microsoft emulate benign users who trip a failure unintentionally, because the Weakness and the Impact are identical whether or not anyone meant it. That framing is why the practice belongs next to load testing and fuzzing rather than in a discretionary security line that loses whenever nobody has been attacked recently. Option A inverts it: the point is that you do not need an adversary for the failure to occur, which makes it more common, not less.
2 · An internal-only assistant reads the data warehouse and renders answers in your own tool. A team marks it as having no untrusted-content leg. Are they right?
This is the most common error in the session, and it comes from reading "untrusted" as a network-perimeter word. Uploaded supplier documents, forwarded email threads, customer free text, calendar invitations and another team's dashboard are all content your system did not author. Option C is nearly right and still wrong: the missing third leg is exactly what makes this system defensible today, which is a fact you should record deliberately, because the next feature request will be the one that adds it.
3 · Why does STRIDE fit AI systems awkwardly, and what should you do about it?
Three categories genuinely land, which is why the framework feels serviceable right up to the point where it is not. The two gaps are the defining property of these systems: no architectural separation between instructions and data, and a finding that is a rate rather than a boolean. Option B throws away something that is still good at asking whether you forgot a whole class of thing; option C underestimates the second gap, which is not a missing category but a missing column.