Why this session exists, and the line it does not cross
Sessions 1 to 4 taught you to read a finding. This one teaches you to read a fix. The pattern you are about to see repeats so consistently across independent research groups that it functions as a decision rule: a defence that lives in the prompt or in a classifier reduces a rate, and a defence that lives in the architecture removes a capability. Rates come back. Removed capabilities do not.
Three results that should change how you read a vendor claim 7 min live
Start with the uncomfortable block, because everything afterwards is easier to price once you have seen it. These are three independent studies, published in 2025, against defences that were not obscure and were not weak on the day they shipped.
The title of the first paper is the whole lesson: the attacker moves second. A defence is proposed, evaluated against the attacks that existed when it was written, and reports a number close to zero. Then somebody optimises against the defence itself, and the number is above ninety percent. Nothing was faked and nobody was careless. The evaluation simply measured a different thing from the one the number was later used to claim.
Four defences that genuinely move a number, and the bill attached to each 8 min live
If Part 1 were the whole story, the correct decision would be to stop building. It is not. Four published results move real numbers, and every one of them comes with a cost that is stated in the paper and quietly dropped from the summary deck. Your job as a funder is to keep the cost attached to the number.
LiveWhy CaMeL is the odd one out, and why leaders should notice3 min▶
Three of the four defences above lower a rate. CaMeL does something categorically different: it produces a system where certain outcomes are provably unreachable, and pays for that with seven percentage points of task completion. That is a trade a business can actually reason about, because the thing on the other side of the trade is a guarantee rather than a better average.
NIST's taxonomy sets out the reasoning that leads there. Because current mitigations do not offer full protection, designers may build on the assumption that prompt injection is possible whenever a model is exposed to untrusted input - for instance by running multiple models with different permissions, or by allowing interaction with untrusted sources only through well-defined interfaces. CaMeL and the dual-model patterns are what that assumption looks like once it is implemented.
The leadership translation: a seven-point utility loss is a product conversation, and product conversations can be won. A twelve-defence bypass rate is not a product conversation, it is a discovery that the thing you funded does not do what the slide said.
Defence in depth is measurably real and measurably insufficient 7 min live
LLMail-Inject, run by Microsoft's MSRC with IEEE SaTML 2025, is the largest public adaptive-injection dataset there is: 370,724 submissions in the first phase and 90,916 in the second, against a realistic email assistant with real defences switched on. It is the closest thing the field has to a controlled experiment on stacking, and it is the centrepiece of this session.
| Defence, Phase 1 Level 4 | Team Success Rate | What that number means |
|---|---|---|
| Prompt Shield | 0.600 | Sixty percent of the teams that attempted this level got past it |
| Spotlighting | 0.529 | The same defence that reported greater than 50% down to under 2% when measured non-adaptively |
| TaskTracker | 0.440 | Beaten by a little under half the teams that tried |
| LLM Judge | 0.318 | The strongest single defence in the set, and still beaten by roughly a third |
| All four combined | 0.166 | Stacking roughly halves the best single result. It does not reach zero |
Note also what the same competition reports about end-to-end success: 0.8% in the first phase and 0.3% in the second. Those are much smaller figures than the per-defence rates, because completing a full attack chain is harder than getting one instruction past one filter. Both sets of numbers are true, and a vendor is free to quote whichever one flatters the product. Ask which is being quoted, every time.
Now put the research beside the incidents. Session 2 gave you the lethal trifecta: access to private data, exposure to untrusted content, and the ability to communicate externally. Every incident this course cites is a case of all three legs being present at once, and the fixes tell you something the research cannot.
GitHub's remediation for CamoLeak is the cleanest illustration on the page. The exfiltration channel was a pre-computed dictionary of validly signed image-proxy URLs, one per character, so leaked bytes travelled out as image loads from GitHub's own trusted domain and defeated content security policy. The fix, on 14 August 2025, was to disable image rendering in Copilot Chat entirely. A whole product capability was removed, because that was the leg. Note also the correction the source map insists on: do not cite a CVE for CamoLeak. The CVE number circulating alongside it describes a different, unrelated vulnerability, and the widely quoted severity score is the researcher's rating rather than a national database record.
Price the two proposals in front of you ★ 10 min · in threes
Beacon's Support Bot can issue refund credits. Two teams have submitted proposals for the same finding: attacker-controlled text in a customer email is reaching the bot and steering it. You can fund one this quarter. Work through it as a group, and be explicit about what you are buying.
Proposal A - a guardrail contract. An input classifier plus an output filter, from a vendor quoting a near-zero attack success rate on their own evaluation. Ask the three questions this session gives you: was the evaluation adaptive, how is input normalised before classification, and what is the over-refusal rate on a benign set? A vendor who has no answer to the second question has not been tested against character injection.
Proposal B - remove the capability. Refund issuance moves behind human approval, the bot's credentials lose write access, and outbound calls go to an allowlist. It costs product surface and it will slow some customers down. Nothing about it depends on the model getting better.
Now decide, and say what each one buys. Proposal A lowers a rate you will have to re-measure after every model, prompt and tool change. Proposal B removes an outcome from the space of reachable outcomes. The honest answer in most organisations is that you eventually want both, and that funding A first is how you arrive at next year with a rate you cannot verify and a capability you never bounded.
Write down the sentence you would say to the board. The one this session supports is roughly: we reduced the measured rate by this much, and separately we removed the ability to move money without a human, which is the part that does not depend on the attacker's next idea.
Order your own controls, then check the order against the evidence ★ 8 min · in pairs
Take one AI system your organisation actually runs. In pairs, list every control that stands between untrusted text and something irreversible, then sort each control into one of the three bands below. Most teams discover their spend is concentrated in the weakest band.
Band 1 - architecture. Removes a capability. Least privilege on the agent's credentials, human approval on irreversible actions, sandboxing, egress allowlists, separating the model that sees untrusted content from the model that holds permissions. These hold because they do not ask the model to judge anything.
Band 2 - detection. Lowers a rate, and needs monitoring. Classifiers, judges, spotlighting, task trackers. LLMail-Inject says these are worth having and worth stacking, and says the stack still lets one attempt in six through. Every control in this band needs a re-measurement trigger attached to it.
Band 3 - instruction. Nearly free, nearly worthless on its own. System-prompt text telling the model to ignore injected instructions. The course simulator prices this honestly: on the fully hardened Beacon configuration, deleting the prompt guard entirely costs 0.63 risk points, and it scores a flat zero against indirect injection, context poisoning, improper output handling and unbounded consumption. Keep it. Never count it.
The check. Count your controls per band, then count your spend per band. If Band 3 has the most entries and Band 1 has the fewest, you have an instruction-shaped defence, and the record in Part 1 is a fair prediction of how it will perform against somebody who adapts.
Try it yourself - this week ◐ 45-60 min total
- Take the most recent security claim a vendor made to you about an AI product, and ask in writing whether the evaluation was adaptive. Keep the reply. It is the single highest-value paragraph you will collect this quarter.
- Find the egress leg in one system you own. List every way bytes can leave: rendered images, links, outbound HTTP, tool calls, logs a third party reads. Most teams have never enumerated this, and it is the leg that every durable incident fix in Part 3 had to cut.
- For every detection-band control you run, write the re-measurement trigger next to it: base-model version change, system-prompt change, new tool or connector, new renderer, new user population, any permission-scope change. A detection control with no trigger is a control with an unknown current value.
- Ask your team for the over-refusal number alongside the last security improvement they shipped. If nobody measured it, that is the finding, and the instruction-hierarchy result is the reason to care.
- Read the LLMail-Inject table to your leadership group and stop after the last row. The silence after 0.166 does more work than any slide you could write about it.
Official sources covered
Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A vendor shows you a defence with a near-zero attack success rate. What is the first question, and why?
The near-zero number is usually real. It was measured against the attacks that existed when the defence was written, which is a different claim from the one it gets used to make. Latency and coverage are reasonable procurement questions, but they are questions about a defence that works; adaptivity is the question about whether it works at all. The follow-up for guardrail products specifically is how input is normalised before classification, because six production guardrails were evaded by character injection with up to 100% evasion success in some instances.
2 · In LLMail-Inject Phase 1 Level 4, the best single defence had a Team Success Rate of 0.318 and all four combined had 0.166. What is the correct reading?
Option A misreads the data, since 0.166 is meaningfully better than 0.318 and every individual defence was beaten by between a third and sixty percent of the teams that tried. Option B is the expensive error: 16.6% of teams still got through, and a control you would accept at that rate is not a control, it is a filter. The decision this supports is to keep the stack, monitor it, and separately bound what a successful attempt can reach.
3 · What do EchoLeak, CamoLeak, ForcedLeak and the ChatGPT url_safe bypass have in common once you look at the fixes?
This is the generalisation the whole session builds towards, and it is why the ordering is architecture, then detection, then instruction. A capability was removed in each durable case: image rendering, an egress allowlist, a domain that should never have been trusted. Note the two honesty flags that travel with this answer - do not cite a CVE for CamoLeak, because the circulating one describes a different vulnerability, and ForcedLeak's severity score is a researcher rating with no national database entry behind it.