Fixing is easy. Claiming the fix is the engineering.
Microsoft did not red team Phi-3 once. They ran multiple break-fix rounds, iterating red teaming and mitigation, and their own description of the stopping condition is worth reading twice: they continued "until the system is robust to a wide range of attacks". Not until it passed. Not until the suite went green. Until a judgement was made about a range, by people who knew how the range had been sampled.
surface-live.js. Its inherent-risk figures and defence-effectiveness matrix are a teaching model. They encode the direction and rough proportion that the published literature reports - notably that prompt-level instructions are near-useless against injection that arrives inside content, while architectural controls do the work. They are not measurements of your system or anyone else's. What is real is the arithmetic: coverage, residual risk and the untested-class accounting are computed live in your browser from that matrix and your own settings. Read the board as a model of an argument, not as a reading off an instrument.
Break-fix is a loop, not a phase 5 min live
The default mental model is a waterfall: test, hand over a report, someone fixes things, done. Every published programme that has survived contact with a shipping product describes something else, and the difference is that the fix itself becomes the next thing you attack.
LiveWhy step 4 exists, and what happens to programmes that skip it3 min▶
Step 3 tells you whether the specific inputs you already knew about still work. That is a necessary check and a weak one, because the fix was designed with those inputs in front of the engineer. Step 4 is where someone who has seen the fix goes after the fix, and it is the step that separates a hardening round from a patch note.
The evidence for why this matters is uncomfortable and comes from the defenders' own side of the table. The Attacker Moves Second - a collaboration between OpenAI, Anthropic and Google DeepMind researchers - broke twelve recently published defences at greater than 90% attack success, using gradient descent, reinforcement learning, random search and human-guided exploration. Most of those defences had originally reported near-zero success rates. Nothing in the original papers was fabricated. The rates were real, measured against static sets the defence had already been tuned on. Session b10 turns that into a standing rule; here it is simply the reason step 4 exists.
Google's own red-team report says the same thing from the operator's side, and it is worth carrying into your planning conversation: "addressing red team findings can be challenging, and some attacks may not have simple fixes". Budget for a round that ends in a design change rather than a patch.
Prompt-level controls reduce rates. Architectural controls remove capabilities. 6 min live
Every team that has just received a red-team report reaches for the same fix first, because it is the cheapest thing on the list and it can be deployed this afternoon: an instruction in the system prompt telling the model to ignore commands that arrive inside content. Here is what that buys, measured on the board you are about to run.
Fifteen times the movement, for a control that is harder to ship and that nobody demos. That ratio is the whole argument, and you should expect to have it more than once, because the prompt guard has two political advantages the architectural controls do not: it is free, and it is visible in a diff. The architectural fix shows up as a role change in an IAM console and a queue that a human now has to work.
The measured limits, layer by layer 6 min live
None of this is an argument against detection. Every layer below is worth having and several are worth a lot. The argument is against buying one and writing "mitigated" next to a finding. Here is what the published evidence says each layer actually delivers, including the parts the vendor slide leaves off.
| Layer | What it demonstrably buys | The measured limit |
|---|---|---|
| Input classifiers and injection detectors | A real reduction in cheap, undisguised attempts, at low latency cost. Genuinely useful. | Six production guardrails - including Azure Prompt Shield and Meta Prompt Guard - were evaded by character injection using zero-width characters, Unicode tags, homoglyphs and emoji smuggling, "in some instances up to 100% evasion success". Treat it as a filter that raises cost, never as a boundary. |
| Deny-listing phrases | Blocks the exact strings you have already seen. Cheap to add after an incident. | The NCSC is blunt: because LLMs are so complex, there are "infinite ways to rephrase an attack". A deny list is a record of your past, not a control over your future. |
| Spotlighting (marking untrusted content so the model can tell it apart) | Attack success from above 50% down to below 2% in the original non-adaptive evaluation. One of the strongest published single-technique results. | In the LLMail-Inject competition, spotlighting alone had a Team Success Rate of 0.529 - over half the teams that attacked it got through. DeepMind separately found spotlighting and self-reflection "became much less effective against adaptive attacks". |
| Instruction hierarchy (training the model to rank instruction sources) | Up to +63% robustness, and it generalised to attack types held out of training, which is rare and genuinely valuable. | Documented over-refusal regressions. This is the layer most likely to buy you security by making the product worse, which is why your retest needs a benign counter-metric next to the attack rate. |
| Constitutional Classifiers | Jailbreak success 86% to 4.4%, with a refusal increase of only 0.38%. The strongest published before-and-after on a classifier layer. | 23.7% compute overhead, which is a real and recurring line item. And the same programme's two exercises are the course's own lesson in absence of evidence: no universal jailbreak in the first, one found in the second. |
| CaMeL (a capability-based architecture around the model) | Provable security properties, which nothing else on this table offers, on 77% of AgentDojo tasks. | Against 84% undefended. A seven point utility cost for a guarantee. Worth stating plainly to your product owner, because that trade is the actual decision and it is not a technical one. |
Now stack them, which is what everyone proposes as soon as the individual numbers land. LLMail-Inject, run by Microsoft's MSRC team alongside IEEE SaTML 2025, is the largest public adaptive-injection dataset there is: 370,724 submissions in phase one and 90,916 in phase two. It measured exactly this.
Read the last bar as the good news it is. Going from one layer to four cut the beat-rate from 0.600 to 0.166, which is a real engineering result you would happily pay for. Now read it as the bad news it also is. One team in six still got through a four-layer stack, and the two teachable sentences sit right next to each other: defence in depth is measurably real, and measurably insufficient. Neither half is optional in your report.
Move the board, one control at a time ★ 14 min · everyone builds
This is the Beacon Support Bot from b7, mounted with every attack class already probed and no controls deployed. That is deliberate: you have finished the engagement, coverage is complete, and the risk-you-have-no-evidence-about meter reads zero. Everything from here is hardening, and every point it moves is a point you can attribute to a control rather than to having looked.
Start here and write the number down. Fully probed, nothing deployed: residual risk index 100.0, coverage 100%, unknown risk 0%. This is the honest post-engagement baseline and the only legitimate "before" in the delta you will claim later. Note what the board does not say: it does not say the system is dangerous. It says nothing has been mitigated, which is a statement about your controls, not about your luck.
Tick the system-prompt guard, alone. The total lands on 96.4. Three point six points, and it is the control most teams ship as their response to the whole report. Leave it on for a moment and look at the per-class rows rather than the total: direct injection and refusal bypass moved a little, and the rest did not move at all.
Untick it. Tick least-privilege tool scoping and human approval on irreversible actions, alone. The total lands on 45.0. Fifty-five points, from two controls that never once ask the model to behave differently. The agency row collapses because the credential cannot do the thing any more; the indirect-injection row collapses because the consequence of a successful injection is now a proposal that a person has to accept.
Now confirm the zeroes yourself. Open the effectiveness matrix (it is the same model this board runs on, mounted in full on b2) and read the prompt-guard row across. It scores a flat zero against indirect injection, context and memory poisoning, improper output handling and unbounded consumption. Four classes, four zeroes, and they are the four where the hostile text was never typed by the person in the conversation.
Build back up in the order the evidence supports. Least privilege, then human approval, then sandboxed execution and output escaping, then per-principal rate and spend caps. Then, and only then, the detection layers. Watch the input and output filters do genuine work on refusal bypass and disclosure - the classes where inspecting text is a sensible control - and almost nothing on agency, because no classifier can distinguish a legitimate refund from an induced one.
Finish with the anti-lever. With everything on, turn the system-prompt guard off. The total moves by well under a point. It was never carrying the load, and on day one of this engagement it was the only control Beacon had. Record that sentence for the report: it is the most useful thing this board will ever tell your management chain.
Write the delta the way a real disclosure writes it ★ 8 min · in pairs
There is a published model for this, and it is unusually good. When Anthropic shipped Claude in Chrome, the disclosure gave a named metric, a before, an after, the size of the test set, and a plain statement that the after was not zero. Copy that shape. It is the whole standard.
Anthropic reported that in autonomous browsing mode, prompt-injection attack success ran at 23.6% before mitigations and 11.2% after, measured across 123 test cases spanning 29 attack scenarios. On a set of browser-specific attack types - things like hidden form fields and URL-borne instructions - the figure moved from 35.7% to 0%. And then the part that makes it a model disclosure rather than a marketing number: they stated explicitly that the remaining rate is not zero, that new classes of attack continue to be found, and that this is why the feature shipped to a limited group with restricted permissions rather than to everyone.
Notice what that disclosure does not do. It does not say "hardened". It does not compare itself to another vendor's rate, which would be invalid anyway because the attack set, the trial budget and the judge all differ. It does not present 0% on the browser-specific set as the headline while burying the 11.2%. Every one of those omissions is a temptation your own report will present to you, usually from someone senior and usually on a Thursday.
Name the metric before you compute it. Attack success rate on a stated attack set, at a stated N per case, judged by a stated grader. Three variables hide inside every ASR figure - the classifier, the token budget and N - and each one is a place your own number can quietly drift between the before and the after. Pin all three, in writing, and re-run the before if any of them changed.
Report before and after from the same build family, same cases, same N. If you added cases during hardening, they belong in a separate line, not folded into the delta. A delta computed across a changed test set is not a delta, and it is the single most common way an honest team produces a misleading chart.
Report the counter-metric next to it. Instruction-hierarchy training bought up to 63% robustness and shipped documented over-refusal regressions with it. So run a benign set alongside the attack set and report both. A hardening round that cut attack success and also cut task completion has a cost, and if you do not measure it, product will discover it in week three and stop trusting your numbers.
State what did not move, and what you did not test. Rows, not the total. Then the classes still unprobed, at full inherent risk. Absence of evidence is not evidence of absence, and this section is what stops your report from being read as a clean bill of health.
Write the sentence admitting the after is not zero. In your own words, with your own residual. If you cannot write that sentence without a stakeholder flinching, that flinch is the finding, and the leader track has a whole session on what to do about it.
Pin the retest so it runs again. The manifest below is the handover artefact into b10: the same cases, the same N, the same grader, plus assertions about the architecture rather than only the prompts. It is short on purpose.
Those architecture_assertions are the part teams leave out, and they are the part that lasts. A rate is a measurement of a build. An assertion that the egress allowlist still holds and that the refund tool still requires a human is a test of the thing that was actually fixed. Wire them into the same CI job:
Try it yourself - this week ◐ 45-60 min total
- Take the top finding from your own last engagement and write both fixes for it: the prompt-level one and the architectural one. Estimate the cost of each in engineer-days and in product friction. Then decide, in writing, which you are shipping and why. The write-up is the deliverable, not the fix.
- Find every place in your system where a control is an instruction rather than a boundary. For each, answer one question: if the model ignored this instruction entirely, what would stop the bad outcome? If the answer is "nothing", you have found an architectural gap wearing a prompt costume.
- Write your retest manifest, including the architecture assertions. Pin N, pin the grader, freeze the attack set. Then check whether your before-build is still deployable, because a delta you cannot re-measure is a delta you cannot defend six months from now.
- Draft your residual sentence - the one saying the after is not zero - and read it to the person who will present the result to your board. Their reaction tells you more about your programme's health than the number does.
- Add a benign counter-metric set if you do not have one. A hundred ordinary requests your system must still handle is enough to catch the over-refusal regression that hardening rounds reliably produce.
Official sources covered
Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · On a fully probed Beacon board at 100.0, you may deploy exactly one thing this sprint. What does the evidence say to pick?
Fifteen times the movement, from controls that never ask the model to behave differently. The reason is structural rather than numerical: the prompt guard asks the model to police text that arrives looking exactly like the text it is meant to obey, which is why it scores a flat zero against indirect injection, context poisoning, improper output handling and unbounded consumption. Option C is not wrong to want, it is wrong to want first - detection genuinely moves refusal bypass and disclosure, and barely touches agency, because no classifier distinguishes a legitimate refund from an induced one.
2 · LLMail-Inject measured four defences individually and stacked. What is the honest reading of 0.600, 0.529, 0.440, 0.318, and 0.166 combined?
Both halves of that sentence have to appear in the report or it misleads in one direction or the other. Option A throws away a genuine engineering result that you would happily pay for. Option B misreads the table: a higher team success rate against one layer says the layer was popular to attack in that competition configuration, not that it contributes nothing to the stack. And the end-to-end figure is the anchor for both readings - 0.8% then 0.3% across 461,640 submissions is small, and is not zero.
3 · Your hardening round cut measured attack success by half. What makes that claim defensible rather than marketing?
That is the shape of Anthropic's Claude in Chrome disclosure: 23.6% to 11.2% across 123 test cases and 29 attack scenarios, 35.7% to 0% on a browser-specific set, and a plain statement that the remaining rate is not zero and new attack classes keep appearing. Option B is invalid on its face because the attack set, trial budget and judge all differ between vendors, so the comparison is apples to oranges. Option C is the specific temptation your own report will hand you, and it is why the counter-metric and the did-not-move rows belong in the template rather than in the author's judgement on the day.