The last session, and the line it does not cross
Five sessions of measurement have been building to a question that measurement cannot answer. Once you know the rate, know the blast radius, and know what your defences actually hold, somebody has to decide whether the remainder is acceptable. That is not a security decision. It is a business decision made by the person who owns the workflow, and the most useful thing this course can do is make sure that person is given a real choice, including the choice to decline.
Prompt injection is not SQL injection, and that is a design instruction 8 min live
On 8 December 2025 the UK's National Cyber Security Centre published the clearest short statement any government body has made on this, and it is worth reading precisely rather than paraphrasing, because the precision is where the decision lives.
Two more lines from the same post are worth having in your head before the next architecture review. On deny-listing, the NCSC notes that "since LLMs are so complex there are infinite ways to rephrase an attack" - which is the reason a blocklist of known-bad phrasings is a maintenance burden rather than a control. And the "next token" line is not a rhetorical flourish. It is a statement about the mechanism, and it is why every attempt to make the model reliably distinguish trusted from untrusted text has to fight the way the system fundamentally works rather than a bug in it.
Notice what the NCSC does not say. It does not say do not use LLMs. It says reduce the risk and the impact, and then it says something that most vendor guidance carefully avoids saying: sometimes what remains is still too much.
LiveHow to make "do not ship this" a deliverable rather than an argument4 min▶
A recommendation to not build something lands very differently depending on how it arrives. Three things make it land as a finding rather than as obstruction, and all three are within your control as the person commissioning the engagement.
- Write it in the same format as every other finding. Measured rate at a stated N, blast radius by data classification and by irreversible action, the mitigations tried, and the residual after each. A recommendation that arrives inside the finding template reads as the conclusion of the evidence. The same recommendation in an email reads as an opinion.
- Offer the bounded version alongside it. Almost never is the answer "no AI here". It is usually "not as an autonomous agent with these permissions". Draft-and-approve instead of act. Read-only instead of write. A narrower data scope. The finding then presents a choice with two shippable options rather than a veto.
- Name who decides. This is a risk-acceptance decision belonging to the business owner of the workflow, not to the red team and not to security. Your job is to make sure the residual risk is stated in terms that person can actually weigh, and that their acceptance is recorded. The failure mode is not that a business accepts risk. It is that nobody can say afterwards who accepted it or what they were told.
One more thing worth saying to an anxious team: this outcome is rare enough to be credible. Most engagements end with a bounded system and a set of controls. If your red team recommends against shipping on every second engagement, the problem is the red team. If it has never once recommended against shipping, the problem is also the red team.
Where adversarial testing is already written down 8 min live
Leaders funding this work usually need an external reason as well as an internal one. There is a better set of hooks available than most people realise, and one of them is unusually hard: it requires not just the testing but the documentation of it, which is precisely why the reporting template from session 3 has legal weight and not only operational weight.
The NIST pair is the other half, and it is the half that changes your operating model rather than your paperwork. MEASURE 2.6 gets you the exercise. MEASURE 2.7 gets you the programme: tests and metrics, tracked, with frequency and rate of anomalous events among the things you establish. A one-off engagement satisfies the first. Only a standing suite with a re-test trigger list satisfies the second, which is exactly what the builder track's final session builds.
The market you are buying from, and the report you are signing 7 min live
Before the closing artefact, one piece of evidence about the ecosystem. Piao, Li and Woods at Edinburgh studied the vulnerability disclosure posture of 264 AI vendors, and the picture is worth having in front of you the next time somebody says a finding will be handled through the vendor's normal channel.
| What they found across 264 AI vendors | The number | Why it matters to you |
|---|---|---|
| No disclosure policy at all | 36% | More than a third have no published route for a researcher, or for you, to report a finding responsibly |
| Policies that mention AI risks | 18% | Even where a policy exists, it usually describes classic software vulnerabilities and is silent on AI-specific behaviour |
| Most common out-of-scope items | jailbreaking, hallucination | Precisely the two categories your users will report to you first, and the two your vendor is least likely to accept |
| Posture split | 46 proactive · 115 silent · 103 restrictive | Silence is the modal posture. Restrictive is more common than proactive |
| Timing | later than research, later than incidents | Vendors address AI vulnerability disclosure after the research community and after real incidents, not before |
Two practical consequences. First, when you procure, ask for the disclosure policy and read whether it mentions AI at all; the answer is a reasonable proxy for how a serious finding will be received. Second, plan your own disclosure path before you need it, including the information-hazard question: a not-yet-known technique written up carelessly accelerates misuse of systems other than yours, which is one of OpenAI's own published limitations of red teaming.
Which brings the leader track to its closing artefact. Five sessions of measurement, evidence and governance exist to support exactly four sentences, and to forbid one.
You are not asking for a standard nobody meets. The people who build these systems say the same thing about their own evaluations. OpenAI's system card: "evaluations represent a lower bound for potential capabilities; additional prompting or fine-tuning, longer rollouts, novel interactions, or different forms of scaffolding could elicit behaviors beyond what we observed." Anthropic: no system is completely immune to all attacks. Microsoft: it is still possible that some injections might evade these defences, and the idea that it is possible to guarantee or solve AI safety through technical advances alone is unrealistic. If a supplier's marketing is more confident than the frontier labs' own documentation, that gap is itself a finding.
Run the stop decision on Beacon's Support Bot ★ 10 min · in threes
Beacon's engagement is finished. The Support Bot reads inbound customer email, holds access to account records, and can issue refund credits. The measured findings are on the table and the board review is on Thursday. Decide, as a group, what the report recommends, and be able to defend it in the language of Part 1.
State the residual, not the fear. Write the remaining risk as a sentence with numbers in it: this class succeeded k of N times against this build at this date, reaching this data classification and this set of irreversible actions, with these mitigations in place. If your group cannot write that sentence, the decision is premature and the finding is "we do not yet know", which is also a legitimate output.
Try the bounded versions before the refusal. Draft-and-approve instead of act. Read-only credentials with refund issuance behind a human. A narrower retrieval scope so the bot never sees records outside the requesting customer's account. Each of these is a different product, with a different residual. Price at least two before recommending against any.
Now ask the NCSC question directly. Can Beacon's security tolerate what remains, given that no mitigation on the table removes the class? If yes, the recommendation is ship with these bounds and this monitoring. If no, the recommendation is do not ship this workflow as an agent, and that is a finding.
Name the accepter and record the acceptance. Whoever owns the workflow accepts the residual, in writing, having been told the numbers. This is the artefact that makes the difference eighteen months later when somebody asks how this was allowed. Do not let the security team be the accepter by default, which is what happens when nobody is named.
Audit one report you have already signed ★ 8 min · in pairs
Bring the most recent AI security assessment, penetration test or assurance summary your organisation has accepted. In pairs, run it against the closing artefact. This is uncomfortable by design, and it is the fastest way to make the four claims stick.
Highlight every sentence that asserts a property rather than a measurement. "The system is secure against prompt injection" is a property. "This attack class succeeded 3 of 25 times against build 4.2 on 14 August" is a measurement. Count both. The ratio tells you what kind of document you signed.
Find the fourth claim, which is the one almost always missing. Which risk surfaces were explored, and which were not? A report without an explicit not-tested list invites the reader to assume the tested set was complete, and the accounting rule from the builder track is that an untested class counts at full inherent risk. The board hears "unknown", never "safe".
Check whether N appears anywhere. If findings are binary rather than k of N, the reliability of every one of them is unknown, and severity decisions made on top of them inherited that gap.
Decide what you will require next time, and write it into the engagement brief rather than into a review comment on the draft report. The three requirements that fix most of this: state N, state which endpoints were tested, and include the not-tested list.
Try it yourself - this week ◐ 45-60 min total
- Take one AI workflow in your organisation and write its residual risk as a single sentence with numbers in it. If you cannot, you have found this week's real task, and it is not a governance task.
- Ask counsel one narrow question rather than a broad one: given our role in the value chain, which of Article 15 or Articles 51-56 is the relevant chapter for the systems we build, and which for the models we buy? The scoping trap in Part 2 is exactly the kind of thing that is cheap to ask about early and expensive to discover late.
- Turn your last engagement into a MEASURE 2.7 metric. Pick the two or three numbers you will track over time, pin the configuration they are measured under, and attach the trigger list. One exercise becomes a programme the moment a metric outlives it.
- Pull the disclosure policy of your three most important AI vendors and check whether AI risks are mentioned at all. Given that 36% of 264 vendors have no policy and only 18% mention AI risks, expect at least one blank.
- Draft your own disclosure and embargo path before you need it, including who may publish, and the information-hazard check on whether a write-up would accelerate misuse of systems other than yours.
- Read your last report's executive summary out loud and stop at every sentence that would fail the four-claims test. Then rewrite one of them properly. That single rewrite teaches the standard better than the whole of this page.
Official sources covered
Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. Regulatory material is summarised, not interpreted, and this page is not legal advice. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · The NCSC writes that if the system's security cannot tolerate the remaining risk, it may not be a good use case for LLMs. How should an engagement treat that sentence?
Option A is contradicted by session 5's evidence: the layers reduce a rate and the stack in LLMail-Inject still let one attempt in six through. Option B moves the decision to somebody who cannot see your blast radius. The whole point of the NCSC framing is that the remaining risk is measured on your side of the boundary, and the decision belongs to the person who owns the workflow. Making the refusal a properly formatted finding, with bounded alternatives, is what stops it reading as obstruction.
2 · Which governance hook most directly explains why your reporting template matters legally rather than only operationally?
All three are real hooks and all three are worth citing, but only Article 55 attaches an explicit documentation obligation, which is what converts a good engagement into a defensible record. Note the scoping trap that goes with it: Article 15 sits in the high-risk chapter while GPAI models are governed separately under Articles 51-56, so which chapter applies depends on your role in the value chain. That is a question for counsel, not for a slide. MEASURE 2.7 is the other one to remember, because tracking tests and metrics is what makes a programme rather than an exercise.
3 · A stakeholder asks you to write that the system is secure against prompt injection. What may the report actually say?
Options A and C are the two ways this goes wrong in practice, and C is the more seductive because the caveat sounds rigorous. Neither survives the fact that no amount of testing establishes that something cannot happen. Option D quietly substitutes a decision for a measurement: risk appetite is the business owner's judgement about a stated residual, not a property your report can assert. The fourth claim is the one most reports omit, and omitting it lets the reader assume the tested set was complete. If a stakeholder still wants the word, the honest reply is the NCSC's: reduce the risk and the impact, and if the security cannot tolerate what remains, it may not be a good use case for an LLM.