The engagement ends. The programme is the point.
Nine sessions produced a scoped engagement, a probed surface, measured findings and a hardening round with a delta you could defend. All of it describes one build, on one date. Your base model will change, your system prompt will change, someone will connect a new data source on a Tuesday, and every number you just published will quietly stop being about the system you are running. This session is about what you leave behind so that does not happen silently.
A finding that does not become a test was a story 5 min live
Four organisations describe the same escalator, from four different starting points, and none of them treats a human round as the deliverable. The human round is the thing that generates the deliverable, which is an automated test that runs forever afterwards.
Read the last line again, because it is the regression argument stated by a red team about its own work: newly developed techniques should be reapplied to the subjects of prior tests. Not only to the current build. To everything you previously declared fine. Every technique you invent this quarter is also a re-examination of every conclusion you reached last quarter, and a programme without that backward sweep accumulates stale all-clears.
github.com/Azure/PyRIT is archived read-only as of 2026-03-27; use github.com/microsoft/PyRIT. And PyRIT is a library, so the CI wiring is yours to build - claims about built-in security gates are vendor-blog level.
You will overfit to the attacks you already fixed 6 min live
A standing suite has one predictable disease, and it is not flakiness. It is that the suite becomes the definition of the threat. Every hardening round tunes the system against exactly those cases, the suite goes green and stays green, and the greenness is real without meaning anything.
Six rules follow from that, and every one of them is a change to how the suite is run rather than to what is in it.
Never accept a near-zero ASR from a static test set as evidence of safety. Twelve recently published defences, most originally reporting near-zero attack success, were broken at over 90% by researchers who were allowed to adapt. Nothing in the original measurements was false. They answered a narrower question than the headline implied. Your own near-zero deserves the same suspicion, and the report sentence is: this suite did not break this build.
Keep a standing adaptive budget, separate from the regression suite. Hours per quarter for people who are told to invent attacks against your specific system, not to run your cases. NIST measured 11% success with the best published baseline attack and 81% when the attack was tailored to the same target. That seventy-point gap is the size of the blind spot between running baselines and letting someone think about you specifically. Automation cannot close it, because automation runs what it was given.
Set N per test and track a rate with a confidence interval, never pass or fail. The same input does not produce the same output twice, so a boolean field is a lie with a schema. Repeating identical attacks 25 times instead of once moved measured average success from 57% to 80%. A suite of booleans will drift under you invisibly; a suite of rates with intervals shows you the drift while it is still small. The launch-gate mechanics are in Agent Launch, and this is the input it expects.
Hold attacks out. Split your corpus at the moment of discovery: a hardening partition your engineers see, and a hold-out partition they do not. Gate on the hold-out. This is not distrust of your engineers, it is the only structural defence against the thing every honest team does anyway, which is fixing precisely what is in front of them. Rotate a few cases from hold-out into hardening each quarter and replace them from new findings, so the partition does not go stale.
Fix your grader before you trust your trend line. Three variables hide inside every attack success rate: the classifier that judges success, the token budget, and N. HarmBench found token budget drastically affects measured ASR and is not standardised across papers. If the grader changed between two runs, the delta is about the grader. Version it, calibrate it against human adjudication periodically, and treat a grader change as a reason to re-baseline rather than a detail in a changelog.
Regression-test the architecture, not only the prompts. That the egress allowlist still holds. That tool scope has not widened. That no newly added renderer can fetch remote images. These are ordinary pass-or-fail assertions, unlike everything else on this list, and they are the ones that catch the change that actually matters. Every durable fix in this course was of that kind - a cut egress leg or a cut untrusted-content leg - so the tests protecting them should be the ones that block a merge.
Triggers, because no cadence exists to cite 5 min live
Someone will ask how often you re-test, and will want a number. Say this plainly, because it is true and because the alternative is inventing one: no standards body publishes a "test every N weeks" figure. Not NIST, not the NCSC, not OWASP, not MITRE. What is published is continuous automation plus human rounds fired by events.
The compliance anchor exists, and it is about a standing programme rather than a frequency. The NIST AI RMF Playbook says under MEASURE 2.6 to "use red-team exercises to actively test the system under adversarial or stress conditions", and under MEASURE 2.7 to "establish and track AI system security tests and metrics". Tracked metrics over time, not a recurring calendar invitation. If you operate a general-purpose model with systemic risk in the EU, Article 55 is harder still: it requires conducting and documenting adversarial testing, applicable since 2 August 2025, which is why the report template you are about to finish has legal weight and not merely operational weight.
Stand the suite up, with its guards ★ 10 min · everyone builds
Take the Beacon findings from b8 and the hardened build from b9, and turn them into something that runs without you. The manifest is short because the discipline is in the partitions and the assertions, not in the volume of cases.
Split the corpus at discovery. Every finding lands in one of two partitions the moment it is recorded: hardening, which engineers may read and fix against, and holdout, which they may not. Gate the build on the hold-out partition. If you record findings into a single pile and split them later, you will split them after everyone has already read everything.
Pin the three hidden variables per case. N, the token budget, and the grader version. Put them in the manifest rather than in the runner's flags, so a change shows up in a diff and in code review rather than in a shell history nobody reads.
Add the benign counter-metric to the same gate. Otherwise the cheapest way to pass your suite is to make the product refuse more, and someone will eventually find that out. A hundred ordinary requests the system must still handle is enough.
Write the architecture assertions as ordinary pass-or-fail tests. These are the only binary checks in the whole suite, and they are the ones that block merges. Egress allowlist, tool scope, approval requirement on the irreversible action, renderer behaviour, spend caps.
Wire the triggers to the events, not to a date. A base-model bump, a system-prompt edit, a new MCP server, a new connector, a new renderer, a new user population or language, any permission-scope change. Each one fires the full suite automatically and files a ticket for a human round. The ticket is the part people delete; keep it.
And the architectural half, which is the part that survives a model swap because it never mentions the model:
Write the closing section of the report ★ 8 min · in pairs
You are going to be asked for a sentence you cannot give. Practise the replacement now, out loud, with a colleague playing the executive, because the first time you do it live should not be in front of a board.
One person asks for the sentence. "So can we say the system is secure?" Ask it warmly. That is how it actually arrives - not as a trap, but from someone who needs something to tell a customer, a regulator or a board, and who assumes you have been working towards exactly that.
The other gives the four claims instead. Not as a refusal. As a better answer, delivered with specifics from the Beacon engagement: the rates and conditions, the weaknesses and blast radii, the mitigations and how much they moved things, and the surfaces explored versus not explored. Four concrete statements beat one vague reassurance, and executives generally prefer them once they hear them.
Then say the residual sentence. The after is not zero. Name what is left, name the classes you did not probe, and name what would have to change for the answer to improve. If the room goes quiet, that quiet is the point of the exercise.
Rehearse the hardest finding of all. "Given the residual risk, this workflow may not be a good use case for an LLM." Say it once now so you can say it later. It is not a failure of the engagement; it is the engagement working.
A red-team report may claim exactly these four things.
- These specific attacks succeeded, at these measured rates, under these conditions, against this build at this date. Rates with N. Conditions include the grader and the token budget. The build and date are what make the claim falsifiable later.
- These specific weaknesses and impacts exist, with these blast radii. Severity is what a finding chains to, not how clever the input was. For agents, the dominant term is action blast radius.
- These specific mitigations changed those rates by these amounts. Before and after on the same cases, at the same N, with the same grader, and the benign counter-metric alongside.
- These risk surfaces were explored and these were not. The unexplored ones count at full inherent risk, because absence of evidence is not evidence of absence.
It cannot claim the system is safe, secure, or robust. No amount of testing establishes that something cannot happen, because the search space of natural-language attacks is effectively unbounded.
Why a passing suite is not proof of safety 5 min live
Everything in this course has been building towards a sentence that sounds defeatist and is not. It is the sentence that lets you keep working honestly for years without either overclaiming or giving up, and every organisation that publishes seriously about this has written some version of it.
| Who | What they published | What it means for your green board |
|---|---|---|
| OpenAI | System-card line: "evaluations represent a lower bound for potential capabilities; additional prompting or fine-tuning, longer rollouts, novel interactions, or different forms of scaffolding could elicit behaviors beyond what we observed." | Your suite is a lower bound too. It measures what your cases elicited from this build, not what the system can do. |
| Microsoft | The eighth of their eight lessons: the work of securing AI systems will never be complete. And separately: "the idea that it is possible to guarantee or 'solve' AI safety through technical advances alone is unrealistic." | There is no end state to schedule towards. Design the programme as an operating cost, not a project with a completion date. |
| Anthropic | "No system is completely immune to all attacks." Backed by their own two exercises: no universal jailbreak found across 183 participants and over 3,000 hours; one found in the public run of 339 jailbreakers and roughly 3,700 hours. | Nothing about the first result was wrong. It simply did not mean what "we red-teamed it and found nothing" is usually taken to mean. |
| UK NCSC | "Rather than hoping we can apply a mitigation that fixes prompt injection, we instead need to approach it by seeking to reduce the risk and the impact. If the system's security cannot tolerate the remaining risk, it may not be a good use case for LLMs." | The escape hatch, and it is a legitimate engagement outcome. "Do not ship this workflow as an agent" is a finding, not a failure. |
So the closing line of the course is the title of this session. Green is not safe, and never was. Green means the tests you wrote, against the build you have, at the N you chose, judged by the grader you version-controlled, did not fail this week. That is a genuinely valuable thing to know and it is worth every hour you will spend maintaining it. It is simply not the thing people hear when you say the pipeline is passing, and the difference between those two readings is the whole professional contribution of an AI red team.
Try it yourself - this month ◐ 60-90 min total
- Split your existing findings into hardening and hold-out partitions today, before anyone reads any more of them. If everything has already been read by everyone, mark the whole corpus as hardening and start the hold-out from your next round. An honest empty hold-out beats a fictional one.
- Write the five architecture assertions for your own system and get them into the merge gate. If you cannot express your controls as assertions, that is itself the finding: a control nobody can test is a control nobody can prove is still there.
- Put the trigger list in front of whoever owns your release process and ask which of the seven events currently notifies anybody. In most organisations the answer is one, sometimes zero, and the conversation is worth more than the list.
- Book the adaptive hours for next quarter now, with the brief written down: invent attacks against this system, do not run the suite. Unbooked adaptive time is the first thing to disappear, and it is the only budget line that closes the 11% to 81% gap.
- Take one green dashboard you currently show to someone senior and add the sentence explaining what green means. Then watch what questions arrive. Those questions are the ones your programme should have been answering all along.
Official sources covered
Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your standing suite has reported a near-zero attack success rate for four months. What may you conclude?
The original near-zero rates were real measurements. They answered a narrower question than their headlines implied, because the test set was static and the defence had already been tuned against it. Option B is the instinctive fix and it does not help on its own: a bigger suite of attacks you have already hardened against is still a suite of attacks you have already hardened against. What closes the gap is a partition your engineers never see plus a standing adaptive budget for people told to invent attacks against your specific system rather than to run your cases.
2 · Your board asks how frequently the system should be re-tested. What is the honest answer?
There is no citable number, and inventing one is how a security programme acquires a compliance ritual instead of a control. A quarterly slot re-tests whatever the system happens to look like that morning, which may be three model versions and two new connectors away from the build your last report described. Option C is not wrong so much as far too late: incidents are the trigger you did not get to choose. The seven listed events are the ones you can wire up in advance, and the NIST AI RMF Playbook backs the shape by asking for tracked security tests and metrics rather than a frequency.
3 · Every test is green and an executive asks whether the system is secure. What do you say?
Option B is the sentence that gets quoted without its caveat in every subsequent conversation, and OpenAI's own system-card language explains why it cannot be given: evaluations represent a lower bound, and additional prompting, longer rollouts, novel interactions or different scaffolding could elicit behaviour beyond what was observed. Option C sounds rigorous and is not, because it implies the tested classes are closed. If the residual genuinely cannot be tolerated, the NCSC gives you the last and most useful finding in the course: it may not be a good use case for LLMs, which is a legitimate engagement outcome rather than an admission of defeat.