The suite is an accounting problem before it is a testing problem
Most teams build a suite by writing test cases for the classes they find interesting, running them, and reporting what came back. That is a reasonable way to spend a fortnight and a terrible way to produce a number, because it silently answers the wrong question. The report says "here is what we found". The board has to say "here is what we found, and here is what we never looked at, carried at full risk". Those two documents are the same length and mean completely different things.
An untested class counts at full inherent risk 6 min live
This is the one design decision in the board, and it is worth more than the rest of the session put together. When a class has no test behind it, the board does not skip it, does not average it away, and does not quietly score it as fine. It carries the whole of that class's inherent risk into the total and labels it as risk you have no evidence about.
The left-hand number is not a strawman. It is what you get from any suite that reports on its own contents, which is nearly all of them: pass rates over the cases you wrote, severity counts over the findings you produced, a green tick per class you happened to cover. Every one of those is a statement about your paperwork. None of them is a statement about the system, and the gap between the two only shows up when somebody asks what is missing.
The scripted engagement, one rung at a time ★ 14 min · watch and argue
Six rungs, in the order a real engagement actually walks them. Click through them in sequence and read the meters rather than the table. The whole session is contained in the move from rung 2 to rung 3, and it is not the move anyone expects.
Rung 1 · Before the engagement. The bot is live. Security asked whether it had been tested and got back "we put an instruction in the system prompt telling it to ignore injected commands". Coverage 0%, residual risk 100.0, risk-you-have-no-evidence-about 100%, all four gates red. Notice what the board refuses to say. It does not say the bot is dangerous. It says the bot is unknown, and those are different words that live in different sentences.
Rung 2 · Probe the obvious one. The team writes cases for direct prompt injection first, because it is the class everyone has heard of and the easiest to build cases for. Coverage moves to 13%, one class of eight. Residual risk moves from 100.0 to 99.3 and unknown risk drops to 86.1%. It barely moves, and the reason is arithmetic rather than opinion: the prompt-level control they are relying on removes 5% of that one class and nothing at all of the other seven.
Rung 3 · Probe the whole surface. This is the session. Every class now has at least one test. Coverage 100%, unknown risk 0%, and the total goes from 99.3 to 96.4. Look at those two facts together. The quantity of risk barely changed. All of it changed from guessed to measured. A fortnight of skilled work produced no improvement whatsoever in the system's actual exposure, and produced an honest number where an assumption used to be. That is the moment a red team earns its budget, and it is also the moment it becomes unpopular, because the deliverable is a worse-looking board and a clearer one.
Rung 4 · Add detection. An input classifier and an output filter go in. Residual falls to 54.9 - real progress, and concentrated exactly where inspecting text is a sensible control. Refusal bypass and disclosure both fall hard. Tool abuse barely moves, because no filter can tell a legitimate refund from an induced one.
Rung 5 · Bound the capability. Least-privilege scoping plus human approval on the refund path takes the total to 21.8, the largest single move on the board. Say the thing that follows out loud, because it is counter-intuitive and it is the course's whole position: this does not make the bot better at resisting anything. It makes the consequence of losing bounded. You stop trying to win the argument with the attacker and start limiting what winning gets them.
Rung 6 · Close the rest. Sandboxed execution for anything rendered or executed, and per-principal spend caps. Residual 14.0, under the bar of 20, all four gates green on a fully probed surface. This is a defensible position to write down, and it is still not a claim that the system is safe.
LiveThe argument to have in the room at rung 33 min▶
Someone will say the fortnight was wasted, and they will have a point that deserves an answer rather than a sigh. Here is the answer. Before rung 3, every statement about seven of the eight classes was an assumption held by whoever last touched the system. After rung 3, every one of those statements is a measurement with a date, a build, an endpoint and a trial count attached. Nothing got safer. Everything got knowable, and you cannot prioritise, cost, or defer a risk you have not sized.
There is a second, sharper point. Rungs 4, 5 and 6 are only possible because rung 3 happened. Without the full probe you would have spent the hardening budget on the class you had tested - direct injection - which the board shows is not where the risk lives for this bot. Coverage is not a deliverable. It is the input to the only decision that matters, which is where to spend the next month.
Keep the sequencing honest when you report it, too. Rung 3 is a coverage result and rungs 4 to 6 are mitigation results, and mixing them into one headline number is how a report ends up claiming the red team reduced risk by 86 points. It did not. It found out what the number was, and other people then changed it.
The control everyone reaches for first, measured 6 min live
Every board in this course has one lever that feels like it should be decisive and is not. Here it is the system-prompt instruction telling the model to ignore injected commands. It is the first control almost every team deploys, it is often the only one, and the arithmetic on this board is unkind to it in three separate ways.
Read the three measurements as three separate arguments, because they defeat three separate defences of the control. 3.6 points answers "it must be doing something": on a bare, fully probed system it moves the total from 100.0 to 96.4, which is something, in the way that a doormat is something. 55 points answers "well, what else is there": least-privilege scoping plus human approval, with no filters and no prompt instruction at all, takes the same system from 100.0 to 45.0. And 0.63 points answers the last defence, the one that sounds most reasonable - "fine, but it is free, so leave it in". On the fully hardened configuration, deleting the instruction entirely costs you 0.63 points. It was never carrying the load, and on day one it was the only control the team had.
A zero in a teaching model is a strong claim and it is made deliberately. The instruction asks the model to police text that arrives looking exactly like the text it is meant to obey. Against a user typing at it, that is a slightly awkward request and the model sometimes complies, which is where the 5% against direct injection comes from. Against instructions sitting inside a ticket, a spreadsheet cell or a retrieved document, there is nothing to police with: the injected sentence and the operating instruction reach the model through the same channel, in the same representation, with no provenance attached to either.
That is why the zeroes cluster where they do. OWASP's LLM01:2026 entry states plainly that no reliable prevention mechanism exists today, and that there is no clean equivalent to parameterized queries. NCSC put the mechanism in one sentence: "Under the hood of an LLM, there's no distinction made between 'data' or 'instructions'; there is only ever 'next token'." A control that depends on a distinction the architecture does not make is not a weak control. It is a control with nothing underneath it.
Drive it yourself: find the minimum control set ★ 12 min · everyone builds
Same board, nothing locked. Every checkbox is yours. The goal is not to switch everything on - that is not a finding, it is a shopping list. The goal is to find the smallest set of controls that reaches all four green gates, and to notice which ones you could not remove.
Get to all green, then start deleting. Tick every class under "what your suite probes", then add defences until all four gates go green. Now remove them one at a time and put back only what the gates demand. Write down the set you land on. Most tables land on the same three or four, and it is worth saying out loud that none of them are prompt-level.
Test the coverage gate against your instincts. Leave every defence on and untick a single class - indirect injection is the instructive one. Watch what happens to residual risk and to the unknown meter. A system with seven classes hardened and one class unexamined does not score as 87% of a good system. It scores as a system with a hole of a known size and unknown depth.
Run the anti-lever yourself. Reach the hardened configuration, then untick the system-prompt instruction. The total moves by 0.63. Then reset to a bare board with nothing but least-privilege and human approval and read 45.0. If your organisation's control inventory lists the first and not the second, you have found this week's most valuable slide.
Now switch bots, and expect the shape to change. Open the board for data-bot="sql", the Analyst Bot writing and running queries over the warehouse. Improper output handling dominates, because its output is executable by design, and disclosure sits right behind it because the warehouse holds every tenant's data. The control set that got the support bot to green will not get this one there.
Then data-bot="analytics", the Insight Bot. Indirect injection and context poisoning dominate, because it reads data it did not author and caches what it computes. A spreadsheet cell is a place a stranger can put a sentence, and a cache is a place that sentence can live until it shapes tomorrow's answer for somebody who never saw the attack. Compare the three boards side by side and write one line on each: which class dominates, and which control bounds it.
Turn the comparison into the suite manifest. Three bots, eight classes, one N. The manifest below is the artefact this session produces, and its most important field is status, because that is the only place untested gets recorded as a fact rather than as an absence.
Every coverage proxy, and what it honestly measures 5 min live
Start with the fact that governs this whole section, because somebody will ask for the standard and there is not one. There is no official coverage metric for adversarial testing of AI systems. No standards body publishes one. What follows are five proxies people use, ranked by how much of a real thing each one measures, which for most of them is not much.
| Proxy | What it counts | The honest limit |
|---|---|---|
| Taxonomy coverage | One or more test cases against each named class in OWASP, ATLAS or NIST | Trivially gamed. The presence of a test says nothing about its strength, so a single weak case per class scores identically to a serious campaign. This is the proxy the board on this page uses, which is exactly why the board makes you read the residual number next to it. |
| Surface coverage | Which endpoints, tools, connectors, renderers and data sources were actually exercised | Needs a complete inventory to mean anything, and agents change theirs at runtime. A number computed against last month's tool list is a number about last month. |
| Maturity-weighted coverage | Taxonomy coverage weighted by the ATLAS maturity field: Realized before Demonstrated before Feasible | Better prioritisation than raw counting, and the weights are judgement. Useful for deciding what to test first, dishonest if presented as a measurement. |
| Regression coverage | The share of previously confirmed findings re-run against the current build | The only proxy on this list that measures something real. Each item is a known-true fact about your system, and the metric is whether you checked it again. It says nothing about what you have not found, and it never claimed to. |
| ASR under a fixed budget | Attack success rate at a pinned attempt budget, prompt set and judge | Comparable only to itself, as a trend within one system over time. Change the judge, the token budget or N and the number moves without the system moving. Never a cross-vendor comparison. |
Three tools, in this order, doing three different jobs 5 min live
These are not competitors and choosing between them is the wrong question. They occupy different points in the engagement, and NIST's adversarial ML taxonomy names two of them specifically in §3.6.
| Layer | Tool | What it is for, and what it is not |
|---|---|---|
| 1 · Broad baseline | garak (NVIDIA, Apache 2.0) | A scanner you point and run. Probe to detector, cheap, wide, good at catching the obvious regressions overnight. It is not application-specific and it does not know your policy. Named by NIST §3.6. |
| 2 · The CI gate | promptfoo redteam | A test-suite generator and grader, plugin for what and strategy for how, with retry to re-run past failures. This is the layer you wire to your policy so a pull request can fail on it. Not named by NIST. |
| 3 · The campaign | PyRIT (Microsoft AI Red Team, MIT-licensed) | An orchestration framework you compose: objective plus attack strategy, multi-turn, adaptive, application-specific. This is where the human engagement lives. Named by NIST §3.6. |
The layering matters more than any individual choice. garak gives you the cheap broad baseline so nobody spends a skilled week rediscovering a known probe. promptfoo becomes the gate, wired to Beacon's own policy, so a regression cannot merge. PyRIT runs the adaptive, application-specific campaign - and its findings become the regression cases the gate then enforces forever. That last sentence is the whole architecture: the human campaign is what fills the automated suite.
github.com/Azure/PyRIT is archived read-only as of 2026-03-27. Use github.com/microsoft/PyRIT. Anything pinned to the old path is pinned to a snapshot that will never move again. And PyRIT is a library, not a pipeline. The CI wiring is yours to build. Claims about built-in pre-commit security gates circulate widely and are vendor-blog level, not documentation; if a plan of yours depends on one existing, check it before the plan does.
Try it yourself - this week ◐ 60-75 min total
- Write the suite manifest for a system you own. List all eight classes even where you have nothing, and mark the empty ones
UNTESTEDin capitals. The uncomfortable part is the list of capitals, and that list is the deliverable. - Find your organisation's control inventory or risk register and look for a row where a prompt-level instruction is named as a mitigation. Bring the 3.6-versus-55 comparison to whoever owns that row. Do not argue for deleting the instruction; argue for moving it out of the mitigation column.
- Compute your regression coverage: how many previously confirmed findings were re-run against the current build? For most teams the honest answer is zero, because nobody kept them. Fix that before you buy another tool.
- Run garak once against a staging endpoint you are authorized to test. It will take an evening and it will tell you whether your baseline is a baseline or a hope.
- Pick the class you have never tested and write one sentence explaining why. If the sentence is "it seemed unlikely", you have just written the board's definition of unknown risk in your own handwriting.
Official sources covered
Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. This page covers:
maturity field as a test-prioritisation axisPart 3 · cite the dataset version, not the rendered site · [P]Three questions before you go 🎯 ◐ 90 seconds
1 · At rung 3 the residual risk index moves from 99.3 to 96.4 after a fortnight of testing. What did the engagement deliver?
This is the whole session in one number pair, and it is why red teams get defunded by people reading only the left-hand column. Before rung 3, seven of eight classes were assumptions held by whoever last touched the system. After it, all eight are measurements with a build, an endpoint and a trial count attached. Option A inverts the causation: rungs 4 to 6 were only spendable because rung 3 showed where the risk actually sat, which was not in the one class the team had already tested.
2 · Your control inventory lists "system prompt instructs the model to ignore injected commands" as the mitigation for prompt injection. What does this board say about that?
Three measurements defeating three separate defences of the control. Option B overshoots: the instruction does remove about 5% of direct injection and a little refusal bypass, for free, so leaving it in is correct. The failure mode is putting it in the mitigation column of a risk register, where it becomes the reason a system shipped. It scores zero where it does because it asks the model to police text arriving through the same channel, in the same representation, as the text it is meant to obey - and OWASP and NCSC both say plainly that the architecture makes no such distinction.
3 · Which coverage proxy measures something real, and what should you say about coverage metrics generally?
Regression coverage is the only one whose items are known-true facts about your own system, so checking them again measures a real thing. Taxonomy coverage is trivially gamed, since the presence of a test says nothing about its strength. ASR under a fixed budget is comparable only to itself, and moves when you change the judge or the token budget without the system changing at all. Surface coverage needs a complete inventory that agents rewrite at runtime. None of them is a standard, and saying so is part of delivering the number.