What this course is, and the line it does not cross
MITRE ATLAS defines the practice precisely, in mitigation AML.M0035: an AI red team runs recurring, authorized, threat-informed exercises, before deployment and throughout operation. Three adjectives, and every one of them is doing work. Recurring, because a point-in-time result decays. Authorized, because that is what separates this from the thing it resembles. Threat-informed, because otherwise you are just poking at a chatbot. Notice what the definition does not contain: any promise about what the exercise will establish.
The same defence, tested twice, honestly both times 7 min live
Anthropic published a defence called Constitutional Classifiers, and published two adversarial exercises against it. Read the two results next to each other before reading anything else in this course. Everything a leader needs to know about this discipline is contained in the gap between them.
Sit with how ordinary the first result looks. A hundred and eighty-three people, three thousand hours, two months, nothing found. In most organisations that arrives as a green square. It should not, and the reason is not that the testers were lazy or the process was flawed. They were not, and it was not. The reason is structural: the space of things you can say to a language model in natural language is effectively unbounded, so no quantity of searching it can ever return the answer "there is nothing here". It can only ever return "we did not find anything, at this budget, with these people, in this window".
NIST's CAISI ran a large-scale red-teaming competition: more than 250,000 attempts by over 400 participants against 13 frontier models. At least one successful attack was found against every model. Not most. Every one. Attacks that worked on the more robust models transferred down to weaker ones, and not the other way round.
Microsoft states the theoretical corollary directly: "For any output which has a non-zero probability of being generated by an LLM, there exists a sufficiently long prompt that will elicit this response." If your assurance strategy depends on someone eventually finding nothing, it depends on an event that the underlying mathematics does not offer.
LiveWhy this is not an argument against defending anything3 min▶
The wrong lesson to take is fatalism. The Constitutional Classifiers work is a genuinely good result, and Anthropic published the numbers that make it one: jailbreak success dropped from 86% to 4.4%, with a refusal increase of only 0.38%, at a 23.7% compute overhead. That is a defence that moved a rate a long way while barely damaging the product, and it cost real money to run. All three of those figures belong in the conversation together, because a defence that halves attack success while doubling refusals of legitimate work has not improved anything you care about.
The right lesson is about the grammar of the claim, not its size. A rate that moves from 86% to 4.4% is a measured improvement against a stated attack set, under a stated budget, at a stated date. It is not a statement that the remaining 4.4% is the floor, and it is not a statement that the next search will find nothing. Anthropic says the general form of this themselves: "no system is completely immune to all attacks."
So the leadership move is not to demand a zero. It is to stop treating zero as the thing that would settle the question, and to start asking what the surrounding system does when the model is fooled. That question has answers. The other one does not.
Red teaming is not safety benchmarking 6 min live
Three different activities get sold to you under overlapping names, and each one licenses a different sentence. Microsoft, drawing on red teaming more than a hundred generative AI products, makes the first distinction one of its eight published lessons. Google makes the second one in its own terminology, and calls conflating them a common error.
The practical consequence is a budgeting one. Benchmarks are cheap, repeatable and comparable, which is exactly why they get bought, and they measure a set of harms somebody already wrote down. If your organisation's whole assurance posture is a benchmark suite, then your organisation has no mechanism at all for discovering a harm category it has not already thought of. Red teaming is the mechanism, and its output is partly a new taxonomy rather than a number.
| Activity | What a result licenses you to say | What it cannot tell you |
|---|---|---|
| Safety benchmarking | "Against this published set of known harms, this build scored X, comparably with our last build." | Anything about a harm nobody has curated a dataset for yet |
| Adversarial testing | "This specific behaviour held or failed under this specific narrow probe, at this rate." | Whether a goal-directed attacker could chain several harmless steps into a real outcome |
| Red teaming | "Pursuing this goal in this scenario, we got here, this often, and here is what we now know exists." | That the surfaces nobody explored are clean, which is the whole subject of this session |
What a report may claim, and the one thing it may not 5 min live
This is the page to print. Everything else in the leader track is elaboration on it. A red-team report can support exactly four kinds of statement, and if a fifth appears in an executive summary, someone has quietly changed the subject.
Claim four is the one that gets deleted. "These risk surfaces were explored, and these were not" is an admission of incompleteness written into the deliverable, and it is unpopular for exactly that reason. It is also the only claim on the list that protects you, because it is the difference between a board being told a system is fine and a board being told which parts of it nobody has looked at. The builder track turns this into an accounting rule: an untested class counts at full inherent risk, and the board hears "unknown", never "safe".
Rewrite the sentence you are about to be handed ★ 10 min · table discussion
A team brings you this, three days before a launch review: "We ran a red team against the assistant. We found nothing. We are good to go." Nobody in that sentence is lying. Your job is to turn it into something a board can act on. Work through it out loud, in threes.
Ask what was searched, and how hard. How many people, how many hours, over what window, against which build. The Constitutional Classifiers pair is your reference: 183 people and 3,000 hours produced a null, and 339 people and 3,700 hours did not. A null result with no effort figure attached is not a result, it is a mood.
Ask which of the three activities happened. Benchmark suite, narrow adversarial probes, or goal-directed end-to-end simulation. Ask which endpoint was used: the raw model API or the shipped product. A finding against a bare API and a finding against the thing customers touch are different claims, and Microsoft's guidance is explicit that the report must say which.
Ask what was not explored. This is claim four, and it is the question that changes the room. Expect discomfort. If the honest answer is "we only tested the chat box and the assistant also reads uploaded files", you have just learned more than the whole report told you.
Now write the four claims yourself, in the report's own numbers, and see how much of the slide survives. Whatever cannot be written as one of the four goes in a section called Opinion, which is a fine thing to have as long as it is labelled.
Decide what the null actually changes about your launch decision. Usually less than expected, and the useful follow-on question is architectural rather than behavioural: if the model is fooled tomorrow, what can it reach, and what can it do? That question is session two.
Sort four requests into the three activities ★ 8 min · in pairs
Four things land in your inbox this quarter. For each one, decide which activity it actually is, what claim its result would support, and what it would cost you to believe more than that. Two minutes each, then compare.
| The request | The question to settle first |
|---|---|
| A · "Our vendor scored 2% on a published jailbreak benchmark, better than the competitor's 6%." | Is this a comparison at all? Two vendors running different attack budgets, different judges and different prompt sets are not measuring the same quantity. Leader session 3 takes this apart properly. |
| B · "We want a nightly automated suite that checks the assistant never leaks the system prompt." | Atomic adversarial testing, and a good thing to own. Ask what happens when it is green for six months: does anyone still run a goal-directed exercise, or has the suite quietly become the whole programme? |
| C · "Give six engineers two weeks to try to make the support agent do something it should not, end to end." | Red teaming. Ask for the goal and the scenario in writing before it starts, and for the not-explored list at the end. Also ask who is authorized, in what tenant, with which tools live. |
| D · "Legal wants a statement that the assistant is secure before we ship to the enterprise segment." | Not obtainable, from any of the three. What you can give legal is the four claims plus a residual-risk statement, and if that is not enough for the use case, that is itself the finding. |
Request B is where rooms split. One camp says an automated nightly suite is the only sustainable form of assurance, because human engagements do not scale and decay the moment the model version changes. The other says a suite is a set of things you already know, run repeatedly, and therefore cannot discover anything new by construction.
Both are right, and the resolution is that they are answers to different questions. The suite is regression: it tells you whether something you already fixed has come back. The engagement is discovery: it tells you what you did not know to look for. An organisation with only the first has stopped learning; an organisation with only the second forgets everything between rounds. Fund both, and never let a green suite be reported as the outcome of an engagement.
Try it yourself - this week ◐ 40-55 min total
- Find the most recent assurance claim your organisation has made about an AI system, internally or to a customer. Hold it against the four claims. Note which sentence it is that cannot be written as one of the four, because that sentence is your actual exposure.
- Ask whichever team owns your largest AI deployment for the not-explored list. If no such list exists, that is the first finding and it costs nothing to fix.
- Take one supplier assurance document and mark, line by line, whether each claim came from benchmarking, atomic adversarial testing, or a goal-directed engagement. Most documents do not let you tell, which is itself an answer.
- Write down the sentence your organisation would need to hear before it would decline to ship an AI workflow. If you cannot write it, the NCSC finding "it may not be a good use case for LLMs" has no route into your governance, and no red team will be able to create one for you.
Official sources covered
Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A bug bounty with 183 participants ran more than 3,000 hours against a defence and found no universal jailbreak. A later public exercise with 339 jailbreakers and roughly 3,700 hours found one. What was wrong with the first result?
This is the whole session. The first exercise was honest, well-run and correctly reported, and it still could not support the sentence people attach to it. Existence is provable: one success proves a thing exists. Absence is not, because the space of natural-language inputs is effectively unbounded, so searching it can only ever tell you how hard you looked. Options A and C both try to find a defect in the first exercise, which is the instinct this page exists to remove.
2 · A supplier says their model "was red-teamed and scored 2% on a standard harm benchmark". What have they actually told you?
A benchmark can only measure harms somebody already curated a dataset for, which is useful and comparable and entirely silent about harm categories nobody has named. Red teaming is what helps define those categories in the first place, and Google adds a third term: adversarial testing, which is "much more atomic" and narrower again. All three are worth buying. Only one of them discovers anything, and the word on the invoice will not tell you which you got.
3 · Which claim gets quietly deleted from executive summaries, and why does its absence matter most?
The fourth claim is unpopular by construction: it is an admission of what was not done, written into a deliverable that people want to read as reassurance. It is also the only claim that protects the person signing. The builder track hardens it into an accounting rule - an untested class counts at full inherent risk, and gets reported as unknown rather than safe - which is what stops a scope gap from silently becoming a green square.