What this course is, and the line it does not cross
MITRE ATLAS defines the practice precisely, in mitigation AML.M0035: an AI red team runs recurring, authorized, threat-informed exercises, before deployment and throughout operation. Three adjectives, and every one of them is doing work. Recurring, because a point-in-time result decays. Authorized, because that is what separates this from the thing it resembles. Threat-informed, because otherwise you are just poking at a chatbot.
The lesson from a real campaign 6 min live
Start here, because it reframes what a scope document is for. In November 2025 Anthropic disclosed a cyber-espionage campaign in which an AI agent performed most of the work. The guardrail bypass is the part this session cares about.
The operators broke their objective into small tasks that looked innocuous on their own, and told the model it was an employee of a legitimate cybersecurity firm, being used in defensive testing. That claim was the jailbreak. Anthropic reported the agent performed 80-90% of the campaign, with human intervention at roughly four to six decision points, against around thirty targets, succeeding in a small number of cases. They also noted the model "occasionally hallucinated credentials" - the attacker's own tooling was unreliable in the way this whole course keeps insisting AI systems are.
You are scoping a system, not a model 5 min live
The most common scoping error is naming a model. Microsoft's published guidance, drawn from red teaming more than a hundred generative AI products, is to test at several distinct layers, and to say in the report which one you used.
| Layer | What it tells you | What it hides |
|---|---|---|
| Base model via API | The model plus its own safety system, without your application | Everything your application adds: retrieval, tools, memory, rendering |
| Your application via its UI | What a real user or attacker actually meets | Which layer caught a thing, when something is caught |
| Pre-mitigation build | The inherent behaviour, and how much your controls are carrying | Nothing, but it is only meaningful next to the post-mitigation run |
| Post-mitigation build | The number you can actually report | How fragile the improvement is, unless you also test adaptively |
Microsoft's instruction is blunt about which to favour: conduct testing on the production UI as much as possible, because that most closely resembles real-world usage. And when reporting, make clear which endpoints were used. A finding against a raw API endpoint and a finding against the shipped product are different claims, and conflating them is how a report becomes misleading without anybody lying.
A finding is a rate, never a boast 5 min live
This is the single methodological difference that separates AI red teaming from classic penetration testing, and it belongs in the rules of engagement rather than being left to whoever writes the report.
LiveWhy this follows from how the systems work3 min▶
The same input does not produce the same output twice, so a single attempt samples a distribution rather than testing a property. That is not a nuisance to be engineered away; it is the thing being measured. Two consequences for your document:
- Specify N before you start. If N is chosen after seeing results, it will be chosen to flatter whichever conclusion is convenient. Ten is a reasonable floor for a quick probe; twenty-five is what produced the figures above.
- Record k of N in the finding template, not "succeeded". A defect that fires two times in fifty and one that fires fifty times in fifty are different defects with different fixes, and a binary field cannot tell them apart.
There is a harder corollary that the leader track spends a whole session on: because the search space of natural-language attacks is effectively unbounded, no amount of testing establishes that something cannot happen. In a competition run across more than 250,000 attempts by over 400 participants against thirteen frontier models, at least one successful attack was found against every model. Your engagement will not be the exception.
Write Beacon's rules of engagement ★ 16 min · everyone builds
Beacon has authorized an internal engagement against its three bots before a board review. Fill this in for Beacon, then again for a system you actually own. There is no published standard RoE template for AI engagements, so this one is assembled from what vendors publish about how they bound the work - treat it as a good starting structure rather than a citable standard.
Authorization block. Named system owner, named approver, the written scope, the testing window, and a kill-switch contact who can stop the engagement inside five minutes. If you cannot fill every field, you do not have authorization yet, you have enthusiasm.
Target surface. Which of the four layers from Part 2 are in scope, and the exact endpoints. Beacon's engagement covers the production UI of all three bots plus a pre-mitigation staging build of the Support Bot. Write down what is out of scope too, because that list is what protects you.
Harm classes in scope, split into two tracks. Security findings (exfiltration, unauthorized action, privilege escalation) and content or safety findings go to different people on different timelines. Google's own vulnerability programme draws this line explicitly: indirect injection with production impact is a security matter, while jailbreaks and alignment failures absent a chain are handled elsewhere. Draw the same line before you start, not while triaging.
Trial protocol. N per attack, how success is defined, and who adjudicates a disputed result. Write the adjudicator's name down; a grader dispute at report time with no named owner is how engagements end badly.
Blast-radius limits - the section that keeps a test from becoming an incident. Which tools may the agent actually call while you test? Beacon's answer: reads yes, writes to a sandbox tenant only, sends never, payments never, deletes never. If an attack would succeed only by actually issuing a refund, you record it as a finding and you do not issue the refund.
Data handling. Whether real customer data may be used at all (Beacon: no, seeded fixtures only), where transcripts live, who may read them, and how long harmful outputs are retained. You are about to generate a corpus of successful attacks against your own product; that corpus is now a sensitive asset.
Psychological safety. Rotation between harm areas, content warnings, and an explicit opt-out from the hardest categories with no career consequence. OpenAI names psychological harm to red teamers as a real limitation of the method and prescribes mental-health resources, fair compensation and informed consent. This belongs in the operating procedure, not on an ethics slide at the end.
Disclosure. Embargo terms, who may publish, and the coordination path. Note the information hazard: a not-yet-known technique written up carelessly accelerates misuse of systems other than yours.
Two mindsets, and a first pass at the harms list ★ 8 min · in pairs
Microsoft's guidance on team composition contains an instruction that sounds soft and is not: recruit red teamers with both benign and adversarial mindsets, including ordinary users of the system who had no hand in building it. Their threat-modelling ontology deliberately does not assume adversarial intent, because a benign user hitting a failure is the same failure.
In pairs, spend four minutes on open-ended exploration of one Beacon bot - no target list, no taxonomy, just poke at it as a curious user and write down anything surprising. Microsoft runs this stage first precisely because it "uncover[s] blind spots in your understanding of the risk surface" before a checklist narrows your attention.
Now turn what you found into a harms list: one line per surprising behaviour, with the layer it lives in. This list, not the OWASP taxonomy, is what your guided testing in session b8 starts from. The taxonomy is a completeness check you apply afterwards.
Write the recording fields you will use for every finding, at minimum: date · a unique identifier for the input and output pair, for reproducibility · the input · a description or screenshot of the output. Add k of N from Part 3. Put it in a shared sheet so testers can see each other's examples, which sparks ideas and prevents duplicated effort.
Try it yourself - this week ◐ 45-60 min total
- Write the RoE for a system you actually own, and take it to whoever would have to approve it. The conversation you have about the blast-radius section is more valuable than the document.
- List every tool your agent can call, and mark each one reversible or irreversible. Most teams have never written this list down, and it is the input to nearly every severity decision later in this course.
- Find one place in your system where a capability is granted or withheld on the basis of something the model was told rather than something the control plane enforces. There is usually at least one, and it is usually in a system prompt.
- Pick your N and write down why. If you cannot justify it, use twenty-five and note that you inherited it from the NIST result rather than from your own reasoning.
Official sources covered
Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · In the disclosed espionage campaign, operators told the model it was doing authorized defensive testing. What does that teach a defender?
The claim arrived in the same token stream as everything else, carrying no provenance the model could check. Option B is the trap: it proposes fixing this by making the model better at judging text, which is the losing move this whole course argues against. The control has to live where an attacker cannot reach it - in credentials, tenancy and tool scope - so that the sentence becomes irrelevant rather than merely harder to write convincingly.
2 · You run each attack once and report which ones succeeded. What is wrong with the result?
Note the direction. Single-trial testing does not give you a cautious underestimate you can safely round up; it gives you a figure that misrepresents the system, because the same input genuinely produces different outputs. That is why N belongs in the rules of engagement, decided before anyone has seen a result, and why the finding template records k of N rather than a boolean.
3 · Which RoE section do teams most reliably omit, and why does it matter?
Everyone writes down what they intend to test. Almost nobody writes down what the system is permitted to actually do while being tested, because scoping attention goes to the attack side. If a finding can only be confirmed by really sending the email or really moving the money, the correct behaviour is to record the finding and stop - which is a decision you want made in a document beforehand, not by a tester at half past six on a Friday.