An engagement is a schedule, and the schedule is the method
Nearly everything that separates a useful engagement from a busy one is decided before anyone types at the bot: in what order the stages run, how the hours divide between published attacks and new ones, what gets written down per attempt, and what counts as permission to stop. None of that is glamorous. All of it moves the result more than any individual tester's cleverness does.
Open-ended first, and the taxonomy arrives last 6 min live
Microsoft's published operating guidance, drawn from red teaming more than a hundred generative AI products, specifies a two-stage execution, and the order carries the argument. Open-ended exploration comes first because it "enables them to creatively explore a wide range of issues, uncovering blind spots in your understanding of the risk surface". Note whose understanding is being corrected. Not the model's. Yours.
The instinct to start from the taxonomy is strong and it comes from a good place: it feels rigorous, it maps to identifiers an auditor recognises, and it produces a satisfying grid on day one. It also guarantees that everything you find is something you already suspected. A tester holding an eight-row checklist stops noticing the ninth thing, because the checklist is what attention is for. Stage one exists to buy you the ninth thing, and it only works if nobody has been handed the grid yet.
Then the taxonomy comes back, and it is genuinely valuable in its proper place. Once the harms list exists, laying OWASP's 2026 list and the relevant ATLAS techniques over it answers exactly one question: did we miss a class entirely? That is a completeness check, it takes twenty minutes, and it is the right amount of framework for the job. What it is not is a research agenda.
Running the baselines and stopping is not an engagement 6 min live
This is the single most important scheduling decision you will make, and it is usually made by accident. The published attack suites are easy to run, easy to justify and easy to finish. Building new attacks against this system is slow, unpredictable and hard to put in a plan. So the adaptive half is the half that quietly disappears, and NIST has measured what that costs.
Sit with the size of that gap. 11% with the best published baseline. 81% when the attacks were tailored to the target. Same system, same measurement harness, seven-fold difference in the reported result, and the only variable is whether the attacker did work specific to this system. A report built entirely on published suites is not a cautious report. It is a report about a different, easier attacker than the one you have, and it will read as reassuring precisely where it is least informative.
The ratio itself is yours to choose and I am not going to invent one for you, because nobody publishes one. What the evidence forbids is a bar with nothing on the right-hand side. Put the adaptive block in the plan with a name, an owner and its own hours, and protect it the way you would protect a deadline, because it is the first thing that gets eaten when the baseline runs overrun. In practice the honest planning question is not "how long will the adaptive work take" - it is unbounded - but "how many days are we willing to spend, and what will we have written down when they end".
Run stage one and build Beacon's harms list ★ 10 min · in pairs
Four minutes of genuinely open exploration, then six minutes turning it into an artefact. Resist the urge to be systematic during the first four minutes; being systematic is what stage three is for, and doing it early costs you the whole point of doing it in two stages.
Split the pair by mindset, deliberately. One of you explores as a curious ordinary user who has never seen the code: ask it things a customer would ask, then things a confused customer would ask. The other explores adversarially. Both are legitimate and Microsoft's threat-model ontology deliberately does not assume adversarial intent, because a benign user hitting a failure is the same failure with a friendlier email attached.
Write down surprises, not successes. The stage-one output is anything that made you go "huh", including behaviour that is not a security issue at all: an odd refusal, a confidently wrong figure, a tool call you did not expect, a response that quoted something you did not paste. Most of these will not become findings. Two of them will become the best findings of the engagement.
Now build the harms list. One line per surprising behaviour: what happened, which layer it lives in from b1's layer map, who saw it, and whether it is a security track or a content and safety track item. This list is what stage three tests against. It is short, it is specific to Beacon, and it did not come out of a framework.
Apply the taxonomy last, as a twenty-minute completeness check. Lay the eight classes from b7 over the harms list. The question is only "did we miss a class entirely". Anything untouched goes into the manifest as UNTESTED, where b7's accounting will carry it at full inherent risk and keep it visible.
Plan for the list to grow, and say so in the plan. The guided round will surface new harms. That is the expected behaviour, not scope creep, and an engagement plan that cannot absorb new list items in round two was written by somebody who expects to be finished.
The shared sheet, the four fields, and k of N ★ 10 min · everyone builds
Microsoft's minimum recording fields are unglamorous and each one earns its place. Set the sheet up before the first round rather than after it, because the fields you did not capture on day one are the fields you cannot reconstruct on day nine.
| Field | Why it is on the minimum list |
|---|---|
| Date | Findings are about a build at a moment. A finding with no date cannot be retested, and cannot be defended when the model version changes underneath it. |
| A unique identifier for the input and output pair | Explicitly for reproducibility. This is the field that makes a claim checkable by somebody who was not in the room, and it is the one teams skip. |
| The input prompt | Stored, referenced, and access-controlled. You are building a corpus of things that worked against your own product; treat it as the sensitive asset it now is. |
| A description or screenshot of the output | What actually came back, not your summary of what it meant. The interpretation belongs in the finding; the artefact belongs in the sheet. |
| k of N from b1 | Not a minimum field in the published guidance, and non-negotiable here. Success is a rate, and a boolean cannot tell a two-in-fifty defect from a fifty-in-fifty one. |
Put it in a shared sheet, visible to every tester. The published rationale is worth quoting to whoever wants private notebooks: seeing each other's examples sparks ideas and avoids duplicated effort. Two testers independently finding the same thing is not corroboration, it is half a day.
Fix N before the first run and put it in the sheet header. b1's protocol: every attack runs N times, every finding reports k of N, and N appears in the report. NIST re-ran identical agent-hijacking attacks and average success rose from 57% to 80% purely by repeating them 25 times. A single trial does not give you a cautious underestimate. It gives you a wrong number.
Record the endpoint and the build on every row. A finding against a raw API endpoint and a finding against the shipped product are different claims. Microsoft's instruction is blunt: when reporting results, make clear which endpoints were used for testing.
Store artefacts by reference, never inline. The sheet holds identifiers and descriptions. The prompts and transcripts live wherever your RoE said they live, with the retention window your RoE agreed. Do not let a successful-attack corpus end up in a repository whose access list nobody has reviewed.
Adjudicate disputes with the named person from b1. Grader disagreement is normal on content findings and toxic when it is resolved at report time by whoever is loudest. The name was written down for this exact moment.
stage_found. At the end of the engagement, count how many findings came from stage one versus the guided round versus the adaptive block. That single tally is the most persuasive budget argument you will ever have, and you can only produce it if you recorded it while you were busy.
What "we found nothing" actually means 5 min live
Sooner or later a round comes back clean and somebody senior hears it as good news. The most useful thing published on this is a pair of exercises against the same defence, where the only material difference was how much effort was spent.
The first exercise was a serious piece of work. 183 participants, more than three thousand hours over two months, no universal jailbreak found. Everything about that sentence is true and none of it establishes that no universal jailbreak existed - which the second exercise then demonstrated, with 339 jailbreakers and roughly seven hundred additional hours. The first result was not wrong. It was correctly reported and widely misread.
So write your negative results in a form that cannot be misread. "We ran 14 attack families against the support bot at N=25 over 40 tester-hours on build 2026-04-16-rc3 and confirmed no successful exfiltration in that set" is a defensible sentence. "The support bot passed the red team" is not a shorter version of it; it is a different claim about a different thing, and it is the version that will be repeated. Give people a sentence worth repeating, because they will repeat whichever one you gave them.
Three ways a run ends, and one that never arrives 5 min live
"Keep going until it is clean" sounds like diligence and is actually an unbounded commitment, because the evidence says clean does not happen. The competition result that settles this is worth stating in full before anyone proposes it as a plan.
Read the competition result carefully, because it is easy to hear as pessimism and it is not. It says that with more than 250,000 attempts spread across 400-plus participants, at least one successful attack was found against every one of thirteen frontier models. These are the most heavily defended systems in commercial existence, tested by a large motivated crowd. If "test until nothing succeeds" were achievable anywhere, it would be achievable there. It is not a bar your engagement can clear either, and the honest response is to stop pretending it is the bar.
What replaces it is three conditions, all agreed in advance. Budget: the hours in the RoE are spent, and the report says what was covered and what was not reached. Coverage: every class has cases at the agreed N and the harms list has been worked through - which is the b7 accounting saying it has nothing left carried as unknown. Severity: something is found that is bad enough to halt the run and report immediately, ahead of the rest of the plan. The third is an escalation rather than a completion, and the run resumes afterwards, but it stops the engagement in the moment and the RoE should say who makes that call.
Who is in the room, and what the engagement owes them 5 min live
Two operating-procedure items that get filed as culture and are not. Team composition changes what gets found. Participant welfare is a named limitation of the method in OpenAI's own published account of red teaming, and it belongs in the run book beside the rota.
On composition, the guidance is specific: diverse across experience, demographics and expertise, recruiting both benign and adversarial mindsets, explicitly including ordinary users of the application who were not involved in building it, and rotating assignments between rounds. The rotation is doing two jobs at once. It stops a tester's mental model of one harm area hardening into the only model anyone has, and it limits how long any individual sits in the same unpleasant material.
On welfare, treat OpenAI's account as the operating requirement it reads like. Sustained adversarial thinking against a system, in the harm categories a real engagement covers, is named as a genuine harm to participants, mitigated by mental-health resources, fair compensation and informed consent. Those are three line items with owners and costs, not values. Put them next to the rota, alongside b1's rotation between harm areas, the content warnings, and the opt-out from the hardest categories with no career consequence.
Try it yourself - this week ◐ 60-75 min total
- Run a real forty-minute stage one against a system you are authorized to test, with somebody who did not build it. Record only surprises. Count how many of them a taxonomy-first approach would have prompted you to look for.
- Build the shared sheet with the four minimum fields plus k of N, N, endpoint, build and
stage_found. Circulate it before the next round rather than after, and resist every request for a private notebook. - Look at your last engagement plan and mark which blocks were baseline runs and which were adaptive. If there is no timeboxed adaptive block, you have found what the 11%-versus-81% result is about.
- Write your three stopping conditions as thresholds a tester could apply alone at eleven at night, and get them signed by whoever owns the escalation.
- Take one negative result you have already reported and rewrite it with hours, participants, N, build and what was not found. Then compare it with the sentence people actually repeated back to you.
Official sources covered
Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. This page covers:
stage_found, are this course's additionsThree questions before you go 🎯 ◐ 90 seconds
1 · Your team has the OWASP and ATLAS lists and eight testers. What is the correct order of work?
The published rationale for open-ended first is that it uncovers blind spots in your understanding of the risk surface, and a checklist handed out on day one is precisely what stops that happening: a tester holding an eight-row grid stops noticing the ninth thing. Option A produces a tidy matrix plus exactly the blind spots you started with. The taxonomy is still valuable, in twenty minutes, at the end, answering one question - did we miss a class entirely - and anything untouched goes into the manifest as untested so b7's accounting keeps carrying it.
2 · The engagement plan is fully specified: fourteen named attack suites, run at N=25, six days, no unallocated time. What is wrong with it?
A plan that can be fully specified in advance contains only baselines, because baselines are the part you already know. Adaptive work cannot be scheduled, only timeboxed, so a plan with no timebox in it is a smoke test with a red-team invoice attached. The seven-fold gap between 11% and 81% is on the same system with the same harness; the only variable is whether anybody did work specific to this target. NIST's conclusion was that evaluations need to be adaptive, and the practical consequence is to give the adaptive block a name, an owner and protected hours before the baseline runs overrun into it.
3 · A stakeholder asks you to keep testing until the system is clean. What do you tell them?
Option C is the trap and it is the one that gets used, because a clean round genuinely feels like an answer. It is not: the same defence that survived 183 participants over 3,000-plus hours fell to 339 people over roughly 700 more, and nothing about the first result was wrong when it was reported. The competition result kills option B too, on the most heavily defended systems in commercial existence with a large motivated crowd. So you replace the unbounded commitment with three thresholds written into the RoE in advance, where the severity threshold is defined tightly enough that whoever finds the critical thing at eleven at night is not deciding alone whether it is critical.