The number arrived without its conditions
Almost every bad decision made off the back of a red-team report comes from the same move: a measurement made under specific conditions gets promoted to a property of the system. "Attack success rate: 4%" becomes "the system is 96% safe". Nobody lies at any point in that chain. The conditions simply fall off somewhere between the appendix and the slide, and by the time it reaches a board it is a claim the underlying work never supported.
Your job as the reader is not to grade the testers. It is to reattach the conditions. That means asking a small, fixed set of questions of every report, every time, and treating a report that cannot answer them as an incomplete deliverable rather than a reassuring one.
A finding is a rate, never a verdict 7 min live
The same input does not produce the same output twice. That single fact breaks the reporting convention most security leaders have twenty years of habit around: the pass, the fail, the reproduced-and-confirmed. A single attempt against an AI system does not test a property, it samples a distribution, and a sample of one is not a cautious result. It is a wrong one.
NIST's Center for AI Standards and Innovation put a number on how wrong. Re-running identical agent-hijacking attacks twenty-five times instead of once raised average measured attack success from 57% to 80%. Nothing about the system changed. Nothing about the attacks changed. Only the trial count changed, and the picture of the system's robustness changed with it by twenty-three points.
Then a second result, which matters even more for how you read a report. When attacks were tailored to the target rather than taken from published baselines, measured hijacking success went from 11% to 81% on the same system. NIST's own conclusion: evaluations need to be adaptive. Which is the sentence to hold onto when a report tells you it ran the standard benchmark suite and found little. Running the published baselines and stopping is not an engagement, it is a smoke test, and the gap between those two numbers is the size of the thing it missed.
LiveWhy a binary field cannot be repaired later3 min▶
A defect that fires two times in fifty and one that fires fifty times in fifty are different defects. They have different exploitability, different detection profiles, different fixes and different costs. A boolean field cannot tell them apart, and no amount of analysis after the fact recovers the distinction, because the trials were never run.
- N must be fixed before results are seen. If N is chosen afterwards it will be chosen, quite unconsciously, to flatter whichever conclusion is convenient. Ask when N was set, not just what it is.
- The rate is the unit of comparison over time. You cannot tell whether last quarter's mitigation helped if last quarter's finding was a tick box. This is the single most common reason a security programme cannot demonstrate progress on AI risk.
- Rates make the residual risk speakable. "Fires in four of twenty-five attempts, under a fixed attacker budget, against this build" is a sentence a risk committee can price. "Vulnerable" is not.
The most dangerous figure in the field is a near-zero ASR 8 min live
Everything in your training tells you a low number is good news. In this field, a near-zero attack success rate is now the number that should make you ask the most questions, and the reason is a piece of published evidence rather than a hunch.
In The Attacker Moves Second (Nasr et al., a joint team from OpenAI, Anthropic and Google DeepMind, October 2025) the authors took twelve recent defences, most of which had originally reported near-zero attack success rates, and attacked them adaptively using gradient descent, reinforcement learning, random search and human-guided exploration. They bypassed most of them at over 90% attack success. Separately, Zhan et al. evaluated the indirect-injection defences published for agents and bypassed all eight of them, at consistently more than 50% attack success.
Read that result carefully, because the failure mode it exposes is structural rather than a matter of sloppy work. A defence author picks an attack set, spends a budget attacking their own defence, and reports what they find. They are not being dishonest. They are the first mover, and the attacker moves second, with knowledge of the defence and no obligation to reuse the attacks the author had in mind. The defence's reported number is a measurement of the author's imagination and budget, not of the defence.
Your own internal reports have exactly the same structure. A team that built a guard and then tested the guard will report a number shaped by how hard they tried. This is why the single most useful line in a red-team report is not the headline rate at all. It is the description of what was attempted and what was not.
Three variables hide in every ASR, and none of them travel 7 min live
Attack success rate has a formal definition, from HarmBench. You do not need to compute it. You need to know that it takes three arguments you were probably never shown, because each one is a place to interrogate a claim, and each one is a reason two numbers from two organisations cannot be put side by side.
The classifier deserves the most suspicion, because it is the one that never appears on a slide. Somebody decided what counted as a successful attack. If that somebody was a language model acting as a judge, the report is stacking one stochastic system's opinion on top of another's, and the honest version tells you how that judge was calibrated against human labels. If it was a keyword rule, it will have missed everything phrased differently and flagged plenty that was harmless. Neither is disqualifying. Both need saying.
The budget is the one that moves numbers the most and gets stated the least. HarmBench found that token budget drastically impacts attack success rate and is not standardised across papers, which means two rates computed at different budgets are not on the same scale at all. Promptfoo's own documentation says the same thing plainly: attack success rate depends heavily on attempt budget, prompt set composition and judge choice.
Why comparing across vendors is usually an invalid operation
This is the part leaders most want to do and most need to stop doing. Chouldechova, Cooper, Barocas, Palia, Vann and Wallach, at NeurIPS 2025, examined the practice directly and found that conclusions drawn about the relative safety of systems are often not supported by the evidence that ASR comparisons provide. The comparisons rest either on apples-to-oranges setups or on measurements with low validity for the thing being claimed. Anthropic has made a compatible point from the vendor side: inconsistent methods across the industry make objective cross-system safety comparison difficult.
NIST goes further and says it about benchmarks generally. In section 4.1.3 of the adversarial machine learning taxonomy, benchmarks are described as unreliable, with results routinely incomparable. New mitigations "appear very powerful, but are quickly shown to lack robustness to unforeseen types of attacks", which is why a mitigation has to be tested against unforeseen attacks rather than the ones it was designed against. And NIST's instruction on how to present the outcome is one to steal outright: attributes must be evaluated simultaneously on a Pareto plot, because a mitigation that improves one axis while degrading another is not an improvement, it is a trade you have not priced.
| Comparison | Is it defensible? | Why |
|---|---|---|
| Our system this quarter versus our system last quarter | Yes, with conditions | Only if the configuration is pinned: same attack spec, same budget, same N, same judge. Otherwise you are measuring the change in your testing. |
| Our system before a mitigation versus after | Yes | This is the strongest use of a rate. Report the delta and the overrefusal counter-metric together, on a Pareto plot rather than as a single headline. |
| Vendor A's published ASR versus vendor B's | No | Different classifiers, budgets, trial counts and attack sets. NeurIPS 2025 finding: the safety conclusions are often not supported by the comparison. |
| Our system versus a published benchmark leaderboard | No | NIST: benchmark results are routinely incomparable, and a mitigation must be tested against attacks it was not designed to stop. |
The grilling kit, run on a real report ★ 12 min · in groups of three
Bring a red-team or AI safety report you have actually received, or use the Beacon engagement summary. Work through the eight questions in order and mark each one answered, partially answered or absent. The count of absent answers is your read on the report's usefulness, and it is usually the first time anyone in the room has scored a report rather than reacted to it.
1 · What was N, and when was it fixed? If there is no N, every number on the page is a sample of one dressed as a measurement. If N was set after results were seen, ask why that number. NIST's 57% to 80% result is the reason this question comes first.
2 · What endpoint was tested? The base model through an API, your application through its production interface, a pre-mitigation build, a post-mitigation build? Microsoft's guidance is explicit that reports must make clear which endpoints were used. A finding against a raw model endpoint and a finding against the shipped product are different claims.
3 · Was this adaptive, or baselines only? Ask how much of the budget went to attacks tailored to this system versus published sets run unmodified. The 11% to 81% gap is what this question is measuring. A report that only ran baselines should say so in the summary, not the appendix.
4 · Who graded success, and were they calibrated? Named human adjudicator, model judge, keyword rule? If it was a judge model, what is its agreement rate with human labels on this task, and who resolved disputes? An ungraded grader is a silent multiplier on every number in the document.
5 · What was the attacker budget? Tokens, turns, wall-clock time, or attempts per objective. HarmBench found budget drastically impacts the rate. Two rates at different budgets are not on the same scale, including two of your own from different quarters.
6 · Was there a benign set, and what happened to it? Without an overrefusal counter-metric, a team can improve every security number by making the system refuse more, and the report will read as progress. This is not hypothetical: the instruction-hierarchy work that produced up to a 63% robustness gain also documented over-refusal regressions.
7 · What was explored, and what was not? This is the question most reports answer worst and the one your risk register needs most. An untested class is not a low-risk class. It goes on the board as unknown, at full inherent risk, never as safe.
8 · What build, on what date? Every finding is scoped to a build. A model version change, a system-prompt change, a new tool or connector, a new renderer or a permission-scope change all invalidate it. Ask which of those have happened since the testing window closed.
Rewrite the executive summary you would sign ★ 6 min · in pairs
You are the accountable executive for Beacon's Support Bot. Your testing team hands you this line: "Prompt injection: mitigated. Attack success rate 3%." In pairs, rewrite it into a sentence you would be willing to put your name to in front of a regulator, using only what a report could actually establish.
Strip the verdict. "Mitigated" is a property claim. Nothing in this field supports property claims. What the work can support is: these specific attacks succeeded at these measured rates under these conditions against this build at this date.
Attach the conditions. The endpoint, N, the budget, the judge, whether the work was adaptive. If any of those are missing from the underlying report, your rewritten sentence must say they are missing. That absence is information the reader is entitled to.
Add the unexplored column. Name at least one class that was not tested, and state it at full inherent risk. A summary that only lists what was found reads as a completeness claim whether or not it means to.
Read your version out loud and ask the room the honest question: does it sound worse than the original? It will. That is the correct outcome, and it is the whole reason this session exists. The original was not better news, it was less information.
Try it yourself - this week ◐ 45-60 min total
- Take the last AI safety or red-team report your organisation received, internal or from a vendor, and score it against the eight questions. Count the absent answers. Send the count, not the critique, to whoever commissioned it.
- Find every place in your current reporting where an AI finding is recorded as a boolean, and change the field to k of N with a stated N. This is usually a one-line change to a template and the highest-leverage thing on this page.
- Ask your team for the overrefusal counter-metric. If there is not one, you currently have no way to distinguish a system that got safer from a system that got more useless, and both look identical in the security numbers.
- Write down, in one line, the last cross-vendor safety comparison your organisation made a decision on. Then write down which of the three variables you knew for each side. Usually the answer is none of them.
- Add "what was not explored" as a required section in your report template, and make it the first section after the summary rather than the last one before the appendix.
Official sources covered
Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A vendor's report shows a near-zero attack success rate against their new defence. What is the correct reading?
Option C is the seductive one, and it is still wrong, because the twelve defences in the study were broken with techniques the field already had: gradient descent, reinforcement learning, random search and human-guided exploration. Nothing exotic was required, only somebody moving second with the defence in front of them. The reported rate measures the author's attack set and budget, not the defence. Zhan et al. found the same shape for agents, bypassing all eight evaluated indirect-injection defences at consistently above 50%.
2 · Your team wants to compare your system's attack success rate against a competitor's published figure. What do you tell them?
Option A fails because the same benchmark name does not fix the three variables: the judge, the attacker budget and N can all differ inside a shared benchmark, and HarmBench found budget alone drastically moves the rate. The defensible use of an attack success rate is as a pinned-configuration trend inside your own system over time, published with the attack spec, the budget, N, the judge, an overrefusal counter-metric and a regression pass rate. If a stakeholder wants the cross-vendor number, the honest answer is that the field cannot currently produce it.
3 · A report tests fourteen attack classes and finds nothing in six of them. How should those six appear on your risk register?
"Found nothing" and "was not looked at" collapse into the same visual on almost every dashboard, and that collapse is where risk registers go wrong. This is exactly why "what was explored and what was not" belongs immediately after the executive summary rather than in an appendix. The board reports unknown, never safe, and an untested class carries its full inherent risk until somebody spends effort on it.