learn-ai-red-team-with-phoebe / Builder session 6 of 10
Learn AI Red Team with Phoebe · Builder track · Session 6 of 10

Tool abuse and agency: severity is what the credential can do

This is the agentic session, and it is the one where the findings stop being about text. Once a model can call a tool, the interesting question is no longer what it said but what ran, under whose credential, and whether anybody could undo it. This session builds the permission manifest that bounds that, runs the test that needs no attacker at all, and makes the structural point about MCP security that changes who in your organisation owns this work.

🟠 Builder track Security engineers · AI engineers · platform leads 45 min
0-3 · Welcome 3-22 · Excessive agency, the technique map, MCP 22-40 · Build-along: manifest and pre-deployment checks 40-45 · Q&A
Part 0

Where the model stops being a component and becomes an actor

⚠ Authorization This session is about testing AI systems you own, or are contracted in writing to test, inside the scope document you wrote in b1. It names attack classes so you can build test cases and pairs each with the defence that addresses it. There are no payloads, no injection strings and no attack craft on this page. Note b1's blast-radius rule especially here: if confirming a finding would require actually issuing the refund, sending the mail or deleting the record, you record the finding and you do not fire the action. Testing a system you are not authorized to test is a crime in most jurisdictions.

OWASP draws the boundary cleanly: the LLM Top 10 owns risk while the model is a component inside your application, and once it becomes an actor - tools, cross-session memory, downstream consequences - risk moves to the Agentic list. Neither list covers the ground alone. The framework data agrees: in ATLAS collection 2026.07, Agentic AI is now the largest platform tag, with 119 techniques carrying it against 92 for generative AI and 70 for predictive AI.

Live - presented in session Self-study - read after class Official sources covered
★ What you walk out with today A tool permission manifest for Beacon with every tool marked reversible or irreversible and every credential scoped to a task rather than to a system, the hallucination-alone test run against all three bots, an MCP consent and token check you can hand to whoever operates your servers, and the two pre-deployment heuristics - the lethal trifecta and the Rule of Two - applied to a real workflow.
Where this sits in the track b3 and b4 covered getting the model to do something. b5 covered getting the system to reveal something, and ended on the discovery of tool definitions and call chains, which is the reconnaissance for this session. b7 turns all of it into a suite with measured coverage.
Part 1 · the root causes

Excessive Agency, and the test that needs no attacker 7 min live

LLM03:2026 Excessive Agency moved up three places in the 2026 list, from LLM06 in 2025, and the reason is simply that more applications now have tools than had them a year ago. OWASP splits it into three root causes, and the split is useful precisely because each one has a different owner and a different fix.

Three root causes, three different owners, three different fixes. Excessive functionality The agent can call tools the task does not need, or tools that do more than the task needs them to do. Test Remove it. See what breaks. Excessive permissions The tool's credential can do more than the agent needs - write where read would do, all tenants where one would. Test Whose credential, and what else does it reach? Excessive autonomy The action fires with no human and cannot be undone. Autonomy is only safe where undo is cheap. Test List every irreversible tool. That list is the risk. The test that matters, and it needs no attacker at all Take the adversary out of the picture entirely. If the model simply gets something wrong - and it will, because that is what these systems do - can that mistake ALONE trigger a damaging action? If the answer is yes, the finding exists today, with nobody attacking you, and its name is excessive agency.
🔍 Click to zoom - the three causes, and the one question that does not require an adversary

Run that last test first, in every engagement, because it is the cheapest finding you will ever write and the hardest to argue with. There is no attack chain to reproduce and no k of N to defend: you are simply asking whether a known property of the system - that it is sometimes wrong - is sufficient to cause harm. Anthropic's own disclosure of an AI-run espionage campaign contains the same observation from the attacker's side, noting the model "occasionally hallucinated credentials". The attacker's tooling was unreliable in exactly the way yours is. Build for that, not for the adversary you imagine.

Part 2 · the technique map

What ATLAS has actually observed, and where the map runs out 7 min live

Use the maturity field as a test-prioritisation axis: Realized means observed in the wild, and it is where an engagement with finite time should start. Three of the techniques below are Realized, and two of those have no ATLAS mitigation mapped at all.

IDTechniqueTacticMaturity
AML.T0053AI Agent Tool InvocationExecution, PrivEsc, Lateral MovementDemonstrated
AML.T0086Exfiltration via AI Agent Tool InvocationExfiltrationRealized
AML.T0101Data Destruction via AI Agent Tool InvocationImpactRealized
AML.T0080 (.000 Memory, .001 Thread)AI Agent Context PoisoningPersistenceDemonstrated
AML.T0110AI Agent Tool PoisoningPersistenceRealized - no mitigation mapped
AML.T0109AI Supply Chain Rug PullDefense EvasionRealized - no mitigation mapped
AML.T0077LLM Response RenderingExfiltrationDemonstrated - no mitigation mapped

Two of those deserve a sentence each, because they break the mental model most people arrive with. Tool poisoning and rug pull are not attacks on your model or your prompt at all: they are attacks on the definitions your agent trusts, where a tool that behaved correctly during review changes behaviour afterwards. Your test for these is not a probe, it is a supply-chain control - pinning, review of changed definitions, and alerting when a tool description changes under a running agent. Response rendering is the egress leg: a client that renders whatever the model returns, as an image or a link or markup, is a channel out. GitHub's durable fix for the Copilot Chat image-exfiltration finding in August 2025 was to disable image rendering in Copilot Chat entirely, which is the shape of every fix that has actually held.

Follow the chain to the end. The severity is at the end, not at the start. Untrusted content reaches the agent The agent decides to call a tool The tool runs with a credential you gave What that credential can do IS the severity Steps 1 and 2: you cannot fix these You cannot make an agent immune to being convinced. Twelve recent defences, most originally reporting near-zero success, were later bypassed above 90%. All eight evaluated indirect-injection defences for agents were bypassed, consistently above 50%. Steps 3 and 4: you can bound these Scope the credential to the task. Restrict which tools untrusted data may reach at all. Put a human on the irreversible ones. Segment so a compromised agent reaches one tenant. None of this makes the agent harder to fool. It makes being fooled cheap, which is the only version anybody has achieved. Say this part out loud, because a framework rarely admits it ATLAS maps NO mitigation at all to several Realized techniques on this page: AML.T0110 tool poisoning and AML.T0109 rug pull, plus AML.T0077 response rendering. That is the residual-risk column.
🔍 Click to zoom - two halves of the chain, only one of which you can do anything about
LiveThe same ground on the OWASP Agentic list3 min▶

OWASP released its Top 10 for Agentic Applications in December 2025. The mapping below is the one to keep next to the ATLAS table, because ATLAS tells you what has been observed and the ASI list tells you what to design against.

ASIWhat it coversNearest ATLAS technique
ASI02 Tool Misuse and ExploitationThe tool does something the workflow never needed it to doAML.T0053, AML.T0086, AML.T0101
ASI03 Identity and Privilege AbuseWhose credential the tool call runs under, and what else it reachesAML.T0053 under PrivEsc
ASI06 Memory and Context PoisoningSomething written once that changes behaviour laterAML.T0080, .000 Memory, .001 Thread
ASI07 Insecure Inter-Agent CommunicationOne agent trusting another agent's output as instructionAML.T0051 family, applied between agents
ASI08 Cascading FailuresOne bad step becoming many, because each step trusts the lastAML.T0080 plus call chains from AML.T0084.003
ASI09 Human-Agent Trust ExploitationThe approval that gets clicked because it always gets clickedThe failure mode of M0029, discussed in Part 3
ASI10 Rogue AgentsAn agent running outside the inventory anybody is watchingAML.T0109, AML.T0110

An honesty note on ASI01. This course does not state its final title. OWASP's own repository file is ASI01_Agent_Behaviour_Hijack.md and the December announcement says "Agent Behavior Hijacking", while secondary coverage of the final release says "Agent Goal Hijack". Until the released PDF is checked, write it as ASI01 (agent behaviour / goal hijack) and do not put either title in a report as if it were settled. The same caution applies more weakly to the exact final wording of ASI02 to ASI10, which come from draft filenames plus agreeing secondary summaries.

Part 3 · MCP, and what actually bounds this

Almost none of it is AI-specific 8 min live

Here is the structural point of the session, and it changes who owns the work. Read the documented Model Context Protocol threats one by one and notice what class each belongs to. They are OAuth problems, SSRF problems, session problems and supply-chain problems, arriving in a new component with a new name.

The documented threat, what it actually is, and the control that is already written down. Confused deputy (proxy servers) An OAuth problem Proxy servers MUST get per-client consent. Exact-match redirect URI, no wildcards; single-use state value. Token passthrough An audience problem Servers MUST NOT accept a token that was not issued for them. Validate the audience on every request. SSRF on OAuth metadata discovery An SSRF problem Block private and reserved ranges on discovery fetches, including 169.254/16 cloud metadata. Session hijacking A session problem Sessions MUST NOT be used for authentication. Bind the session to a user and verify the credential per call. Over-broad scopes A least-privilege problem Request the narrowest scope the workflow needs, and re-request rather than hold a broad one. Read the middle column again Almost none of the documented MCP threats are AI-specific. They are OAuth, SSRF, session and supply-chain problems arriving in a new component, which is Google's lesson - that traditional security controls significantly mitigate many of these attacks - in its most concrete available form.
🔍 Click to zoom - five threats, five familiar classes, five controls somebody in your org already knows

The practical consequence is a staffing decision rather than a research programme. If your MCP review is being run entirely by the AI team, it is being run by the wrong people, or at least not only by the right ones. Google's published lesson from its own AI red team says traditional security controls significantly mitigate many attacks, and this is what that sentence looks like when it reaches an implementation. Pair an application-security reviewer with an AI engineer, hand the first one the OAuth and SSRF rows, and the review gets better the same afternoon.

Real world

The Amazon Q Developer incident is the cleanest published illustration of an old control failing in a new place. AWS's own security bulletin gives the root cause as "an inappropriately scoped GitHub token in their CodeBuild configuration", which let an outside actor land code in a release. The injected instruction told the assistant to wipe resources. It did not execute, the bulletin notes, "unsuccessful in executing due to a syntax error" - which is luck, not a control. The failure was token scope. The AI was the payload's delivery vehicle, not its cause.

The defences that actually bound this

ATLAS names them, and they line up exactly with the right-hand half of the chain in Part 2. None of them makes the agent harder to fool. All of them make being fooled cheaper.

MitigationWhat it doesWhere it bites
M0026 / M0027 / M0028Permission configuration across the agent, its tools and its dataThe credential at step 3. The highest-leverage control in this list
M0029 Human In-the-LoopA person approves before the action firesIrreversible tools only. See the honesty note below
M0030 Restrict Tool Invocation on Untrusted DataContent from an untrusted source cannot reach the tools that matterCuts the chain between step 1 and step 2, which is the only place it can be cut
M0031 Memory HardeningBounds what can be written to memory and what memory can causeAML.T0080 and ASI06. Persistence is the difference between an incident and a recurring one
M0032 SegmentationA compromised agent reaches one tenant, one dataset, one environmentBlast radius. This is what turns a Critical into a High
Honesty note on human-in-the-loop, and on what we do not know M0029 degrades under approval fatigue: a human who approves forty prompts a day is a rubber stamp with a salary, and OWASP concedes the point directly - which is also what ASI09 Human-Agent Trust Exploitation is about. What nobody can currently give you is a number. This course's source map found no primary study of approval-fatigue rates for AI agents; every available source is a vendor or practitioner blog. So treat it as a well-founded hypothesis rather than a measured quantity, and if you need supporting literature, cite the general security-warning habituation research and label it explicitly as a non-AI analog. The design consequence stands either way: put approval on the few irreversible actions, not on everything, because an approval queue you can clear without reading is worse than no queue at all.
Demo 1 of 2

Beacon's tool permission manifest ★ 12 min · everyone builds

Support Bot is the T3 bot that can issue refund credits, so it is the manifest to write first. This artefact is the input to nearly every severity decision for the rest of the engagement, and most teams have never written it down at all. One row per tool, and the two columns everyone forgets are reversible and untrusted_data_may_invoke.

beacon/agents/support-bot.tools.yaml# One row per tool. If you cannot fill every field, the tool is not reviewed. # Maps to ATLAS M0026/27/28 permission configuration and M0030. agent: support-bot credential_principal: svc-support-bot # NOT the calling user, NOT a shared admin segmentation: tenant-scoped # M0032 - one tenant per session tools: - name: lookup_order scope: read:orders:own-tenant reversible: true untrusted_data_may_invoke: true # safe: read-only, tenant-scoped human_approval: false - name: issue_refund_credit scope: write:credits:max-50-per-day # least privilege, with a ceiling reversible: false # this is the column that sets severity untrusted_data_may_invoke: false # M0030 - untrusted input cannot reach this human_approval: required # M0029 - and ONLY on tools like this one - name: send_customer_email scope: write:mail:templated-only # no free-text body = no egress channel reversible: false untrusted_data_may_invoke: false human_approval: required renderer: images: disabled # AML.T0077 - the egress leg, closed links: text-only-no-auto-fetch

Fill the reversible column first, for every tool. This is the b1 homework arriving with interest. Reversible tools can tolerate autonomy; irreversible ones cannot, and the whole of M0029 should be spent on that short list rather than spread across everything.

Name the credential principal explicitly. If the answer is "it runs as the calling user", you have inherited every permission that user has, including the ones the workflow never needed. If the answer is a shared service account with broad rights, step 4 of the chain is as wide as that account, and that is your severity.

Set untrusted_data_may_invoke honestly. Any tool reachable while untrusted content is in context is reachable by that content. This is M0030 written as configuration rather than as a hope, and it is the one place the chain between steps 1 and 2 can actually be cut.

Now run the hallucination-alone test. For each irreversible tool, ask: if the model simply invents a reason to call this, with nobody attacking, what happens? On Beacon's Support Bot the answer was a credit issued to the wrong account, which is a finding with no adversary in it and no k of N to argue about.

Close the renderer. AML.T0077 has no ATLAS mitigation mapped, and the durable real-world fix has been to remove the channel: GitHub disabled image rendering in Copilot Chat outright. Beacon does the same, and the manifest is where that decision is recorded so that a future feature request has to argue with it.

Why the manifest is the deliverable and the transcript is not A finding that says "we got the agent to call the refund tool" ages out at the next prompt change. A manifest row that says this tool is irreversible, holds a write-scoped credential with a daily ceiling, is unreachable from untrusted content, and requires approval, is a control that survives the next three model upgrades. Write the second one.
Demo 2 of 2

The MCP check, and the two pre-deployment heuristics ★ 8 min · in pairs

Two artefacts to finish on. The first is the review checklist for any MCP server your agent talks to, written as assertions rather than as prose so that it can be automated. The second is the pair of heuristics that decide whether a workflow should ship as an agent at all.

beacon/checks/mcp_review.py - assertions, not prose# Every assertion below is a general security control, not an AI control. # Failing any one of them is a finding regardless of what the model does. def review(server, client, request): # Confused deputy: a proxy server must not consent on a client's behalf assert server.obtains_consent_per_client(client) assert server.redirect_uri_match == "exact" # no wildcards, ever assert request.state.single_use and request.state.stored_after_consent # Token passthrough: audience is not optional assert request.token.audience == server.identifier # SSRF on OAuth metadata discovery assert server.discovery_fetch.blocks_private_and_reserved_ranges assert "169.254.0.0/16" in server.discovery_fetch.denied # cloud metadata # Sessions are not an authentication mechanism assert not server.authenticates_by_session assert server.verifies_credential_per_call # Scope minimization assert server.requested_scopes == workflow.minimum_scopes

Run the checklist against one real MCP server, in pairs, four minutes. Then note which of the six assertions your AI team could have answered and which needed the application-security team. That split is the staffing argument from Part 3, made with your own system.

Apply the lethal trifecta. Simon Willison's heuristic: access to private data, exposure to untrusted content, and the ability to communicate externally. Any one alone is safe. All three together mean attacker text can move your data out, and removing any one leg removes the risk. Walk Beacon's three bots through it and write down which leg you would cut on each.

Apply the Rule of Two. Meta's formulation, which OWASP cites as a floor: treat simultaneous (A) untrusted input, (B) sensitive data and (C) state change or external communication as high risk, and require per-action human approval for an agent holding all three. Note OWASP's own caveat while you use it: the rule is silent on autonomy depth, so an agent with three legs and forty sequential steps scores the same as one with three legs and two steps, which is a real gap rather than a quibble.

Write the sentence that closes the session. For each Beacon bot: this workflow is safe to run as an agent because the credential can only do X, or this workflow should not run as an agent, which is a legitimate finding rather than a failure. NCSC's phrasing is the one to borrow when it lands that way: if the system's security cannot tolerate the remaining risk, it may not be a good use case for LLMs.

★ The line to leave the room with You cannot make the model refuse to be fooled. You can bound what being fooled gets an attacker. Every durable real-world fix in this course's incident list cut the egress leg or the untrusted-content leg, and not one of them fixed the model.
Homework

Try it yourself - this week ◐ 45-60 min total

Source material

Official sources covered

Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. This page covers:

✓OWASP LLM03:2026 Excessive Agency (up three places from 2025's LLM06) and its three root causes: excessive functionality, permissions, autonomyPart 1 · OWASP GenAI Top 10, 2026 edition · [P]
✓MITRE ATLAS AML.T0053, AML.T0086 (Realized), AML.T0101 (Realized), AML.T0080 with .000 Memory and .001 Thread, AML.T0110 and AML.T0109 (Realized, no mitigation mapped), AML.T0077 (no mitigation mapped)Part 2 · ATLAS collection 2026.07, format-version 6.0.0 · [P]
✓ATLAS platform tags: Agentic AI 119, Generative AI 92, Predictive AI 70 - Agentic AI is now the largest tagPart 0 · cited from the machine-readable dataset, not the rendered site · [P]
✓OWASP Agentic Top 10: ASI02, ASI03, ASI06, ASI07, ASI08, ASI09, ASI10 mapped to ATLAS techniquesPart 2 card · released 9-10 Dec 2025 · [P] with the caveat below
✓MCP security: confused deputy and per-client consent, exact-match redirect URI, single-use state; token passthrough and audience validation; SSRF on metadata discovery including 169.254/16; sessions MUST NOT authenticate; scope minimizationPart 3 · [P]
✓ATLAS mitigations M0026/27/28, M0029 Human In-the-Loop, M0030 Restrict Tool Invocation on Untrusted Data, M0031 Memory Hardening, M0032 SegmentationPart 3 · [P]
✓Bypass evidence: 12 recent defences bypassed above 90% ASR (Nasr, Carlini et al. 2025); all 8 evaluated indirect-injection defences for agents bypassed, consistently above 50% (Zhan et al., NAACL 2025)Part 2 figure · [P]
✓Amazon Q Developer: AWS Security Bulletin AWS-2025-015, root cause an inappropriately scoped GitHub token; GitHub disabling image rendering in Copilot Chat, 14 Aug 2025; Google's "traditional security controls significantly mitigate" lessonParts 2-3 · [P] · no CVE is cited for either incident, per the source map's corrections
✓Lethal trifecta (Willison) and Meta's Rule of Two, with OWASP's caveat that the rule is silent on autonomy depthDemo 2 · [P]
◐ASI01's final titleCOULD NOT VERIFY. Repo file says "Agent Behaviour Hijack", the December announcement says "Agent Behavior Hijacking", secondary coverage says "Agent Goal Hijack". Written here as ASI01 (agent behaviour / goal hijack) only
◐Approval-fatigue rates for AI agentsCOULD NOT VERIFY. No primary study located; all sources are vendor and practitioner blogs. Taught as a hypothesis, with the general security-warning habituation literature named as an explicitly non-AI analog
◐The manifest schema and the MCP assertion fileAssembled for this course from the controls above · no published standard manifest format exists; treat as a starting structure. Beacon values are illustrative
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · Which excessive-agency test can you run before any attacker is involved, and why is it the one to run first?

A and B are both real tests and both cost time, need N trials, and produce findings someone can argue with. The hallucination-alone test costs a reading of your own tool list and produces a finding that cannot be disputed, because it relies only on a property everyone already agrees the system has: it is sometimes wrong. If a plain mistake can issue a credit or delete a record, that is excessive agency today, and the adversary is a distraction.

2 · Your review of an MCP server finds no per-client consent in the proxy, a token accepted without audience validation, and discovery fetches that reach link-local addresses. What kind of findings are these?

The tell is that every control is already written down somewhere older than the protocol: exact-match redirect URIs, single-use state stored only after consent, audience validation, and blocking private and reserved ranges including 169.254/16 for cloud metadata. Treating these as AI problems routes them to the team least likely to have the relevant muscle memory, which is the staffing failure this session exists to prevent.

3 · You propose human approval on every agent action as the fix. What is the strongest honest objection?

C is simply false: M0029 Human In-the-Loop is a named ATLAS mitigation, and a good one when it is aimed. A is true and is a product argument rather than a security one. B is the security argument, and the honesty matters as much as the point: OWASP concedes that human-in-the-loop degrades under fatigue and ASI09 Human-Agent Trust Exploitation is built on it, but the specific rates circulating in vendor blogs have no primary study behind them. Say the mechanism, decline to quote a number, and put the approvals where the irreversible actions are.

Builder session 6 cheat sheet · pin this

LLM03:2026 Excessive AgencyUp three places. Three root causes: excessive functionality, excessive permissions, excessive autonomy.
The test with no attackerCan a hallucination ALONE trigger a damaging action? If yes, the finding exists today.
Realized firstT0086 exfiltration and T0101 data destruction via tool invocation are Realized. Test those before the theoretical ones.
Where the map runs outT0110 tool poisoning and T0109 rug pull are Realized with no ATLAS mitigation mapped. T0077 rendering likewise.
Severity is the credentialSteps 1 and 2 of the chain cannot be fixed. Steps 3 and 4 can be bounded. That is the whole job.
MCP is not an AI problemPer-client consent, exact-match redirect URI, audience validation, block 169.254/16, sessions never authenticate, minimum scopes.
M0030 over M0029Restrict which tools untrusted data may reach; reserve human approval for irreversible actions, because approval degrades under fatigue.
Trifecta and Rule of TwoPrivate data + untrusted content + external comms. Cut a leg. [A,B,C] needs per-action approval, and the rule is silent on autonomy depth.