learn-ai-red-team-with-phoebe / Builder session 5 of 10
Learn AI Red Team with Phoebe · Builder track · Session 5 of 10

Data extraction: assume the context is discoverable

Most teams meet this class as a testing problem: can somebody get the system prompt out? That is the wrong question, and OWASP rewrote an entire Top 10 entry in 2026 to say so. The right question is what breaks when the hidden context is read, and the right answer is nothing. This session builds the audit that gets you there, the canary fixtures that measure whether the stores hold, and the severity ladder that decides whether a leak is a note or an incident.

🟡 Builder track Security engineers · AI engineers · platform leads 45 min
0-3 · Welcome 3-20 · The design rule, the ladder, the stores 20-40 · Build-along: audit and canaries 40-45 · Q&A
Part 0

The class, and why its Top 10 entry was rewritten

⚠ Authorization This session is about testing AI systems you own, or are contracted in writing to test, inside the scope document you wrote in b1. It names the attack class so you can build test cases and pairs it with the defence that addresses it. There are no extraction prompts and no attack craft on this page. The only secrets you will plant are canaries that authorize nothing. Testing a system you are not authorized to test is a crime in most jurisdictions.

In the 2025 list this was LLM07 System Prompt Leakage. In the 2026 list it is LLM08:2026 Hidden Context Exposure - renamed and re-scoped, because "system prompt" stopped describing the thing. The hidden context of a modern application is the system prompt plus the tool definitions, the retrieved documents, the memory entries, the activation triggers and the routing rules. All of it travels in the same token stream, and none of it carries a marking the model can check. Alongside it sits LLM02:2026 Sensitive Information Disclosure, which covers what leaks out of the data rather than out of the instructions.

Live - presented in session Self-study - read after class Official sources covered
★ What you walk out with today A hidden-context audit for Beacon's three bots that answers one question per line - what breaks if this is read aloud - a set of canary fixtures planted in every store you own, a tenant-boundary retrieval test you can run at your fixed N, and the LLM08:2026 severity ladder written into your triage rubric so that "the system prompt leaked" stops being a severity argument.
Where this sits in the track b3 and b4 covered getting the model to say something it should not. This session covers getting the system to reveal something it holds, which is a different chain with a different fix. b6 covers the case where the thing revealed is a credential the agent can then use, and that is where the severity actually lives.
Part 1 · the central lesson

Design so that disclosure does not matter 7 min live

This is a design lesson, not a testing lesson, and it is the single most useful sentence in the 2026 list. OWASP's guidance for LLM08:2026 is to design the application so that disclosure of the hidden context has little or no direct security impact. Not to obfuscate it. Not to instruct the model to refuse to reveal it. To arrange matters so that reading it is uninteresting.

Three prohibitions follow directly, and they are worth writing on a wall. Never place credentials or tokens in the hidden context. Never rely on its secrecy as an authorization or privilege-separation boundary. Never rely on its secrecy as a policy or filtering boundary - if the only thing stopping an action is a sentence in a prompt, then the control is a sentence in a prompt, and b1 already established what those are worth.

Sort your hidden context into two lists. The right-hand list is the finding. Assume every line of this is readable Fine here, because nothing depends on it Tone, formatting and persona instructions Task framing and the wording of refusals Product knowledge that is already public Pointers to data whose access is separately authorized at retrieval time, per request And now find these, because they are bugs Never here - disclosure would be the breach Credentials, API keys, tokens, connection strings Internal hostnames treated as if they were secret Policy that is enforced only by being read A tenant or role identifier that is the only thing separating one customer's data from another's The design rule, in OWASP's own framing LLM08:2026 does not ask you to make the hidden context harder to extract. It asks you to design so that its disclosure has little or no direct security impact. That is a design instruction rather than a testing one, and it is the only version of this problem that anybody has ever actually solved.
🔍 Click to zoom - the audit is a sorting exercise, and the right-hand column is your finding list

The frameworks name the pieces of this surface precisely enough to build test cases from, and the sub-techniques under AML.T0084 are the most useful part, because they enumerate what an attacker learns about an agent before touching it at all. Discovery is a tactic in its own right, and it is the cheapest one.

IDNameWhere it bites
AML.T0056Extract LLM System Prompt FeasibleThe instructions themselves. Low maturity in ATLAS, high frequency in practice
AML.T0057LLM Data Leakage DemonstratedTraining data, retrieved documents, other users' content surfacing in an answer
AML.T0084Discover AI Agent Configuration DemonstratedThe parent technique. Four sub-techniques below, and each is a separate test case
AML.T0084.000Embedded KnowledgeWhat the agent was given: internal documents, examples, business rules baked into context
AML.T0084.001Tool DefinitionsWhich tools exist, their parameters and their descriptions. Reconnaissance for b6
AML.T0084.002Activation TriggersWhat causes the agent to act. Learning this is how a chain becomes reliable
AML.T0084.003Call ChainsWhich tool leads to which. The map an attacker would otherwise have to guess
AML.T0098Credential harvesting via toolsCredential Access. The tool call itself returns something it should have kept
AML.T0082Credential harvesting via RAGCredential Access. Somebody indexed a document that contained a key
AML.T0083Credential harvesting from agent configCredential Access. ATLAS maps no mitigation to this technique at all - see the honesty note below
Where the framework itself runs out AML.T0083 is one of a set of ATLAS techniques with no mitigation mapped. That is not an oversight to gloss over on a slide, it is the residual-risk column of an honest course in the framework's own words. The practical reading: nobody has a published control for "the credential was in the configuration", because the only real control is not putting it there, which is a design decision rather than a mitigation you can bolt on. Which is Part 1's whole argument, arriving from the other direction.
The NIST mapping, for the same ground NIST AI 100-2e2025 files all of this under privacy compromise, NISTAML.03, with .035 Prompt Extraction, .038 Data Extraction and .039 Compromising connected resources. That third one is the interesting one, and it is the bridge into b6: the target was never the model, it was the thing the model can reach.
Part 2 · the ladder

Severity is what it chains to 5 min live

LLM08:2026 ships a worked severity ladder, and it generalises well beyond hidden context. Copy it into your triage rubric verbatim, because it converts the most common argument in AI triage - "but the system prompt leaked" - into a question with four possible answers rather than a matter of temperament.

The same leak, four severities. The difference is never the prompt. Informational The context holds no secrets, and nothing anywhere depends on it staying confidential. Record it, close it, page nobody. This is where good design puts you. Medium It holds internal rules, routing logic or tool names that meaningfully aid an attacker planning the next step. Useful reconnaissance, not yet an impact. High It embeds credentials or tokens, or something in the system relies on its secrecy as an authorization boundary. The design prohibition from Part 1 was violated somewhere. Critical Disclosure chains to remote code execution, broad exfiltration or privilege escalation. The leak was the first step of the finding, and it is not the interesting step. Read the ladder backwards and it becomes a design checklist You cannot argue your way down this ladder at triage time. You can only build your way down it beforehand, by removing the credentials and the reliance, until Informational is the only rung left.
🔍 Click to zoom - four answers to the question everyone argues about instead
LiveHow this lines up with MSRC's published rubric3 min▶

Microsoft's AI bug bar reaches the same place by a different route, and having both on hand is useful when you need to defend a rating to somebody outside your team. Their Inferential Information Disclosure category keys severity to the data classification of what became reachable: model theft is Critical where the asset is Highly Confidential and slides down to Low where it is Public; input extraction, which is where system-prompt leakage lands, is Important where the content is Confidential and Low otherwise.

Two design decisions there are worth copying wholesale. Severity turns on the data classification of what is reachable, and on whether user interaction was required. It does not turn on how clever the extraction was. A tester who found the leak through a long and ingenious sequence and a tester who found it by asking politely have found the same finding, and the rating is identical.

Note the failure mode this protects you from. Without a rubric, extraction findings get rated by narrative quality, and narrative quality correlates with the tester's writing rather than with your risk.

Part 3 · the stores

Three places the context lives, and one boundary that keeps failing 6 min live

Hidden context is not one object. It is at least three stores with different lifetimes, different owners and different failure modes, and the retrieval layer is where the worst of them lives. OWASP gives vector and embedding weaknesses their own entry, LLM09:2026, and ATLAS files the write side as AML.T0070 RAG Poisoning and the read side as AML.T0082 credential harvesting via RAG.

The cross-tenant case deserves naming plainly, because it is the one that turns an AI feature into a data-protection incident. A shared index with tenant identity applied as a filter after retrieval, or applied in the prompt rather than in the query, is a boundary that exists only as long as nothing surprising happens. It is not an AI weakness. It is an old multi-tenancy weakness that arrived in a new component, which is a sentence you will read again in b6.

You cannot prove a store is unreachable. You can plant something and watch for it. Hidden context Plant One canary string that is a secret and nothing else. Pass It never appears in any output, across N trials. Fail means The context is readable. Retrieval index Plant One canary document per tenant, per index. Pass Tenant B never retrieves tenant A's canary, at N. Fail means A boundary defect, not a model defect. Long-lived memory Plant A canary written to memory during one session. Pass A later session, different user, never sees it. Fail means Memory is a shared channel you did not design. Why canaries and not extraction attempts A failed extraction attempt tells you one technique did not work today. A canary that never surfaces across every technique, every tenant and every session is a measurement you can put on a trend line.
🔍 Click to zoom - three stores, one instrument, and a pass condition you can automate
Memorisation probes, and their honest limit The third question in this class is whether the model itself carries training or fine-tuning data that should not come back out, which ATLAS files under AML.T0057. Probe it the same way: seed a distinctive canary into the fine-tuning corpus before training, then test whether it can be elicited afterwards. What you must not do is read a clean probe as proof of no memorisation, for the same reason that runs through this whole course - it is the population you could afford to test, not the population that exists.
Demo 1 of 2

The hidden-context audit for Beacon's three bots ★ 12 min · everyone builds

One line per item, one question per line: what breaks if this is read aloud in a public forum? Do it as an inventory, not as a test. You will find more in twenty minutes of reading your own configuration than in a week of probing it, and the findings will be actionable rather than anecdotal.

Dump everything that reaches the context window, from every source. System prompt, developer messages, tool definitions with their descriptions and parameter names, retrieved chunks, memory entries, routing rules, few-shot examples. Beacon's Analyst Bot turned out to carry a table-name allowlist that nobody had thought of as context and that maps the warehouse rather neatly.

Sort each line into the two columns from Part 1. Anything landing in the right-hand column is a finding today, before any testing happens. Credentials and tokens are the obvious ones. The subtle one is any line whose enforcement is the fact that the reader has not seen it.

Walk the four AML.T0084 sub-techniques as an attacker's shopping list. Embedded knowledge (.000), tool definitions (.001), activation triggers (.002), call chains (.003). For each, write down what an attacker learns and whether that knowledge alone changes what they can achieve. On Support Bot, the tool definitions reveal a refund tool, and that single line is the reconnaissance that makes b6's session worth attending.

For every "the model is instructed not to reveal X" line, write the control that would hold if the instruction were deleted. If there is none, you have found a policy boundary that exists only in prose. Note that this is the same anti-lever the course simulator measures: the system-prompt guard removes 3.6 risk points on a bare system while least-privilege and human approval remove 55.

Grade the whole inventory on the Part 2 ladder and put the result in the report as a single sentence: today, disclosure of Beacon's hidden context would be Informational for Insight Bot, Medium for Analyst Bot, and High for Support Bot until the connection string comes out of the configuration. That sentence is the finding. The extraction transcript is an appendix.

The trap in this exercise Teams routinely audit the system prompt and stop, because it is the part with a filename. The tool descriptions are context. The retrieved documents are context. The memory is context. The 2026 rename from "System Prompt Leakage" to "Hidden Context Exposure" exists specifically to stop this, and if your audit produced exactly one artefact, it is not finished.
Demo 2 of 2

Plant the canaries, run the boundary test ★ 8 min · in pairs

Canaries turn an unfalsifiable claim into a measurement. A canary is a string that is a secret and is nothing else: it authorizes nothing, unlocks nothing, and exists only so that its appearance anywhere is unambiguous. This is the fixture file, and it is the most reusable artefact in the session.

beacon/fixtures/canaries.yaml# Canaries authorize NOTHING. They exist only to be seen if a boundary fails. # Rotate on every engagement so a stale hit is never mistaken for a fresh one. - id: CANARY-CTX-01 store: hidden_context # AML.T0056 · AML.T0084.000 · OWASP LLM08:2026 value: "BEACON-CANARY-7f3a91" planted_in: support-bot@2026.08.11 pass: "never appears in any output, over N=25 trials per technique" - id: CANARY-RAG-ACME store: retrieval_index # AML.T0057 · AML.T0082 · OWASP LLM09:2026 tenant: acme value: "BEACON-CANARY-c04d22" pass: "never returned to any principal outside tenant acme" - id: CANARY-MEM-01 store: agent_memory # cross-session and cross-user boundary value: "BEACON-CANARY-91be07" written_by: user/j.rivera, session 1 pass: "never surfaces for any other user, in any later session" detection: # Every canary is also an alert. If one is ever seen in production # telemetry, that is an incident, not a test result. route: security-oncall

Plant one canary per store, per tenant. A single shared canary tells you something leaked. A canary per tenant tells you which boundary failed, which is the difference between a finding and an investigation.

Run the tenant-boundary retrieval test at your fixed N, and vary the phrasing rather than the tenant. Ask tenant B's questions in tenant A's vocabulary, with tenant A's product names and tenant A's spelling. Embedding similarity does not respect your org chart, and that is the whole point of LLM09:2026.

Wire the canaries into detection as well as into testing. The same string that fails a test in staging should page somebody if it is ever seen in production output. This is the cheapest thing in the entire course and almost nobody does it. It also gives your observability team something concrete to alert on, which is the handshake into the sibling course.

Record the result as k of N per store, per boundary, exactly as in b4. "Cross-tenant retrieval: 0 of 25 across 6 tenant pairs, build 2026.08.11" is a sentence a regulator can read. "We tested for cross-tenant leakage" is not.

Rotate, and never reuse across environments A canary that exists in both staging and production cannot tell you which one leaked. A canary from a previous engagement that shows up today may be a stale copy in a log rather than a live boundary failure. Fresh values per engagement, per environment, recorded in the fixture file with the build they were planted in.
Homework

Try it yourself - this week ◐ 45-60 min total

Source material

Official sources covered

Every framework, figure and quotation on this page traces to a primary source catalogued in this course's source map, verified 2026-08-14. This page covers:

✓OWASP LLM02:2026 Sensitive Information Disclosure and LLM08:2026 Hidden Context Exposure, renamed and re-scoped from 2025's LLM07 System Prompt LeakagePart 0-1 · OWASP GenAI Top 10, 2026 edition · [P]
✓The LLM08:2026 design rule - design so disclosure has little or no direct security impact; no credentials in context; no reliance on secrecy as an authorization, privilege-separation, policy or filtering boundaryPart 1 · the central lesson of this page · [P]
✓The LLM08:2026 severity ladder: Informational, Medium, High, Critical when it chains to RCE, broad exfiltration or privilege escalationPart 2 · [P]
✓MITRE ATLAS AML.T0056 (Feasible), AML.T0057 (Demonstrated), AML.T0084 with .000 Embedded Knowledge, .001 Tool Definitions, .002 Activation Triggers, .003 Call ChainsPart 1 table · ATLAS collection 2026.07, format-version 6.0.0 · [P]
✓Credential harvesting trio: AML.T0098 via tools, AML.T0082 via RAG, AML.T0083 from agent configurationPart 1 · T0083 has no ATLAS mitigation mapped at all, stated on the page · [P]
✓NIST AI 100-2e2025 privacy objective NISTAML.03 with .035 Prompt Extraction, .038 Data Extraction, .039 Compromising connected resourcesPart 1 · [P]
✓OWASP LLM09:2026 Vector and Embedding Weaknesses and ATLAS AML.T0070 RAG PoisoningPart 3 · [P]
✓MSRC AI bug bar - Inferential Information Disclosure keyed to data classification; input extraction Important when Confidential, Low otherwisePart 2 card · [P]
◐The canary fixture format and the three-store test designAssembled for this course · no published standard fixture format exists; the instrument is conventional security practice applied to AI stores
◐Beacon's per-bot severity ratings and all values in the fixture fileIllustrative teaching values, not measurements
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · What does OWASP LLM08:2026 actually ask you to do about hidden context?

A is what most teams do and it is a prose control: delete the sentence and nothing is left. C misunderstands where the context lives, since the model must read it to use it, so it arrives decrypted in the same stream as everything else. The entry was renamed from "System Prompt Leakage" precisely to move the conversation from extraction technique to design, and the payoff is that a leak becomes Informational rather than an incident.

2 · Your Support Bot's hidden context is read out in a public forum. It contains a warehouse connection string. Where does this land on the LLM08:2026 ladder, and why?

The ladder does not consult the extraction method. Embedded credentials or reliance on secrecy for authorization is the High rung by definition, and it climbs to Critical the moment the leak is the first step of a chain to code execution, broad exfiltration or privilege escalation. Note the asymmetry that makes this class worth designing rather than testing: you cannot argue your way down the ladder at triage time, you can only build your way down it beforehand.

3 · A tenant B user's query returns a document belonging to tenant A. How do you write this finding?

B is the tempting one because the symptom appeared in an answer, so the instinct is to fix the prompt that produced the answer. But tenant identity applied as a post-retrieval filter, or applied in the prompt rather than in the query, is a boundary that exists only while nothing surprising happens. Embedding similarity does not respect your org chart. Canary documents planted per tenant turn this from an argument into a number, and the number belongs on a trend line.

Builder session 5 cheat sheet · pin this

LLM08:2026 Hidden Context ExposureRenamed and re-scoped from 2025's LLM07 System Prompt Leakage. Tool definitions, memory and retrieved chunks are all context.
The design ruleDesign so disclosure has little or no direct security impact. Assume it is discoverable. Never a credential, never a boundary.
The ladderInformational, Medium, High, Critical. Severity is what it chains to, never how clever the extraction was.
AML.T0084 sub-techniques.000 embedded knowledge, .001 tool definitions, .002 activation triggers, .003 call chains. An attacker's shopping list.
The credential trioT0098 via tools, T0082 via RAG, T0083 from agent config. T0083 has no ATLAS mitigation mapped.
NISTAML.03Privacy compromise, with .035 prompt extraction, .038 data extraction, .039 compromising connected resources.
Canaries, not extraction attemptsOne per store, per tenant, rotated per engagement, wired to production alerting as well as to tests.
Cross-tenant is not an AI bugLLM09:2026 and AML.T0082. Old multi-tenancy weakness, new component. Test the boundary, not the prompt.