Nine sessions arguing for more telemetry. One session on the cost of that argument
Everything in this course has pushed one direction: instrument more, keep the attributes, log the tool arguments, capture what happened because you cannot replay it. This session is where that push meets data-protection law, security review, and a genuine trade-off that nobody gets to dodge. It is also the session where a vendor is on record admitting both halves of the problem about its own product, which is the most useful thing anyone has published on the subject.
One vendor, two published positions, and the split between them 7 min live
You could argue this trade-off from first principles for an afternoon. You do not have to, because the same organisation published both halves of it within a year: the cost of protecting content, and the pattern that works anyway. Read them next to each other. That pair is this session.
LiveWhat the metadata-only default actually costs you4 min▶
Most writing on this topic sells the privacy-preserving default as free. It is not free, and pretending otherwise is how you lose the argument with the engineer who has to debug at 03:00. Be precise about what you keep and what you give up.
| Metadata only, the default | With sampled content | |
|---|---|---|
| Where did it fail? | Answerable. Spans, durations, statuses and counts localise the failing step exactly | Answerable, identically |
| Was retrieval empty? | Answerable. Document count, ids and scores are metadata | Answerable, plus you can read what came back |
| Did it loop or truncate? | Answerable. Step counts, finish reasons, token counts | Answerable, identically |
| Why did the model do that? | Frequently not answerable. This is the real cost and it is the expensive half of debugging | Answerable on the sampled fraction, which is usually enough |
| Legal exposure | Low. Nothing personal left your boundary | Real, bounded, and manageable if it is consented, scoped and time-boxed |
Which is why the course's simulator ends the way it does. Switching on full content capture as the final lever moves retention from 4.6 GB to 48.1 GB, a 10.5x increase, flips the PII flag to true, and leaves the diagnosis score unchanged at 5 of 5, because the message content was never what was broken. Note the honest boundary of that lesson: it holds for this incident, which was a swallowed timeout and a stale cache. For a prompt-injection incident or a subtle instruction-following regression, content is the diagnosis. That is the case the sample exists to serve.
Six sensitive attributes, three patterns, and one inverted default 7 min live
The specification has already made most of these decisions for you, and it made them in the safe direction. The verbatim default: "OpenTelemetry instrumentations SHOULD NOT capture them by default, but SHOULD provide an option for users to opt in." Everything below is the shape of that opt-in.
Content lives in three Opt-In attributes, and three more are flagged sensitive. All six matter, and the second group is the one teams forget:
| Attribute | Status | What is actually in it |
|---|---|---|
gen_ai.input.messages | Opt-In | The user's words. Structured, with a mandatory JSON schema |
gen_ai.output.messages | Opt-In | The model's words, including anything it repeated back from the input |
gen_ai.system_instructions | Opt-In | Your prompt engineering. Rarely personal data, frequently a trade secret |
gen_ai.retrieval.query.text | Sensitive | What was searched for, which is often more revealing than the answer |
gen_ai.tool.call.arguments | Opt-In | The account number. The email address. The date of birth you passed to a lookup |
gen_ai.tool.call.result | Opt-In | The row that came back out of your own database |
gen_ai.input.messages, declares the trace store clean, and passes review. Meanwhile gen_ai.tool.call.arguments is still on, because it is the field that makes tool debugging possible, and the tool arguments are frequently where the account number is. The user asked a vague question in natural language; your agent turned it into lookup_customer(nric="S1234567A"). The structured, machine-readable, easily-indexed version of the personal data is on the tool span, not on the prompt.
Self-studyThe env-var nuance, precisely2 min read▶
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT is the de-facto name of the opt-in switch, and you will find it in most instrumentation libraries. Be careful how you say that in a design review, because the spec mentions it only as "for example". What is mandated is that an opt-in gate exists and defaults to off. The name is a convention that libraries happen to share.
The practical consequence: never assume a library is safe because you did not set that variable. Two failure modes follow directly.
- An instrumentation may honour a differently named flag, or a constructor argument, or a config file key. Grep for what the library reads rather than trusting the name.
- Handling of streaming chunks is a bare
TODOin the spec, so the behaviour of a streaming instrumentation with content capture is not specified. Test it rather than reasoning about it.
Verification is one span, not one document: run one real call through your pipeline in a staging environment and read the raw exported span. If prompt text, tool arguments or retrieval query text appear in it, your gate is not where you think it is.
Self-studyThe inverted default: OpenInference captures content unless you stop it3 min read▶
This is the single most consequential difference between the two dialects, and it is a privacy posture rather than a technical one. Both emit valid OTLP; every OpenInference trace is a valid OTLP trace. But:
- OTel GenAI: content capture is Opt-In and off by default. You must act to expose data.
- OpenInference: content is captured on by default. You hide it with the
OPENINFERENCE_HIDE_*flag family, and every one of those flags defaults to False. You must act to protect data.
Two details worth knowing because they show up in review. OpenInference redaction replaces the value with the literal string "__REDACTED__", which is genuinely useful: the field's presence is preserved, so you can still see that there was a prompt and that it was hidden, rather than wondering whether instrumentation was broken. And OPENINFERENCE_BLOB_UPLOADER offloads oversized media, which is the same idea as the OTel upload hook except that it is implemented rather than being a TODO.
The lesson generalises past these two libraries: read the default of every instrumentation you install, and assume nothing from the family it belongs to. If you migrate a dialect, or add one framework instrumented by a different project, your privacy posture can invert with no change to your own code.
Scrub before ingestion, not after 6 min live
Where you redact matters more than how well you redact. Once content reaches a vendor endpoint you have made a cross-border transfer and created a processor relationship, and deletion becomes a request rather than a fact. Everything downstream of that line is somebody else's promise. So the whole design question is: how much can you strip while keeping the trace useful, and can you do it before the exporter.
Real numbers, and three principles that bite a trace store 8 min live
Retention policy is usually written as a document and then quietly overridden by a technical detail nobody surfaced. Below are the published numbers from one provider, and then the three data-protection principles that apply to your trace store whether or not anyone has told your team they do.
| Surface | What the vendor documents [P] | What it means for your design |
|---|---|---|
| Conversation content | Prompts and outputs are not retained by default | The provider's default is already the good one. Your own trace store is usually the leak, not theirs |
| Zero Data Retention | Exists per organisation, but does not cover code execution, Claude for Excel, or Teams and Enterprise interfaces | ZDR is not a property of your account, it is a property of a path. Enumerate your paths |
| CORS under ZDR | Unsupported | A browser app must go through a backend proxy. This is an architecture decision disguised as a retention setting |
| Covered Models | Claude Fable 5 and Mythos 5 require 30-day retention, are ZDR-ineligible, and a ZDR org calling them receives 400 invalid_request_error | Your retention policy can be set by your model choice, not by your policy document. A model swap is a compliance change |
| Compliance surfaces | Activity Feed and remote session transcripts run to 6 years | Ask which of your surfaces are compliance surfaces before you promise anyone a 30-day deletion window |
| GDPR Art. 5(1) [P] | How it applies to a trace store |
|---|---|
| (c) Data minimisation | Capturing full prompt text on 100% of traffic "in case we need to debug" is the textbook violation. So sampling and redaction are the compliance mechanism, not merely a cost mechanism. That reframing is what gets the work funded |
| (e) Storage limitation | An unbounded trace-retention default is non-compliant by construction. "We keep traces until the disk fills" is a policy, and it is the wrong one. Set a TTL per span category, with content on the shortest |
| (f) Integrity and confidentiality | Traces hold prompts, tool arguments and retrieved documents in one place, usually in third-party SaaS, usually with broad internal read access. Trace stores deserve the access controls you give production databases, and most do not have them |
| Erasure requests | They must reach the trace store. If you cannot delete one user's spans, you cannot comply. Which is why pseudonymous keys are an engineering requirement and not a nicety: you need a join key to find the rows |
Build the redaction layer for Beacon, before the exporter ★ 10 min · everyone builds
Beacon's Insight Bot answers executive questions about revenue, which means its tool arguments contain customer ids and its retrieved documents contain unreleased financials. Build the layer that lets the trace stay useful and stops any of that leaving the process.
Write the allow-list first, as data rather than as code. The starting set is the low-cardinality metadata: provider, model, operation, token counts, finish reasons, document counts and scores, plus your own tenant.id and feature.id. Everything not on the list is redacted. Review the list in a pull request like any other security artifact.
Redact structurally, not destructively. For every field you drop, keep its shape: length, token count, whether it parsed as valid JSON, and the ids and scores of anything retrieved. This is what keeps the trace diagnostic, and it is the part people skip because dropping the field is one line shorter.
Hash the identifiers with a keyed hash and a rotation plan, so you can still group by user and still cannot read the user. Note the tension you have just created and write it down: rotating the key breaks your ability to join across the rotation boundary, which is a cost worth paying and a surprise worth documenting.
Put it in the right place. A ReadableSpan is immutable by the time on_end is called, so a span processor cannot rewrite attributes: in production this belongs in an exporter wrapper or in the collector's own redaction and transform processors. The sketch below is deliberately the readable version rather than the correct one, and if you ship it you will get a very quiet no-op.
Verify with a real export. Send one live request through staging and read the raw exported payload. Grep it for a known test value that you planted in the prompt and in a tool argument. A redaction layer that has not been verified against a raw export is a belief, not a control.
"__REDACTED__" rather than deleting the key is the pattern OpenInference chose and it is worth copying. A missing attribute is ambiguous: it could mean redacted, or it could mean your instrumentation silently stopped emitting last Thursday. A present attribute with a redaction marker tells you the pipeline is alive and the policy is working, which is the difference between a control and a hope.
The erasure drill: delete one Beacon user from your traces ★ 10 min · everyone builds
This is a fire drill, and almost every team fails it the first time. One named Beacon user asks for their data to be erased. You have 30 days. Do it, with a clock running, and write down every place you had to look.
Find the rows. Given only a user identifier, list every span, event, metric and content blob attributable to that person. If your answer requires a full table scan of a SaaS trace store you may not be able to query that way, you have just discovered that your join key was missing rather than pseudonymous.
Follow the content out to the separate store. Pattern three keeps content externally with references on the spans, and the upload hook runs regardless of the sampling decision, which means content can exist for traces you never sampled. Deleting spans does not delete blobs.
Check whether you can actually delete. Some vendors offer trace deletion by id, some only by project or by TTL, and some only on request. Whatever you find, write the answer into your DPIA: "deletion is a support ticket with an unstated SLA" is a finding, and it is one your DPO needs before the request arrives, not after.
Now set the TTLs so most of this problem expires on its own. Metadata can live long because it is not personal. Content gets the shortest life you can defend, in a separate store, so that the majority of erasure requests are satisfied by a clock rather than by an engineer at a keyboard.
Finally, write the one-page policy: what is captured, at what rate, with whose consent, where it lives, who may read it, how long it survives, and how it is deleted. One page. Reviewed on a date. That document is the deliverable of this session, more than any code you wrote today.
Try it yourself - this week ◐ 55-70 min total
- Export one raw span from your production pipeline and read every attribute. Specifically look at
gen_ai.tool.call.argumentsandgen_ai.retrieval.query.text, not just the prompt. Most teams find personal data in a field they had never thought about. - For every instrumentation library you have installed, look up its content-capture default in the source rather than the README. If any of them captures by default, you have a live exposure and a one-line fix.
- Run the erasure drill for one real user id and time it. Write the elapsed time and the list of places you had to look into your DPIA. That artifact is worth more to your DPO than any policy paragraph.
- Check whether any model id in production configuration is on a retention-mandated Covered Model list, and add a CI assertion that fails the build if an unapproved model id appears in config.
- Set an explicit TTL on every trace category you store, with content on the shortest. If any category currently has no TTL, that is an unbounded retention default, which Art. 5(1)(e) makes non-compliant by construction.
- Write the one-page policy from Demo 2 and put a review date on it. Then diary the review. An alerting and retention policy nobody re-reads decays into exactly the configuration you had before you wrote it.
Official sources covered
This is the legally-sensitive page of the course, so the labelling is strict. [P] is primary, [S] is secondary, and anything derived is named as a derivation. None of it is legal advice.
gen_ai.retrieval.query.text, gen_ai.tool.call.arguments and gen_ai.tool.call.resultPart 2 · and why redacting the prompt alone is insufficientOTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT appears in the spec only as "for example". The spec mandates the gate, not the name. Streaming chunk handling is a bare TODOPart 2 self-study · verify with a raw exportOPENINFERENCE_HIDE_* flags all defaulting False, redaction writing the literal "__REDACTED__"Part 2 self-study · and the pattern copied in Demo 1400 invalid_request_error to a ZDR org; compliance surfaces at 6 yearsRetention section · read live 2026-08-13, re-verify per cohortThree questions before you go 🎯 ◐ 90 seconds
1 · A team switches off prompt and completion capture, declares the trace store free of personal data, and passes security review. What did they miss?
The user asked something vague in natural language; your agent turned it into a typed call with an identifier in it, and then a result row came back out of your own database. Both of those are Opt-In and sensitive in the spec, and both are usually left on because they are what makes tool debugging possible. Add the retrieval query text as a third: what someone searched for is often more revealing than the answer they got.
2 · Your organisation has a Zero Data Retention agreement and a written 7-day retention policy. An engineer merges a config change to a Covered Model. What happens?
This is the teaching point worth repeating out loud: your retention policy can be set by your model choice rather than by your policy document. Claude Fable 5 and Mythos 5 are documented as Covered Models requiring 30-day retention and are ZDR-ineligible. The defence is mechanical rather than procedural: assert in CI that every model id in production configuration is on your approved list, and fail the build otherwise. And note the sibling trap, that CORS is unsupported under ZDR, so a browser app must route through a backend proxy.
3 · You add one framework instrumented by OpenInference alongside your OTel GenAI stack, changing nothing else. What happens to your privacy posture?
Every OpenInference trace is a valid OTLP trace, and the dialects are genuinely interoperable, which is exactly what makes this trap easy to walk into. The defaults are opposite: OTel GenAI is opt-in and off, OpenInference is on and you opt out. So one added dependency can start shipping prompts to your trace store with no change to your own code. Generalise it into a habit: read the content-capture default of every instrumentation you install, in the source, and verify with one raw export.
Ten sessions, and what you now have
You started with a green dashboard held through a full day of executives receiving stale numbers, and you finish able to build the system that would have caught it in twenty minutes. Here is the whole builder track in ten lines, which is also a decent checklist to audit any AI system against.
- b1 · Spans and traces. Six words, the parented tree, and unaccounted time as the number that localises everything else.
- b2 · OTel GenAI conventions. The current attribute keys with their requirement levels, and the deprecations that date most blog posts.
- b3 · Instrumenting an agent. The nesting rules, the plan-owns-its-LLM-call exception, and where the auto-instrumentation stops.
- b4 · Cost and latency per span. Token attribution that survives a loop, the double-counting trap, and TTFT as its own quantity.
- b5 · Reading a broken trace. Coverage rung by rung, up to five of five diagnoses, and the content lever that adds nothing.
- b6 · The failure taxonomy. Failure mode to telemetry signal to fix, and the swallowed exception that starts it all.
- b7 · Sampling and its own cost. Head versus tail, the
decision_waittrap, and cardinality as the reason ids stay off metrics. - b8 · Alerting and SLOs. Burn rate over thresholds, both windows, and an inventory that says page, ticket or neither.
- b9 · Drift and silent changes. Four drifts, the provider's three mechanisms, and the embedding version gate that refuses rather than degrades.
- b10 · Privacy and retention. Metadata at one hundred percent, content on a consented short-TTL sample, and a policy on one reviewed page.
except block on an AI path, and emit gen_ai.response.model next to gen_ai.request.model. Both are afternoon-sized. Then compute unaccounted time on one real request and take that single percentage to your next planning meeting, because a number beats an argument, and this is the number that funds everything else in the course.
Builder session 10 cheat sheet · pin this
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT is de facto. The spec mandates only that a gate exists and is off.