Sampling is not a cost control. It is a decision about which incidents you can solve
Evals tell you whether it works. Observability tells you what actually happened - for the requests you kept. Everybody frames sampling as a budget question, and it is one, but that framing hides the sharp edge. A sampling rule is a standing decision about which future incidents will be investigable and which will not, taken by someone who does not yet know what the incident will be. Session b6 gave you a detector for every failure mode. Every one of those detectors is worthless on a trace you threw away.
tail_sampling config with the four policies that matter, the decision_wait trap that turns your observability pipeline into a silent correctness bug, and a cardinality rule you can defend in a budget meeting: unbounded ids live on spans, never on metrics.
Head sampling cannot see the future 8 min live
The whole distinction is a timing question, and the diagram is the argument. Head sampling makes its decision at the root span. Tail sampling makes it after every span in the trace has arrived. Everything else follows from those two clocks.
LiveWhat head sampling can act on, and why it is not enough here4 min▶
OTEL_TRACES_SAMPLER takes nine values, and the default is parentbased_always_on - which means unless somebody changed it, you are keeping everything and the cost conversation has simply not started yet. The nine cover the always-on, always-off and ratio families, each with a parentbased_ variant, plus the remote-sampler options. Check the exact spellings for your SDK version rather than trusting a blog post.
The GenAI conventions are helpful about what a head sampler can use. Five attributes SHOULD be provided at span creation, which makes them available to the sampling decision: gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, server.address and server.port. That is the low-cardinality set, and it is also what a string_attribute tail policy can match on.
Notice what is missing from that list, and notice that it could not be otherwise. Nothing about the outcome. Nothing about duration. Nothing about token counts. Nothing about whether a tool failed. A head sampler can express "keep 20% of Opus traffic and 2% of Haiku traffic", which is a real and useful thing to say. It cannot express "keep the ones that broke", because at span creation nothing has broken yet.
Self-studyWhy the LLM case is different from HTTP traffic3 min read▶
Uniform head sampling was designed for a workload that AI applications do not have. Two differences do all the work.
- Volume and value are inverted. A web service emits millions of near-identical request traces, so 1% is a fine statistical sample of a homogeneous population. An agent product emits far fewer traces, each expensive to produce, each meaningfully different from the next, and each one attached to a named user who might complain. Uniform 1% sampling on that population throws away the interesting ones and faithfully preserves 1% of the boring ones.
- Slowness is the symptom, not the side effect. In HTTP land a slow trace is usually a slow dependency. In agent land a slow trace is frequently the failure itself: the runaway loop, the retry storm, the agent that did not know it was finished. Session b6 put a number on that - step repetition at 15.7% plus unaware-of-termination at 12.4%. Sampling on latency is therefore sampling on your most common failure category, and it is only possible at the tail.
There is a corollary worth saying out loud to whoever owns the budget. The correct answer for AI traces is usually not a lower ratio. It is a selective policy: keep 100% of errors, 100% of slow traces, 100% of high-span-count traces, and a small ratio of the ordinary ones. That is often cheaper than a flat 10% and dramatically more useful.
The tail_sampling processor, concretely 7 min live
Tail sampling is not a flag you flip. It is a stateful component in the middle of your collector pipeline with four operational facts attached to it, and each one has bitten a real deployment.
decision_wait: 30s against a 7200s workflow
decision_wait defaults to 30 seconds. The GenAI conventions define gen_ai.invoke_workflow.duration with buckets running 1, 5, 10, 30, 60, 120, 300, 600, 1800, 3600, 7200 - the spec authors expect agent workflows to run for up to two hours. If your workflows take twenty minutes and your decision_wait is at its default, the processor evaluates its policies against a partial trace and moves on. There is no error, no warning, no metric that goes red. Your latency policy never sees the total latency, your error policy never sees the error that happened at minute twelve, and every dashboard downstream reports confidently on a biased sample. This is not a performance problem or a tuning preference. It is a silent correctness bug in your observability pipeline, and the traces it corrupts are precisely the slow ones you built the policy to catch. Set decision_wait from the measured p99 of your longest workflow, and re-measure it whenever the agent gains a tool.
decision_wait, num_traces, the policy types and the sampling_strategy default of trace-complete are all verified in this course's source map. The nested sub-keys inside each policy are not, and the processor is beta, which means they are allowed to move between collector releases. Read the processor README for the exact version you are deploying, and treat this snippet as the shape rather than the spelling.
Seventeen policies exist. Four of them are the LLM ones
| Policy | What it decides on | Why it matters for an agent |
|---|---|---|
status_code | Whether any span in the trace carries ERROR | The decision head sampling structurally cannot make. Keep 100% of these - they are rare and they are the whole reason you have a trace store |
latency | Total trace duration | A slow agent trace is usually the runaway loop itself, not a slow dependency. This is the single highest-yield policy in an LLM pipeline |
span_count | How many spans the trace contains | Catches a runaway loop by shape. A trace with more than N spans is by definition an agent that did not converge - no attribute needed, no content read, no model involved. The cheapest detector in this course |
ottl_condition | An OTTL expression over spans and attributes | Spec-aware rules: keep traces where gen_ai.response.finish_reasons contains a truncation, or where a tool span errored. The docs call it "highly recommended... to avoid breaking changes", so prefer it over the older attribute policies |
string_attribute | A named attribute's string value | Works on the low-cardinality sampling-time set - gen_ai.provider.name, gen_ai.request.model, server.address. Useful for per-model or per-tenant ratios, and the one policy that a head sampler could also have expressed |
Self-studyThe "same collector instance" requirement is an architecture constraint3 min read▶
"All spans for a given trace MUST be received by the same collector instance." That single sentence is the reason tail sampling is an infrastructure project rather than a config change, and it is the fact most often discovered late.
If you run a fleet of collectors behind a round-robin load balancer, spans from one trace land on different instances, each instance sees a fragment, and each fragment fails your policies for different reasons. The standard shape is a two-layer collector deployment: a stateless first layer that receives and routes by trace id, and a second layer that owns tail sampling and holds state. Because the processor already groups by trace id internally, you do not need groupbytrace in front of it - a common piece of stale advice.
Two more consequences of statefulness that belong in your capacity plan. num_traces defaults to 50000 traces held in memory, so the memory footprint scales with your trace rate multiplied by decision_wait - which means the fix for the 30-second trap has a bill attached: raising decision_wait to 300s raises the number of in-flight traces roughly tenfold. And because the state is in memory, a collector restart loses every in-flight decision. Neither of those is a reason not to do it; both are reasons to size it before you ship it.
Cardinality: the same attribute costs linearly on a span and multiplicatively on a metric 6 min live
Session b6 had you add loop indices, args hashes, cache ages and conversation ids. That is the right instinct on spans and a catastrophe on metrics, and the difference is arithmetic rather than taste.
gen_ai.request.model is bounded at tens of values, so putting it on a metric is fine forever. gen_ai.conversation.id is unbounded by definition - one value per conversation, growing with your product. Use it as a metric dimension and gen_ai.client.token.usage becomes one 14-bucket histogram series per conversation, multiplied again by gen_ai.token.type and by the model. That is not a metric any more; it is a very expensive log.
The reason every metric attribute set in the GenAI conventions deliberately excludes ids - no conversation id, no response id, no tool call id, no user id - is not an oversight to be worked around. The exclusion is the guardrail. When you find yourself wanting to add an id to a metric, the answer is almost always that you wanted a trace query, not a metric.
Where each attribute belongs
| Attribute | Bound | On a span | As a metric label |
|---|---|---|---|
gen_ai.provider.name | 16 values | Yes | Yes - it is in the spec's metric sets |
gen_ai.operation.name | 18 values | Yes | Yes |
gen_ai.request.model | Tens | Yes | Yes |
gen_ai.token.type | 2 values | Yes | Yes - and required on gen_ai.client.token.usage |
gen_ai.conversation.id | Unbounded | Yes - it is the join key that reassembles a thread across traces | Never |
gen_ai.response.id | Unbounded | Yes | Never |
gen_ai.tool.call.id | Unbounded | Yes | Never |
gen_ai.agent.id | Unbounded | Yes | Never |
gen_ai.request.previous_response.id | Unbounded | Yes | Never |
The memory ids, gen_ai.data_source.id | Unbounded | Yes | Never |
Write Beacon's tail-sampling policy, then set decision_wait honestly ★ 12 min · everyone builds
Beacon's Insight Bot was invisible for a week partly because nobody had ever looked at one of its traces. We are going to guarantee that the next bad one is kept, and that the config is sized against the real workflow rather than against a default.
Measure first. Find the p99 total duration of your longest agent workflow over the last month. Not the p50, not the model-call duration - the whole root span, including the retries and the runaway ones. Write the number down; every other decision on this page hangs off it.
Set decision_wait above that p99 with headroom, and then check what you have just committed to in memory. In-flight traces scale roughly with trace rate multiplied by decision_wait, and num_traces defaults to 50000. Going from 30s to 300s multiplies your in-flight count by ten. If that exceeds 50000, raise num_traces deliberately and size the collector for it, rather than discovering the eviction later as mysteriously missing traces.
Add the three policies from the snippet: status_code on ERROR, latency with a threshold set from your own p95, and span_count with min_spans set just above the top of your normal range. For Beacon the normal range is a handful of spans, so the loop detector triggers early and cheaply. Remember the policies are OR-ed: a trace matching any one of them is kept.
Now the Beacon-specific one, and the reason ottl_condition is worth learning. Beacon's incident produced no error status at all - the exception was swallowed. So add a condition that keeps any trace carrying the b6 attributes: beacon.fallback.used, beacon.truncated, beacon.refused, beacon.empty_end_turn. Session b6 made those failures visible; this step makes sure the trace containing them survives long enough for you to read it.
Check pipeline order and instance affinity. tail_sampling must sit after k8sattributes and any other context processor, or you will be sampling on attributes that have not been attached yet. And confirm that all spans for one trace reach the same collector instance - if you are behind a plain round-robin load balancer today, that is a two-layer collector project and it is better to know now than after the config looks correct and behaves strangely.
Verify by breaking it on purpose. Send one synthetic trace that errors, one that is deliberately slow, one with 60 spans, and one that is entirely ordinary. Confirm the first three arrive at your backend and the fourth is subject to your base ratio. Then set decision_wait back to 30s for one minute and watch the slow trace vanish without a single warning. That is the trap, felt rather than read.
Audit your own cardinality before finance does ★ 10 min · everyone builds
This is arithmetic, it takes ten minutes, and it has stopped real budget surprises. You are going to count the time series you currently produce, and then count the ones you would have produced if the spec had not stopped you.
List every label you currently attach to every metric. For each label, write down the number of distinct values it takes in a month. Multiply the counts together per metric. That product, not the number of metrics, is what you are paying for.
Now do the counterfactual. Take gen_ai.client.token.usage, which is a 14-bucket histogram and requires gen_ai.token.type. Multiply your monthly conversation count by 2 for token type and by your model count. Compare that number to the few hundred series you get from the spec's own attribute set. The ratio is the size of the mistake the spec quietly prevented you from making.
Sort your b6 attributes into two lists: span-only and metric-safe. Anything with "id" in the name goes in span-only without further discussion. beacon.tool.args_sha goes in span-only - a hash is unbounded by design. beacon.fallback.used is a boolean, so it is metric-safe and it is a genuinely useful metric dimension.
Find the shadow high-cardinality label that is not an id. The usual suspects are a free-text error message, a full URL with a query string, a prompt name that includes a version and a timestamp, or a tenant name in a system with self-service signup. None of them look like ids and all of them behave like ids.
Write the rule down where the next engineer will find it, in one line: ids and hashes go on spans, enums and booleans go on metrics, and if you want to slice a metric by something unbounded you actually wanted a trace query. Then check the spec's metric attribute sets and notice that you have just re-derived them.
SPAN cost = spans_per_trace x traces_kept x bytes_per_span → adding one attribute is LINEAR, and adding it to a trace you already keep is nearly free · METRIC cost = distinct(label_1) x distinct(label_2) x ... x distinct(label_n) → adding one unbounded label MULTIPLIES the series count by the number of values it will ever take
Try it yourself - this week ◐ 45-55 min total
- Find out what your
OTEL_TRACES_SAMPLERis actually set to in each environment. If nobody has changed it, it isparentbased_always_onand you are keeping everything, which means your cost conversation has not started and your sampling design has not been tested. Both are worth knowing before the invoice arrives. - Compute the p99 duration of your longest workflow and compare it to your
decision_wait. Ifdecision_waitis smaller, you have a live silent correctness bug in your observability pipeline and it is biased against exactly the traces you care about most. Fix it, then size the memory it costs. - Write one
ottl_conditionpolicy that keeps a failure your status codes cannot see - a truncation, a refusal, or a fallback flag from session b6. This is the step that connects the failure taxonomy to the retention policy, and it is the one most teams never take. - Do the cardinality multiplication for your three most-labelled metrics. Bring the products to whoever owns the observability budget. It is a much better conversation than "we should probably sample more".
- Argue the selective case in writing: 100% of errors, 100% of slow traces, 100% of high-span-count traces, plus a small ratio of ordinary ones - versus a flat ratio across everything. Estimate both volumes from a week of real traffic. For most AI workloads the selective policy is both cheaper and more useful, and having the numbers is how you get to keep it.
Official sources covered
Sampling claims come from the OpenTelemetry sampling documentation and the collector-contrib tail_sampling processor; the bucket and attribute-set facts come from the GenAI semantic conventions. Everything unverified is marked as such.
OTEL_TRACES_SAMPLER: nine values, default parentbased_always_onPart 1 · check exact spellings per SDK versiongen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, server.address, server.portPart 1 · the low-cardinality set a head sampler and string_attribute can act ontail_sampling processor: beta, same-collector-instance requirement, groups by trace_id so groupbytrace is unnecessary, must follow context processors, explicitly statefulPart 2 · the four operational factsdecision_wait default 30s, num_traces default 50000, sampling_strategy default trace-completePart 2 + Demo 1 · the trap and its memory costgen_ai.invoke_workflow.duration buckets 1, 5, 10, 30, 60, 120, 300, 600, 1800, 3600, 7200Part 2 · 7200s versus a 30s wait is the whole traplatency, status_code: ERROR, span_count, ottl_condition ("highly recommended... to avoid breaking changes")Part 2 · four of the seventeenThree questions before you go 🎯 ◐ 90 seconds
1 · Why can head sampling not be configured to keep your error traces, no matter what ratio you choose?
This is a timing fact, not a configuration gap. At the moment a head sampler runs, the root span has just been created, the agent has made zero tool calls, nothing has errored and nothing has taken any time yet. The five attributes the spec says SHOULD be available at span creation are all inputs - operation, provider, model, server address and port - and none of them are outcomes. Tail sampling exists precisely because the two decisions LLM apps need are both tail decisions by definition.
2 · Your agent workflows routinely run twelve minutes. You deploy tail_sampling with default settings. What happens?
Nothing errors. That is what makes it teachable. decision_wait defaults to 30 seconds while gen_ai.invoke_workflow.duration defines buckets out to 7200 seconds, so the spec authors clearly expect agent workflows to run for hours. Your latency policy never sees the real total, your error policy never sees an error that happens at minute nine, and every downstream dashboard reports confidently on a biased sample. Set it from your measured p99 - and remember that in-flight traces scale with decision_wait, against a num_traces default of 50000.
3 · You want to slice token usage by gen_ai.conversation.id. Where does that attribute belong?
Metrics cost scales with distinct time series, which is the Cartesian product of label values. gen_ai.request.model is bounded at tens and is safe forever. gen_ai.conversation.id grows with your product and never stops. That is why every metric attribute set in the GenAI conventions deliberately excludes ids: conversation, response, tool call, agent, the memory ids and gen_ai.data_source.id. The exclusion is the guardrail. Wanting to slice a metric by something unbounded is a reliable sign you wanted a trace query.
Builder session 7 cheat sheet · pin this
OTEL_TRACES_SAMPLER has 9 values and defaults to parentbased_always_on. If nobody changed it, you are keeping everything.decision_wait defaults to 30s; gen_ai.invoke_workflow.duration buckets reach 7200s. A short wait is a silent correctness bug in your pipeline.span_countCatches a runaway loop by shape. More than N spans means the agent did not converge. No attributes, no content, no model. The cheapest detector you own.k8sattributes. No groupbytrace needed. Stateful, num_traces default 50000 in memory.ottl_conditionSpec-aware and "highly recommended... to avoid breaking changes". It is how you keep failures that never set an error status.