Shape is a signal, and it is free
Everything b1 and b2 taught is about content: what is in a span, what it is called, which level it sits at. Today is about arrangement. Arrangement is the cheapest signal in your telemetry because you get it whether you want it or not: every span already has a parent, so the tree already exists. The question is whether you built it in a shape that means something, and whether you know how to read it when it does.
Development, in a repository with no tags and no releases. The nesting rules below are quoted from the agent span conventions as read on 13 August 2026. The reading rules that follow from them are more durable than the names, because they describe the shape of an agent loop rather than the spelling of it.
Four nesting rules, one of them surprising 8 min live
The agent span conventions "extend and override" the base span conventions, which is a polite way of saying that if you learned span structure from a generic tracing tutorial, some of it does not apply here. Four rules, stated as the spec states them.
| Rule | What the spec says | Why it is written that way |
|---|---|---|
| 1 · When not to report a workflow | invoke_workflow SHOULD NOT be reported for standalone agent invocations, nor when the workflow is an internal implementation detail | A wrapper span that always has exactly one child adds a level of nesting and no information. If the workflow is your framework's plumbing rather than your design, it is noise |
| 2 · When to report one anyway | Application-defined workflows SHOULD be reported even when nested, for example a sub-graph invoked as a node | The test is authorship, not depth. A workflow you designed is a unit somebody will want to time and compare, so it earns a span even three levels down |
| 3 · Plan owns its model call | The plan-generating LLM call is a child of the plan span | plan is the only span type in the conventions that owns an inference span as a child. The plan span exists to measure the act of deciding, and the model call is how the deciding happens |
| 4 · The plan's tools are siblings | The tool and task spans that come out of that plan are siblings, under the same invoke_agent span | The plan produced them; it did not contain them. Nesting execution under the plan would make the plan span's duration include all the work, and you would lose the ability to ask how long deciding took |
plan duration is a clean measurement of thinking time, separate from doing time. That gives you a diagnostic almost nobody has: the ratio of deciding to executing across an agent run. An agent spending 0.9 seconds planning and 4.2 executing is a different animal from one spending 6 seconds planning and 0.4 executing, and the second one is usually a prompt problem rather than an infrastructure problem. Nest the tools under the plan and both collapse into one number that means nothing.
LiveFramework entry points the spec names as workflows3 min▶
Rule 1 and rule 2 pull in opposite directions, so the spec grounds the distinction with named examples. These are the framework entry points that map to invoke_workflow:
| Framework | Entry point |
|---|---|
| Google ADK | Runner.run(...) |
| CrewAI | Crew.kickoff() |
| LangChain / LangGraph | *Graph*.invoke(...) |
| Microsoft Agent Framework | Workflow*.run(...) |
| OpenAI Agents | Runner.run(starting_agent=...) |
Read that list as a definition rather than a lookup table. What these calls have in common is that you wrote the thing being run: the graph, the crew, the workflow object. That is the authorship test from rule 2. A framework's internal step dispatcher is not on this list even though it is technically also a workflow, because you did not design it and nobody will ever want a p95 of it.
The practical version for your own code: if a product manager would recognise the name, it earns an invoke_workflow span. If only the framework maintainers would, it does not.
Self-studyWhat "extend and override" means when you are debugging2 min read▶
The agent conventions extend and override the base GenAI span conventions, which has one concrete consequence worth internalising: an invoke_agent span is not just a wrapper, it carries GenAI attributes of its own, including aggregate usage. That is the source of the double-counting trap in Part 2, and it is also what makes the agent span useful on its own.
The mental model that works: invoke_agent is a GenAI span that happens to have GenAI spans underneath it. It is not a generic parent like an HTTP server span. So when you query for spans by gen_ai.provider.name, agent spans come back too, and any naive sum over that result set is wrong. Every aggregate you write over agent telemetry needs to say which level it means.
Three rules that fall out of the structure 8 min live
This is the heart of the session. Three consequences of the nesting rules, each of which lets you diagnose something real from the outline of a trace. They are not spec text; they are what the spec text implies once you have seen a few hundred agent traces.
invoke_agent span. It does not add depth, because nothing nests: the plan, the tools and the model calls all sit at the same level, iteration after iteration. So a deep agent trace usually means composition (workflows, sub-agents, a plan and its call) while a wide one means iteration. A forty-child invoke_agent span is not a complex task; it is a loop nobody bounded.
The two counters from b2's metric table exist for exactly this: gen_ai.invoke_agent.inference_calls and gen_ai.invoke_agent.tool_calls, both histograms, both bucketed 1, 2, 4, 8, 16, 32, 64, 128. Nobody chooses a 128 bucket for a system they expect to behave. Emit those two and a runaway becomes a dashboard shape rather than an archaeology exercise.
invoke_agent spans side by side under the workflow. B is not inside A. This matters because the intuition runs the other way - a handoff feels like delegation, and delegation feels like nesting - and the shape you look for when debugging depends on getting it right. Nested agent spans mean a sub-agent invoked as a tool. Sibling agent spans mean control passed on. Two different designs, two different failure modes, and one of them makes the workflow span the only place the whole conversation exists.
invoke_agent carries aggregate gen_ai.usage.* for the whole agent run, and each child chat span carries its own. Both are correct. Neither is redundant. And a query that sums gen_ai.usage.input_tokens across every span in the trace returns roughly double the truth. Sum at one level only. This is b4's central trap and it is worth meeting today, because the wrong number is plausible, stable and confidently wrong.
LiveThe double-count, with the arithmetic done4 min▶
One Analyst Bot run, three model calls. Here is what the trace actually contains.
| Span | Level | gen_ai.usage.input_tokens | gen_ai.usage.output_tokens |
|---|---|---|---|
invoke_agent analyst_bot | Aggregate | 8,400 | 1,150 |
chat under plan | Leaf | 2,100 | 300 |
chat, iteration 2 | Leaf | 3,000 | 350 |
chat, iteration 3 | Leaf | 3,300 | 500 |
| Sum of leaves | 8,400 | 1,150 | |
| Naive sum of every span | 16,800 | 2,300 |
Exactly twice. Which is the tell, and also the reason nobody catches it: a clean 2x looks like a pricing tier, a currency mistake or a doubled traffic estimate long before it looks like a query bug. And it will not always be 2x - the multiple depends on how many levels aggregate, so a workflow with two agents summed naively is worse.
Two defensible ways to fix it, and you must pick one and write it down:
- Sum the leaves. Filter to spans with
gen_ai.operation.name = "chat"(or your other inference operations) and sum those. Precise, and it survives new agent levels being added above. - Read the aggregate. Take
gen_ai.usage.*off the topinvoke_agentspan and ignore everything below. Cheaper, and it breaks quietly the day a sibling agent span appears next to it.
What you must not do is leave it undecided per dashboard, which is the normal state of affairs. Two dashboards, two levels, two token totals, and one very long meeting.
Self-studyReading a trace outline in five seconds2 min read▶
A drill worth practising, because it turns the three rules into a reflex. Collapse the trace to just its outline and ask three questions in order:
- How wide is the widest agent span? Five to ten children is a working loop. Forty is a bug. If your framework does not bound iterations, this is the number that tells you nobody is.
- How many
invoke_agentspans are there, and are any nested? Siblings mean handoffs. Nesting means an agent is being used as a tool. Both are legitimate designs; only one of them matches what the team thinks they built, and that mismatch is worth finding on a quiet afternoon. - Is there exactly one inference span under each
plan? Two means somebody nested the execution under the plan and your thinking-versus-doing split is gone. Zero means the plan span is decorative.
None of that requires reading an attribute, opening a payload or knowing what the agent was asked. It is available from the outline alone, which is why it survives sampling, redaction and a content-off privacy posture.
The framework reality, and two renamed entry points 7 min live
Everything above assumes something is emitting these spans. Here is the uncomfortable part: upstream OpenTelemetry ships GenAI instrumentation for four things, and the frameworks most teams are actually using are not among them. Third parties fill the gap, which means the dialect and the shape of your agent traces are decided by which package you pip installed.
Two details from that matrix are worth stating out loud. OpenInference ships 37 packages and OpenLLMetry around 33, which is the actual reason both exist: the spec moves faster than upstream contrib can follow, and somebody has to ship. And only OpenLLMetry covers vector stores. Given that b1's whole thesis is that retrieval is where the incident lives, notice that the layer most likely to break is the layer with the narrowest instrumentation coverage in the entire ecosystem.
LiveLangChain: the case where the source beats the docs4 min▶
This is a small correction with a large lesson attached. If you follow LangSmith's public documentation you will set OTEL_ENABLED or LANGSMITH_OTEL_ENABLED. Those are now legacy. The current control is LANGSMITH_TRACING_MODE, and you only find that out by reading client.py.
explicit tracing_mode argument > deprecated otel_enabled argument > LANGSMITH_TRACING_MODE environment variable > legacy OTEL_ENABLED / OTEL_ONLY > default "langsmith" Python modes: langsmith | otel | hybrid JS modes: langsmith | otel (no hybrid) Legacy LANGSMITH_OTEL_ENABLED, if truthy, maps to "hybrid" in Python.
Three things follow. First, Python has three modes and JavaScript has two: there is no hybrid in JS, so a polyglot team cannot assume the same configuration means the same thing on both sides. Second, the legacy variable does not map to otel as most people assume; it maps to hybrid, so a team that thinks it switched to OTLP is in fact still double-emitting to LangSmith. Third, and most useful: the public docs still show only the legacy variables.
That third point is the lesson, and it is worth more than the variable name. When telemetry configuration does not behave the way the documentation says, read the source that resolves it. It is usually one function, it is usually named something like _resolve_*, and it is always the truth.
LiveVercel AI SDK: the same lesson, second time3 min▶
If the LangChain example felt like a one-off, here is the same shape from a completely different project. The Vercel AI SDK renamed its telemetry entry point.
// current import { registerTelemetry, OpenTelemetry } from "@ai-sdk/otel"; registerTelemetry(new OpenTelemetry()); // experimental_telemetry is no longer in the docs // the old ai.* span names (ai.generateText, ai.streamText, ai.toolCall) // are now LegacyOpenTelemetry
The consequence is the same in kind and different in flavour. Here the span names moved too, so a dashboard built on ai.generateText does not error, it just goes quiet, which is the worst way for a rename to reach you. And the migration is not only a code change: anything downstream that matched on ai.* needs rewriting.
Two renamed entry points in two unrelated projects, both with documentation lagging behind the source, is not bad luck. It is what a fast-moving ecosystem looks like from the inside. The habit worth building: pin your instrumentation versions, read the changelog of the instrumentation and not only of the model SDK, and keep one canary dashboard whose whole job is to go blank when span names change.
Self-studyAuto or manual, and the answer that is actually both3 min read▶
The question people arrive with is auto-instrumentation or manual. The answer that holds up is: auto for the model and tool calls, manual for the shape and for everything the SDK cannot see.
| What you get | What it will not do | |
|---|---|---|
| Auto | Inference spans with correct names and attributes, tool spans where the framework knows about tools, token counts, and the dialect the package chose | Know which of your workflows is application-defined, know your own cache and warehouse calls exist, or decide anything about privacy on your behalf beyond its own default |
| Manual | The workflow and agent boundaries you designed, the spans around retrieval, caching and database calls, error status on your own failure paths, the retry counter, tenant and conversation ids | Keep up with provider SDK changes, which is a real maintenance cost and the honest argument for auto |
The division of labour that works in practice: let the package instrument the calls it wraps, and write manual spans for every boundary that exists in your design rather than in your dependencies. b1's incident lived entirely in the second category - the warehouse timeout and the stale cache read were both yours to instrument, and no package was ever going to do it for you.
Give the Beacon Analyst Bot its shape ★ 12 min · everyone builds
The Analyst Bot turns an executive question into a warehouse query and a written answer. It currently emits three flat inference spans under an HTTP server span, which is what auto-instrumentation alone produces. We are going to give it the shape from Part 1, and the point of doing it by hand is that the nesting decisions become visible as decisions.
Start with the workflow question, and be willing to answer no. Is beacon.analyst_flow something you designed and would name in a status update, or is it your framework's dispatcher? If it is yours, it gets an invoke_workflow span named invoke_workflow beacon.analyst_flow. If the answer is that there is only ever one agent under it, the spec says SHOULD NOT and you skip it.
Open invoke_agent analyst_bot for the whole agent run, including all iterations. One span, however many times the loop goes round. Set gen_ai.operation.name to invoke_agent and gen_ai.agent.name to the agent name, and remember this span carries GenAI attributes of its own.
For each iteration, open plan analyst_bot and put the plan-generating model call inside it as a child. This is the only nesting of an inference span in the entire session. Everything you learn about thinking time comes from getting this one right.
Now close the plan span before you execute anything it decided. The tool spans go under invoke_agent as siblings of the plan, not under the plan. If you find yourself keeping the plan span open to parent the tools, you have the shape wrong and your plan duration is about to become meaningless.
Add the spans no package will add for you: the warehouse query and the cache read. In b1's incident those two were the whole diagnosis and neither is a GenAI span, so neither is anyone's auto-instrumentation problem but yours.
Now render the trace and count the levels. Workflow, agent, plan, the plan's model call: four deep at most, and wide in the middle. If your trace is six levels deep, something is nested that should be a sibling - go and find it.
with tracer.start_as_current_span("invoke_workflow beacon.analyst_flow"): with tracer.start_as_current_span("invoke_agent analyst_bot") as agent: for _ in range(max_iterations): # plan owns its model call - the only nesting like this with tracer.start_as_current_span("plan analyst_bot"): with tracer.start_as_current_span(f"chat {model}"): step = decide_next_step() # plan span is now CLOSED - thinking time is measured # siblings of plan, children of invoke_agent with tracer.start_as_current_span(f"execute_tool {step.tool}"): run(step)
Make a runaway visible without opening a trace ★ 10 min · everyone builds
Beacon's Insight Bot has run away twice this quarter and both times somebody found it by noticing the bill. Traces would have shown it immediately, but nobody opens a trace for a request that returned an answer. Two histograms fix that, and one query decision keeps the cost number honest.
Record gen_ai.invoke_agent.inference_calls and gen_ai.invoke_agent.tool_calls once per agent run, when the invoke_agent span closes. Count the children you opened; that is the whole implementation. Both are histograms with buckets 1, 2, 4, 8, 16, 32, 64, 128.
Also record gen_ai.invoke_agent.duration, buckets 0.1 up to 409.6 seconds. Duration and call count together separate the two runaway flavours: many fast calls is a loop, few slow calls is a backend problem, and one metric alone cannot tell you which.
Keep the metric labels bounded. Agent name, model, provider, operation. No conversation id, no response id, no tool call id, no user id - that is the spec's own guardrail and b7 does the arithmetic on why. Unbounded on a span is fine; unbounded as a metric label is a bill.
Now pick your token-summing level and write it into the query, with a comment saying why. Either filter to inference operations and sum the leaves, or read the aggregate off the top invoke_agent span. Not both. The 2x error is invisible until somebody compares your dashboard to an invoice.
Prove the double-count to yourself before you trust the fix. Run the same query twice, once unfiltered and once filtered to the leaves, and look at the ratio. Seeing a clean 2x in your own data is the thing that makes this stick.
Finally, set a bound in the code, not only in the dashboard. Telemetry makes a runaway visible; a maximum iteration count makes it survivable. b8 turns the histogram into an alert, but the alert is a worse fix than the cap.
# when the invoke_agent span closes inference_calls.record(n_chat_spans, {"gen_ai.agent.name": name}) tool_calls.record(n_tool_spans, {"gen_ai.agent.name": name}) # bounded labels only - no conversation id, no user id, no tool call id # token cost, summed at ONE level: the leaves # WHERE gen_ai.operation.name = 'chat' # SUM(gen_ai.usage.input_tokens) # the aggregate on invoke_agent is the same number. Adding both doubles it.
Try it yourself - this week ◐ 45-60 min total
- Take your widest agent trace from the last month and count the children of the busiest
invoke_agentspan. Then find the p99 of that number across a week. If you cannot compute it because nobody emits the call-count histograms, that is the homework: emit them. - Run the double-count experiment on your own data. Sum
gen_ai.usage.input_tokensacross all spans in a trace, then across inference spans only, and write the ratio in your team's runbook next to the query you have decided is correct. - Check whether your plan spans, if you have them, close before execution starts. If they do not, fix one agent and compare the before and after plan duration. The difference is the size of the measurement you were missing.
- Find out which package instruments your main framework, and which dialect it emits. Then find its version in your lock file and go read its changelog rather than the model SDK's. Most teams have never done this and it takes ten minutes.
- Check your LangChain or LangSmith configuration against the precedence chain in Part 3. If you set
LANGSMITH_OTEL_ENABLEDexpecting OTLP only, you are inhybridand double-emitting. If you run both Python and JavaScript, confirm the modes match, remembering there is nohybridin JS. - Grep your dashboards and alerts for
ai.generateText,ai.streamTextandai.toolCall. Anything you find is either already silent or about to be.
Official sources covered
The nesting rules are quoted from the GenAI agent span conventions; the framework coverage was read from the upstream contrib repository and the two third-party projects; the two renamed entry points were read from source rather than documentation, which is the point of including them. All verified 13 August 2026.
invoke_workflow SHOULD NOT and SHOULD be reported, and "extend and override" over the base span conventionsPart 1 · rules 1 and 2, with the authorship testplan owns its LLM call as a child; the tool and task spans from that plan are siblings under the same invoke_agentPart 1 · rules 3 and 4, and Demo 1 builds itinvoke_workflow: ADK, CrewAI, LangChain/LangGraph, Microsoft Agent Framework, OpenAI AgentsPart 1 · read as a definition, not a lookupgen_ai.invoke_agent.inference_calls and gen_ai.invoke_agent.tool_calls, buckets to 128, and the identifiers every metric attribute set excludesPart 2 and Demo 2 · alerted on in b8LANGSMITH_TRACING_MODE and the resolution order from the Python SDK source, three modes in Python and two in JS, legacy variable mapping to hybridPart 3 · source beats docs, first caseregisterTelemetry(new OpenTelemetry()) from @ai-sdk/otel, experimental_telemetry gone from the docs, old ai.* names now LegacyOpenTelemetryPart 3 · source beats docs, second casedecision_wait trapThree questions before you go 🎯 ◐ 90 seconds
1 · An agent plans, then calls two tools based on that plan. What is the correct parentage of the two tool spans?
Causation is not containment, and this is the one place the spec's answer runs against intuition. The plan span owns exactly one child, its own model call, and nothing else. Nest the tools under it and the plan span's duration silently becomes the duration of the whole iteration, so you lose the ratio of deciding to executing while everything still renders and every total still adds up correctly.
2 · You sum gen_ai.usage.input_tokens across every span in an agent trace and get a number roughly twice what the invoice suggests. What happened?
Both numbers are correct and neither is redundant, which is why this survives code review. A clean 2x reads like a pricing or currency mistake long before it reads like a level error. Option A is the plausible distractor and is wrong for a reason from b2: one span covers the whole logical operation including automatic retries, so retries do not add spans at all.
3 · You want to instrument a LangGraph agent calling Claude through the Anthropic SDK. What does upstream OpenTelemetry give you?
This is the most common wrong assumption in the whole session, and b1's asymmetry is the reason it feels wrong: auto-instrumentation usually does give you the model calls for free. It does, but from OpenInference or OpenLLMetry rather than from upstream, and that choice comes with a dialect, a set of span kinds, a version policy and a default privacy posture. Note also the sharpest line in the matrix: vector stores are covered by exactly one of the three, and retrieval is where b1's incident lived.
Builder session 3 cheat sheet · pin this
plan. It is the only nesting of its kind in the conventions.invoke_agent, next to the plan. Close the plan span before execution starts.invoke_agent is a runaway loop, not a hard task.invoke_agent spans side by side under the workflow. Nested agent spans mean an agent used as a tool - a different design.invoke_agent aggregates usage and children carry their own. Both correct. Summing across levels doubles the bill.gen_ai.invoke_agent.inference_calls and .tool_calls, buckets to 128. Bounded labels only. A runaway becomes a number.LANGSMITH_TRACING_MODE not OTEL_ENABLED; registerTelemetry not experimental_telemetry. Read the resolver, pin the instrumentation.