Why this is the last session and not the first
Most teams start here, with a vendor bake-off, and it is the wrong end of the problem. A platform can only show you what you chose to emit, so a tooling decision taken before a coverage decision buys you a beautiful view of the wrong half of your system. Five sessions of deciding what matters is what makes this session short.
The market splits three ways, not two 14 min live
Buy versus build implies two options. In practice there are three lanes, and most bake-offs go wrong by comparing candidates from different lanes on the same criteria. Decide the lane first, then compare inside it.
LiveThe candidate table, with the ingest facts that actually decide things6 min▶
The columns worth arguing about are ingest and self-host, because those two are what determine whether a switch later is a configuration change or a project. Everything else is product taste, which changes faster than your instrumentation does.
| Platform | Ingest | Self-host | Licence |
|---|---|---|---|
| Langfuse | OTLP-native. HTTP JSON and protobuf, no gRPC, Basic auth. Reads three dialects | Yes | MIT except ee/, web/src/ee/, worker/src/ee/ |
| Arize Phoenix | OTLP-native, OpenInference dialect | Yes, described as free to self-host with no feature limitations | Elastic License 2.0, not plain OSS |
| LangSmith | Proprietary SDK plus an OTLP receiver | Enterprise tier | Proprietary |
| Braintrust | OTLP-native, needs a parent-project header | Not verified. Ask | Proprietary |
| W&B Weave | OTLP-native, project name carried as an OTel resource attribute | Dedicated and self-managed offerings | Repo is Apache-2.0 |
| Datadog LLM Obs | Proprietary SDK only. No documented OTLP path for LLM Observability | No | Proprietary. Billing based on the volume of spans you send |
| Grafana Cloud / Tempo | OTLP-native, generic. AI dashboards exist over conventional keys | Tempo is open source | Licence string not verified. Check the repo |
| Honeycomb | OTLP-native, generic | No | Proprietary. LLM feature set not verified |
Two things to notice before the room starts arguing about features. First, seven of the eight take OTLP, which means the wire is not the differentiator and portability is not really at stake at the transport layer. Second, the one that does not is also the one with the richest product, which is not a coincidence and is exactly the trade being offered.
Interop happens at the backend, and the default is inverted 14 min live
Lock-in feels like a transport problem and it is not, because everybody speaks the same transport. It is a vocabulary problem. Two libraries can send a token count over identical wire in two different spellings, and the platform on the far end is what reconciles them.
Both dialects emit valid OTLP. Every OpenInference trace is a valid OTLP trace, so nothing about the transport constrains you. But OTel GenAI treats prompt and completion content as opt-in and ships it off, while OpenInference captures content by default and expects you to switch it off. Nobody presents this as a policy question. It arrives as a library choice made by an engineer in an afternoon, and it decides whether your trace store contains every prompt your users typed. That is a governance decision disguised as a tooling decision, and it is the one place in this session where the default will make the call for you.
Self-studyThe dialect differences that matter to a leader, not to a spec reader4 min read▶
You do not need to memorise the key names. You do need to know which capabilities exist in which vocabulary, because a missing concept is a gap you will end up filling with bespoke attributes, and bespoke attributes are the thing that does not port.
| OTel GenAI conventions | OpenInference | |
|---|---|---|
| Content default | Opt-in, off. Metadata only until you decide otherwise | On by default, hidden with explicit flags |
| Cost | Not in the spec. You compute it yourself | First-class cost fields, including cache read and cache write |
| User identity | No user id at all. The general namespace has a session id | Has a user id field |
| Graph topology | Not modelled | Node, name and parent fields for graph frameworks |
| Stability | Development status, no tagged releases and nothing to pin | Versioned and used in production |
Two consequences for the buying decision. Cost and user attribution are the two things leaders ask for most, and in the conventional vocabulary neither exists, so somebody on your team will invent a spelling for them. Decide that spelling once, deliberately, at the application boundary, and write it down. That single act is the difference between a portable instrumentation layer and one that turns a platform migration into a rewrite.
Second, and worth saying plainly to anyone quoting a blog post: the conventional vocabulary is still moving. Every span, metric and attribute in it carries development status, the repository has no tags and no releases, and the key that most 2024 material uses for the provider name has already been deprecated. Teach it as a moving target and check the source rather than the documentation, which lags.
What ports, what you rebuild, and what you store 11 min live
Frame the buy-versus-build question as a ledger rather than a preference, and it stops being an argument about taste. Two columns, and one decision underneath them that costs more than either.
LiveRetention cost, and why the invoice arrives before the value does5 min▶
Retention is the line item that surprises people, because content capture does not raise your bill by a percentage. It raises it by a multiple, and it does so silently, since nobody reviews a telemetry volume the way they review a model bill.
The builder track puts a modelled number on this: on the course's constructed incident, turning on full content capture takes retention from roughly 4.6 GB to 48.1 GB for the same traffic, a factor of about ten, and the diagnosis score does not improve at all because the message content was never the problem. That arithmetic is modelled rather than measured, and it is labelled as modelled on screen, so substitute your own storage price before quoting it. The shape of the finding is the durable part: content is most of the bytes and rarely the answer.
| Question to answer before you sign | Why it decides the contract |
|---|---|
| What is the billing unit, and is it spans, events, ingested bytes or retained bytes? | Datadog LLM Observability states billing is based on the volume of spans you send. Dollar pricing for the other platforms could not be verified and has to come from the vendor in writing |
| Does content capture change the unit, or only the volume? | If you are billed on spans, a tenfold content increase may cost nothing extra to ingest and everything to retain. Those are two different lines |
| What is the default retention, and what does a longer tier cost? | An unbounded retention default is a compliance problem by construction, not merely a cost problem |
| Can you delete one individual's spans on request? | If you cannot, you cannot service an erasure request, and the trace store becomes the reason a data subject request fails |
| Who inside your company can read a prompt today? | Trace stores usually have broad internal read access. They deserve the controls you give a production database, and most do not have them |
| Where does the data physically go? | Content reaching a vendor endpoint is a transfer and a processor relationship. Scrub before ingestion, because after ingestion deletion is a request rather than a fact |
Pick your lane before you look at a single demo ◐ 12 min · whole group
Run this in the room with nobody's laptop open. Every question here is answerable from what your organisation already is, and answering them first turns a three-month bake-off into a two-week one.
Who reads the traces? If the answer is AI engineers debugging agent behaviour, lane one earns its keep because the native agent view is the product. If the answer is an existing SRE rota that already lives in one pane of glass, lane two is worth serious consideration even though you will build the views yourself.
Do you already have an APM contract? Say the renewal number out loud. A second observability vendor is a second procurement cycle, a second access review and a second place to look during an incident, and that overhead is real even when the licence is cheap.
Does anything about your data force self-hosting? If yes, the shortlist collapses to the self-hostable options, and you must read the actual licence: Phoenix is Elastic License 2.0 and Langfuse is MIT except its enterprise directories. Neither is plain permissive open source across the whole repository, and "it is open source" is not a statement your legal team can act on.
Is anyone proposing lane three? That is a legitimate choice, and the honest way to make it is to say out loud that your producer code becomes the vendor's SDK, since there is no documented OTLP path for their LLM Observability product specifically. Absence of a documented path is not proof that none exists, so ask them directly and get the answer in writing.
Now decide the attribute names, in the room. Cost and user identity do not exist in the conventional vocabulary, so somebody will invent a spelling for them. Pick it here, apply it at the application boundary, and write it into whatever document your team actually reads. This is the cheapest lock-in insurance available to you and it costs one meeting.
A single page: the chosen lane and the reason, the two or three candidates inside it, the self-host requirement stated as yes or no, and the agreed spelling of your non-conventional attributes. Then the list of questions you are sending the vendor in writing, which should include every item on this page's unverified list. A shortlist without that list attached is a shortlist built partly on guesses.
Make the content decision explicitly, before a library makes it ◐ 10 min · in pairs
This is the decision that gets made by a default if you do not make it on purpose. It is also the only decision in this session with a regulator on the other side of it, so it deserves a name, an owner and a date.
Find out what you are capturing right now. Not what the policy says. Open a real trace from a real user in whatever tool you already run and see whether the prompt text is in it. Half the rooms that do this exercise discover the answer is yes, because they installed a library whose default is on.
Write the default down as a sentence. "Metadata on 100% of traffic. Content on a consented sample of X%, in a separate store, retained for Y days, readable by this named group, with access logged." A default that is not written is not a default, it is whatever the library shipped.
Check the fields people forget. Redacting the prompt is not enough. Tool call arguments and tool results are flagged as sensitive for good reason, and the tool arguments are frequently where the account number is. Retrieved document text is the other one.
Allow-list rather than deny-list. Pattern-matching for personal data fails on the case nobody imagined, which for free text is most cases. Name the fields you keep, and drop everything else. Then scrub before ingestion, because once content reaches a vendor endpoint you have made a transfer.
Name the person who can authorise a content capture window. During a real incident somebody will ask for prompts, and they will be right to ask. Decide now who can say yes, for how long, and what gets deleted afterwards. That is the difference between a scoped exception and a permanent policy change made at 2am.
Try it yourself - this week ◐ 45-60 min total
- Open one production trace and check whether prompt text is in it. Then check whether tool call arguments are. Write both answers into the content decision document as the current state, because the current state is usually different from the intended state and the gap is the finding.
- Send every unverified item from this page to your shortlisted vendors as written questions: the billing unit and the dollar figure, self-hosting availability, the LLM feature set, the licence string for the component you intend to run, and whether an OTLP ingestion path exists for their LLM product specifically. Keep the replies with the contract.
- Grep your own instrumentation for attribute names that are not conventional and not on your agreed list. Every one of them is a small piece of lock-in you created yourself, and unlike the vendor's kind it is free to fix this week.
- Price two years of retention at your current volume, then at ten times that volume, and put both numbers next to the sentence about what content capture buys you diagnostically. Deciding this against real numbers takes an hour. Discovering it on an invoice takes a quarter.
- Ask whoever owns access control how many people in your company can currently read a customer prompt out of your trace store. If the number surprises you, that is the first ticket from this session, and it is more urgent than the platform choice.
Official sources covered
Every platform fact on this page was read from vendor documentation and repositories rather than from summaries, and the things that could not be read are listed as unverified rather than filled in. This page covers:
Nothing below was filled in from memory, and none of it should be quoted in your business case without a written answer from the vendor.
- Dollar pricing for every platform except Datadog's basis, which is documented as the volume of spans you send. No price per span, per seat or per gigabyte could be verified for any candidate on this page.
- Braintrust self-hosting. Whether a self-hosted deployment is offered, and on what terms, could not be confirmed.
- Honeycomb's LLM feature set and its pricing. It ingests OTLP generically, which is why it sits in lane two, but LLM-specific capability could not be confirmed either way.
- Grafana Tempo's licence string. Tempo is open source; the exact licence was not read from the repository, so read it yourself before your legal review.
- Whether an OTLP ingestion path exists for Datadog LLM Observability specifically. None is documented. That is a documentation absence, not a proven technical absence, and it is a question for their sales engineer.
Three questions before you go 🎯 ◐ 90 seconds
1 · Seven of the eight platforms in Part 1 ingest OTLP. So where does lock-in actually live?
Everybody speaks the same transport, so the transport is not the differentiator. Interop is resolved at the backend, where the platform normalises whichever dialect arrives, which is why one vendor documents reading three separate vocabularies. What that leaves you is a clean split: the instrumentation is portable if the keys are conventional, and the dashboards, alerts, saved queries and team habits are not portable at all. Budget for rebuilding the second column and spend your discipline on the first.
2 · An engineer picks an instrumentation library on a Friday afternoon. What decision have they just made on the company's behalf?
This is the governance decision disguised as a tooling decision. The conventional GenAI spec says instrumentations should not capture message content by default and should offer an opt-in. OpenInference does the reverse: content is captured unless you set the hide flags. Same valid OTLP on the wire, opposite consequences for what sits in your trace store on day one, and nobody in the approval chain was ever shown a choice. Which is why the answer to "are we capturing prompts" has to come from opening a real trace, not from reading the policy.
3 · A vendor publishes both that they monitor agent decision patterns without monitoring conversation contents, and that their privacy controls prevent engineers examining the interactions needed to reproduce bugs. How should you read that pair?
Both statements are true and they point in opposite directions, which is exactly what a real trade-off looks like when somebody is being straight about it. Poor coverage hides failures; rich content capture creates legal and security exposure. Splitting the difference by capturing a bit of content everywhere gets you the exposure without the diagnostic power. Splitting the decision instead - full metadata always, content rarely and briefly and under access control - is the only arrangement where both halves of that published pair can be honoured.
Leader session 6 cheat sheet · pin this
Six sessions, and the one sentence underneath all of them
You started with a green dashboard that held through a real incident, and you finish with a purchase decision. The thread connecting those two things is simple enough to say in a corridor: a system can only report what somebody decided it should report, and every session in this track was a version of that decision.
| Session | The decision it asked you to make |
|---|---|
| a1 · The trace is the receipt | Accept that "it works in evals" and "it worked at 09:14" are different claims, and that only one of them has a receipt |
| a2 · What to instrument | Choose coverage before you know what you are looking for, because the 62% you cannot see is where incidents live |
| a3 · Dashboards that answer | Build views around the three questions rather than around available metrics, and stop expecting an aggregate to reveal an intermittent fault |
| a4 · What AI actually costs you | Change the denominator to cost per successful task, and accept that reliability work is cost work |
| a5 · On-call for AI | Write the alerting configuration policy, page on three burn-rate rules, and add the two alerts that see silent failure |
| a6 · Buy, build, or stay OTel-native | Pick the lane, spell the attributes conventionally, and make the content decision before a library default makes it for you |