Learn AI Observability with Phoebe

Two thirds of the run was inside no span at all

Tuesday, 09:14. An analytics bot starts sending executives stale revenue numbers, intermittently. No error is raised. No alert fires. The p95 latency dashboard is green and the answers read as confidently as ever. The failing step is not slow, not broken and not lying - it is simply not instrumented, so as far as the trace is concerned it cannot fail at all. This course is about the trace: spans and the OpenTelemetry GenAI conventions with their real current attribute names, instrumenting an agent whose loop is width rather than depth, per-span cost and latency that survive that loop, the failure taxonomy that maps every mode to the one signal that reveals it, sampling that keeps the slow traces instead of a random 1%, burn-rate alerting instead of thresholds that cry wolf, and the privacy split that lets you debug without hoarding your users' prompts. At its centre, a live trace explorer: switch instrumentation coverage on one lever at a time and watch the visible wall clock climb from 32.4% to 100% while the diagnosis goes from zero of five questions answerable to five - then switch on full content capture and watch the diagnosis refuse to move while retention jumps ten and a half times.

2tracks
16sessions
8coverage rungs
5diagnostic questions
Where this sits among its siblings Learn Evals owns retrieval and generation metrics. Agent Launch owns the go/no-go decision. AI Agents owns what an agent is. AI Infrastructure owns the platform layers. This course owns the one thing they all assume and none of them teach: the trace. Evals tell you whether it works. Observability tells you what actually happened.

The leader track 🎯 · for the people who own the on-call rota · 6 x 45 min · no tools required

Judgement mode: why a green dashboard held through a real incident, what to instrument before you know what you are looking for, which dashboards answer questions and which merely glow, what AI actually costs you once you count failed attempts, why static alert thresholds produce the same fatigue that made alarm hazards the top health-technology hazard two years running, and whether to buy, build or stay OTel-native.

The builder track πŸ› οΈ Β· for AI engineers, platform engineers & technical PMs Β· 10 x 45 min Β· hands-on

Craft mode: spans and traces from first principles, the current GenAI conventions including the names that changed under you, instrumenting a real agent loop, cost and latency attribution that survives it, a live root-cause hunt through a broken trace, the empirical failure taxonomy, sampling and what observability costs, burn-rate alerting, drift you did not deploy, and the privacy split that keeps you debuggable and compliant at once.

🧱 Builder 1 · easy

Spans and traces

Six words the course runs on, why a log line stops working the moment there is a loop, and the most useful number you can compute today: unaccounted time.

β–Ά Builders start here
πŸ“— Builder 2 Β· easy

OTel GenAI conventions

The exact current attribute names with their requirement levels - including the one that no longer exists, and the five events that collapsed into one.

The reference
πŸ•ΈοΈ Builder 3 Β· medium

Instrumenting an agent

The loop is width, not depth. Why plan owns its model call while tools are siblings, and why upstream ships no instrumentation for the framework you use.

The tree
⏱️ Builder 4 · medium

Cost and latency per span

Attribution that survives an agent loop, the double-counting trap, and the one ratio that tells you a faster model will not help.

Per-span truth
πŸ” Builder 5 Β· hands-on

Reading a broken trace

The live hunt. Eight coverage rungs, 32.4% of the wall clock to 100%, zero of five questions to five - and one lever that changes nothing but the bill.

β˜… The live trace
πŸ—‚οΈ Builder 6 Β· hands-on

The failure taxonomy

Fourteen measured failure modes from 150 annotated traces. The biggest bucket is not "the model was dumb" - it is specification and design, at 44.4%.

Mode to signal
🎚️ Builder 7 · hands-on

Sampling and its own cost

You want the slow traces, and head sampling cannot give them to you. Tail sampling, the 30-second default that breaks on a 20-minute agent, and cardinality.

What to keep
πŸ”” Builder 8 Β· hands-on

Alerting and SLOs

Burn rate instead of thresholds. Five reasons static limits misfire on LLM systems, and the alerts that fire at HTTP 200.

Waking up right
🌊 Builder 9 · hardest

Drift and silent changes

Four kinds of drift, three ways a provider changes the model under you, and the embedding mismatch that degrades retrieval to noise while every span stays green.

No deploy needed
πŸ” Builder 10 Β· hardest

Privacy and retention

Metadata at one hundred percent, content on a sample. The split that keeps you debuggable and compliant, and the model choice that overrides your policy.

Keeping it lawful
easy - everyone medium hands-on / deciding hardest

Choose your path πŸ—ΊοΈ

Four doors into the same conviction: an un-instrumented step cannot fail, as far as your trace is concerned.

🎯 I own the on-call rota a1β†’ a2β†’ a3β†’ a4β†’ a5β†’ a6
πŸ› οΈ I instrument the systems b1β†’ b2β†’ b3β†’ b4β†’ b5β†’ b6β†’ b7β†’ b8β†’ b9β†’ b10
⚑ Something is broken right now b1β†’ b5β†’ b6
πŸš€ Full journey (instrument AND decide) a1-a6β†’ b1-b10

The knowledge map 🧠

Both tracks at a glance - hover a session to spotlight its concepts, click any node to jump in.