A taxonomy is only useful if each entry comes with a detector
Evals tell you whether it works. Observability tells you what actually happened. Most "agent failure modes" lists you will find are somebody's intuition arranged into bullet points, and intuition is a bad guide here because the failures that hurt you are the ones that do not look like failures. This session uses the one taxonomy built from hand-annotated production traces, and it insists on the same discipline for every row: name the mode, name the field in the telemetry that reveals it, name the fix. A mode with no detector is a story, not an operating procedure.
MAST, and the number everybody gets wrong 8 min live
Cemri et al., Why Do Multi-Agent LLM Systems Fail? (arXiv:2503.13657, NeurIPS 2025 Datasets & Benchmarks) is the only agent failure taxonomy in this course with an inter-annotator agreement figure attached to it. 150 hand-annotated traces, kappa 0.88. MAST-Data extends it to more than 1600 traces across seven named frameworks - ChatDev, MetaGPT, HyperAgent, AppWorld, AG2, Magentic-One, OpenManus - and an LLM annotator reproduces the expert labels at 94% accuracy, kappa 0.77. That last figure is what makes the taxonomy operational rather than academic: you can label your own traces at volume.
The full fourteen, with v3 frequencies
| Failure mode | v3 freq | What it means | First place it shows in telemetry |
|---|---|---|---|
| FC1 · Specification and system design · 44.4% | |||
| Disobey task specification | 11.8% | The agent does something adjacent to what was asked | Output assertion failure with span status OK |
| Disobey role specification | 1.5% | An agent steps outside its assigned role | Tool names appearing under the wrong gen_ai.agent.name |
| Step repetition | 15.7% | The same step executed again and again | Repeating (tool, args-hash) pairs; span count above p99 |
| Loss of conversation history | 2.80% | Earlier context vanishes mid-run | gen_ai.conversation.compacted; input tokens dropping mid-conversation |
| Unaware of termination conditions | 12.4% | The agent does not know it is finished | gen_ai.invoke_agent.tool_calls in the high buckets; duration rising with no new span types |
| FC2 · Inter-agent misalignment · 32.35% | |||
| Conversation reset | 2.20% | The thread restarts from nothing | New gen_ai.conversation.id mid-session; input tokens back to baseline |
| Fail to ask for clarification | 6.80% | Ambiguity resolved by guessing | Zero clarifying turns on requests a human would query |
| Task derailment | 7.40% | The run drifts to a different task | Tool distribution shift within one trace |
| Information withholding | 0.85% | One agent does not pass on what it knows | A retrieved document id never appearing downstream |
| Ignored other agent's input | 1.90% | A handoff result is discarded | Sibling invoke_agent output absent from the next prompt |
| Reasoning-action mismatch | 13.2% | The agent says one thing and does another - including reading an error and answering as if it had not | Leaf span in error, root span OK. This is the absorbed-error pattern in Part 2 |
| FC3 · Task verification · 23.5% | |||
| Premature termination | 6.20% | The run stops before the task is done | Short trace with a missing expected span type |
| No or incomplete verification | 8.20% | Nothing checked the result | Absence of a verification span - a coverage gap, not an error |
| Incorrect verification | 9.10% | Something checked it and got it wrong | Verification span present and OK, output assertion still fails |
LiveWhy the largest bucket being "design" changes what you build4 min▶
If you believe agent failures are mostly the model being insufficiently clever, the roadmap writes itself: wait for the next model, spend on prompt engineering, add more examples. That roadmap is aimed at somewhere between a quarter and a third of your failures.
MAST v3 says the plurality of failures - 44.4% - come from specification and system design. Two of the five modes in that category, step repetition at 15.7% and unaware-of-termination at 12.4%, are things a loop counter detects. Together they are 28% of all documented failures, and the fix for both is written in configuration rather than in prose: a hard cap on steps, a machine-checkable termination condition, deduplication of identical calls.
That is a genuinely encouraging finding, and it is why this course exists. The largest slice of the problem is not waiting on a frontier model. It is waiting on somebody to count the spans.
Self-studyWhat MAST does not cover, and where the numbers came from3 min read▶
Three honest limits worth stating before you build a dashboard on these percentages.
- The corpus is multi-agent systems. The seven frameworks in MAST-Data are agent frameworks - ChatDev, MetaGPT, HyperAgent, AppWorld, AG2, Magentic-One, OpenManus. A single-agent retrieval bot will not have the same distribution, and the FC2 inter-agent category partly does not apply to it. Use the taxonomy as a checklist, and derive your own frequencies from your own traces.
- The percentages are frequencies within an annotated failure set, not a base rate. They tell you how failures divide up, not how often systems fail. Those are different questions and only the first one is answered here.
- The v1/v2 versus v3 split is the reason to name your version. This course could not extract v1's own table to reconcile the 41.8/36.9/21.3 triplet line by line, so it is cited only as "the earlier figures". If you need that reconciliation, read v1 directly rather than trusting a summary.
The mechanically useful part is the 94% LLM-annotator accuracy. It means the taxonomy can be applied to your own trace store as a batch job, which turns a paper into a monthly report.
The three shapes an error takes on its way up 7 min live
Before the table, one diagram, because it explains why the table's middle rows are the hard ones. An error at a leaf span has exactly three ways to reach the root, and only one of them is honest.
The Beacon incident is pattern 2 exactly. The warehouse query times out after 4.1s and returns zero rows, a try / except swallows it, the fallback serves a 19.4-hour-old cache entry, and the model faithfully summarises stale data. Every span is green. The reason no alert fired is not that the monitoring was badly configured; it is that status-based monitoring is structurally incapable of seeing an absorbed error.
Failure mode to telemetry signal to fix
| Failure | The signal that reveals it | The fix |
|---|---|---|
| Wrong tool selection | gen_ai.tool.name distribution shift; a tool result that is never referenced downstream | Shrink and disambiguate the tool set. Anthropic's own test: "If a human engineer can't definitively say which tool should be used in a given situation, an AI agent can't be expected to do better" |
| Tool argument errors | execute_tool span carrying error.type; a tool_result with is_error: true; retry-same-tool-different-args pairs | strict: true on tool definitions. Return machine-actionable error text, not a stack trace |
| Retrieval returns nothing useful | Retrieval span present but low max similarity; the answer carries no citation; downstream input length equals the no-context baseline | Log retrieved ids and scores as span attributes, so null retrieval is visible rather than inferred |
| Context exhaustion and truncation | stop_reason: "max_tokens" and "model_context_window_exceeded" - the docs say treat both as truncated; cumulative input tokens climbing across one conversation id | Compaction, structured note-taking outside the window, sub-agents. Alert on approach to the limit, not on the error |
| Context rot - decay well before the limit | A fixed canary eval score plotted against input length, not against the limit | Chroma's Context Rot, across 18 models: "LLMs do not maintain consistent performance across input lengths". Distractors hurt even at n=1, and models did better on shuffled haystacks than logically structured ones. Anthropic frames the same thing as an attention budget: "Every new token introduced depletes this budget" |
| Runaway loops | Span count per trace above p99; the same (tool, args-hash) repeating; duration rising with no new span types; stop_reason: "pause_turn" from the server-tool 10-iteration default | Hard caps on steps, tokens and wall clock. Explicit machine-checkable termination conditions. Deduplicate identical calls |
| Silent partial failure - the hardest class | An empty 2-3 token response carrying stop_reason: "end_turn" at HTTP 200 - no error, no content, documented as "typically occurs... particularly after tool results". Also a tool_use block with no matching result block | Assert on the output, never on the status code. Post-conditions per step |
| Cascading retries | A retry-attempt counter as a span attribute - invisible if you never emit it; cost per trace going bimodal; parent duration roughly N times leaf duration | Idempotency keys, retry budgets rather than per-call retries, circuit breakers |
| Rate limits and 429s | 429 plus retry-after plus the anthropic-ratelimit-* header family. Alert on *-remaining trending to zero, not on the 429 - by the time you see the 429 you have already dropped the request | Respect token-bucket enforcement: "a rate of 60 RPM might be enforced as 1 request per second", so bursts fail while you are nominally under quota. Acceleration limits will 429 you for ramping too fast |
| Mid-trajectory timeout | Root span in error plus a dangling child span with no end timestamp; p99 clipped exactly at your client timeout value | Checkpoint state so a run resumes rather than restarts. Budget for it: "A single 300k-token generation can take over an hour" |
| Run-to-run non-determinism | Same input, different output; divergence at a specific token index | Part 3. The fix is not reproduction, it is capture |
| Refusals | stop_reason: "refusal" with stop_details, returned as HTTP 200, not an error - completely invisible to error-rate alerting | Track refusal rate as a first-class SLI, segmented by feature and tenant |
stop_reason: "end_turn". A truncation with stop_reason: "max_tokens" or "model_context_window_exceeded". A refusal with stop_reason: "refusal". And an absorbed error where your own code caught the exception. All four are successes to an error-rate dashboard. Any monitoring built on status codes alone is blind to the majority of the failure modes in the table above, which is the entire reason this session exists.
Non-determinism, and the end of "just run it again" 6 min live
Every debugging instinct you have from ordinary software rests on reproduction: find the input, replay it, watch it fail, fix it. That instinct does not transfer, and there is a clean measurement of why.
LiveThe quote that ends the argument3 min▶
Thinking Machines Lab, Defeating Nondeterminism in LLM Inference (10 September 2025), took the prompt "Tell me about Richard Feynman", ran it 1000 times against Qwen3-235B at temperature 0, and got 80 unique completions. The most common one appeared 78 times. All 1000 were byte-identical through token 102 and diverged at token 103, where 992 said "Queens, New York" and 8 said "New York City".
The interesting part is the diagnosis. It is not GPU nondeterminism - the kernels are run-to-run deterministic. They are simply not batch-invariant: the numerical result of a matmul depends on the batch shape it was executed in, and the batch shape depends on how many other people are hitting the same server at the same moment. In their words, "the other concurrent users are not an 'input' to the system but rather a nondeterministic property." With batch-invariant kernels in place, all 1000 completions came back identical.
Sit with the consequence for a second. The other tenants on your provider's inference server are an uncontrolled input to your output. You cannot set them, log them, or replay them. Therefore you cannot replay your own production request and expect the same trajectory. Whatever you needed to know about that run, you needed to have captured while it was running.
And the one lever people reach for here has been taken away: on Claude Opus 4.7 and later, temperature, top_p and top_k are deprecated and return a 400 on non-default values. Temperature-0 reproducibility is not merely unreliable, it is not on the menu.
Self-studyThree habits that follow from non-reproducibility3 min read▶
- Never write a bug report that depends on a rerun. "I ran it again and it was fine" is not evidence of a fix, and "I cannot reproduce it" is not evidence of no bug. Both are the expected behaviour of the system. Argue from the captured trace or do not argue.
- Alert on rates over samples, never on single requests. If 1000 identical inputs give 80 distinct outputs, any per-request quality threshold flaps by construction. Quality is a population statistic here even when the incident is one user's answer.
- Treat coverage as your reproduction budget. Every span you chose not to emit is a question you have permanently given up the ability to answer about that run. In ordinary software, unclear instrumentation is an inconvenience because you can rerun. Here it is final.
Make an absorbed error impossible to absorb ★ 12 min · everyone builds
Beacon's Insight Bot is pattern 2 from Part 2. We are not going to fix the warehouse timeout - that is somebody else's ticket. We are going to make it impossible for the same failure to reach an executive silently again, which is the observability job.
Find the try / except that swallows the warehouse timeout. In the except block, before you return the cached fallback, do three things: span.record_exception(e), span.set_status(Status(StatusCode.ERROR)), and set a plain-language attribute such as beacon.fallback.used = true. The status makes it visible to the dashboard; the attribute makes it queryable.
Now add the thing status cannot give you: the age of what you served. Emit beacon.cache.age_seconds on the span. The Beacon incident served a 19.4-hour-old entry - 69,840 seconds. That number was always available in the code and never left the process. One attribute converts an invisible incident into a chart.
Add an assertion on the output, not the status. For Beacon the assertion is a post-condition: the answer must cite at least one row id that was returned in this trace. Zero rows returned means the assertion fails, which means an error status on the root span, which means somebody is told - even though the HTTP response is a perfectly formed 200 and the prose reads beautifully.
Handle the four HTTP 200 failures explicitly on every model call, using the real stop_reason values. Truncation counts both max_tokens and model_context_window_exceeded. Refusal is refusal. The empty-response case is the nasty one: a 2-3 token body with stop_reason: "end_turn", most often right after a tool result.
Break it deliberately and confirm each detector fires. Force zero rows from the warehouse; confirm the root span goes red. Cap max_tokens at 16 and confirm the truncation flag appears. Then re-read the trace and check that a colleague who was not in this room could tell what happened from the attributes alone. If they cannot, the attribute names are wrong.
TRUNCATED = {"max_tokens", "model_context_window_exceeded"} · if resp.stop_reason in TRUNCATED: span.set_attribute("beacon.truncated", True); span.set_status(Status(StatusCode.ERROR)) · if resp.stop_reason == "refusal": span.set_attribute("beacon.refused", True) · if resp.stop_reason == "pause_turn": span.set_attribute("beacon.paused", True) · if not resp.content or len(text_of(resp)) < 40: span.set_attribute("beacon.empty_end_turn", True); span.set_status(Status(StatusCode.ERROR))
Build the loop and timeout detectors ★ 10 min · everyone builds
Two of MAST's control-loop modes - step repetition at 15.7% and unaware-of-termination at 12.4% - need no model understanding at all to detect. They are shape problems, and shape is cheap. Third, we catch the timeout that leaves a span half open.
Emit a span attribute for the loop index on every iteration of the agent's tool loop, and an args hash for every tool call: beacon.step and beacon.tool.args_sha. Step repetition is now a duplicate-detection query rather than a reading exercise. Run your last week of traces through it and count.
Record the two agent-call-count metrics the GenAI conventions already define - gen_ai.invoke_agent.tool_calls and gen_ai.invoke_agent.inference_calls. Both are histograms with buckets running 1, 2, 4, 8, 16, 32, 64, 128. The high buckets exist precisely so a runaway loop becomes visible without anybody opening a trace. Note which buckets your agent currently lands in; that is your loop baseline.
Now the diagnostic that separates a legitimately long run from a loop: duration rising with no new span types. A run that gets slower because it is doing more kinds of things is working. A run that gets slower doing the same three span names over and over is stuck. Compute distinct span names per trace alongside span count and plot them together.
Find your dangling spans. Query for child spans with no end timestamp whose root span is in error - that is the mid-trajectory timeout signature. Then check whether your p99 latency is clipped flat at exactly your client timeout value. If it is, your p99 is not measuring your system, it is measuring your own configuration.
Last, turn the 429 alert around. Stop alerting on 429 responses and start alerting on the anthropic-ratelimit-*remaining headers trending toward zero. Token-bucket enforcement means "a rate of 60 RPM might be enforced as 1 request per second", so a burst can fail while your average utilisation dashboard shows comfortable headroom. The 429 is the post-mortem; the remaining counter is the warning.
Try it yourself - this week ◐ 50-60 min total
- Take the fourteen MAST modes and mark each one detected, detectable or invisible in your current telemetry. Invisible means you could not tell the difference today between it happening and not happening. Count the invisible ones; that number is your session b6 backlog and it is usually more than half.
- Grep your codebase for every
exceptin the path of an AI request and check each one for two things: does it set span status, and does it record what the user received instead. The second is the one nobody has. Every miss is a potential absorbed error. - Query a week of traces for the four HTTP 200 failures - empty
end_turnbodies, both truncation reasons, andrefusal. If you cannot run that query because you never storedstop_reason, that itself is the finding, and it is a one-line fix. - Plot one canary eval score against input length rather than against your context limit. Context rot means quality decays well before the limit is reached, so a graph that only marks the limit tells you nothing about the region you actually operate in.
- Run the non-determinism experiment on your own stack: one prompt, fifty calls, count distinct outputs and find the token index where they first diverge. Show the result to whoever last asked you to "just reproduce it". This is the fastest way to win that argument permanently.
Official sources covered
Every number on this page comes from a primary source catalogued in this course's source map, and each is cited with its version. Where a claim is derived rather than published, this page says so.
stop_reason semantics: max_tokens, model_context_window_exceeded, refusal, pause_turn, and the empty 2-3 token end_turn body at HTTP 200Part 2 + Demo 1 · the four failures that are not errorsanthropic-ratelimit-* header family, token-bucket enforcement, acceleration limitsPart 2 + Demo 2 · alert on remaining, not on the 429gen_ai.invoke_agent.tool_calls and .inference_calls histograms, buckets 1 to 128Demo 2 · the runaway loop as a metric rather than a reading exerciseThree questions before you go 🎯 ◐ 90 seconds
1 · In MAST v3, which category holds the largest share of documented failures, and what does that imply for your roadmap?
The headline of MAST is that the largest bucket is not "the model was dumb". It is design. And the two biggest modes inside it are countable: the same step executed repeatedly, and an agent that does not know it is finished. Both show in span counts and in the gen_ai.invoke_agent.tool_calls histogram. Note the version: v3 is 44.4 / 32.35 / 23.5, and the widely quoted 41.8 / 36.9 / 21.3 are the earlier v1/v2 figures.
2 · A leaf execute_tool span is in error. The agent reads the error text, apologises in prose, and returns a confident answer. The root span is OK and the HTTP status is 200. Which detector catches this?
This is the absorbed pattern, and it is MAST's reasoning-action mismatch at 13.2%. The whole point is that the status signal was consumed by your own exception handler, so nothing status-based recovers it. The Beacon incident is exactly this shape: a swallowed warehouse timeout, a 19.4-hour-old cache entry served as fresh, and every span green. Assert on what the user received, not on whether your code threw.
3 · One prompt, temperature 0, 1000 completions, 80 unique results, all identical through token 102. What is the operational consequence?
The cause is not GPU nondeterminism - the kernels are run-to-run deterministic but not batch-invariant, and batch size varies with concurrent load, so "the other concurrent users are not an 'input' to the system but rather a nondeterministic property". With batch-invariant kernels all 1000 matched. C fails twice: temperature was already 0 in the experiment, and on Claude Opus 4.7+ temperature, top_p and top_k are deprecated and return 400 on non-default values. Capture, do not replay - which is the argument for tracing.
Builder session 6 cheat sheet · pin this
end_turn; max_tokens AND model_context_window_exceeded; refusal; your own caught exception. Zero of them raise an error.anthropic-ratelimit-*remaining trend to zero. Token-bucket enforcement means 60 RPM can be enforced as 1 per second, so bursts fail under quota.