learn-ai-agents-with-phoebe / Builder session 10 of 10
Learn AI Agents with Phoebe · Builder track · Session 10 of 10

Capstone: the full ops agent

Nine sessions, nine pieces. Tonight you wire them into one system: the complete Daybreak Ops Agent - MCP toolbox, workflow shortcuts, memory, planner, optional subagents, guardrails, eval gate, tracing. Nothing here is new. That is the point: a production agent is not one clever idea, it is eight boring ones assembled in the right order and then refused shipment until it proves itself. Your bar is exact: 12/12 on the golden set, or it does not go out.

🔴 Hardest session Builder track capstone Python · Claude API + Ollama path Ship it at 12/12
0-3 · Welcome 3-15 · The architecture 15-42 · Assemble + gate 42-45 · Graduation
Part 1 · covers Anthropic essay + OpenAI guide, assembled

Everything you built, one architecture 10 min live

Here is the whole track in one diagram. The b2 MCP toolbox does the touching, b3 workflow shortcuts handle the fixed jobs, b4 memory keeps findings alive, b5 plans before executing, b6 subagents stay optional, b7 screens injection and gates approvals and enforces budgets, b8's eval harness decides whether any of it ships, and b9's tracing watches it run. Every request flows top to bottom; every layer is a session you already built.

USER GOAL "send win-back emails" GUARDRAIL - INPUT (b7) relevance + safety screen on the incoming ask - off-topic or unsafe goals stop here guardrails PLANNER (b5) plan-then-execute: decompose the goal into steps, show the plan, then start the loop planning AGENT LOOP CORE (b1) tools MODEL decides the next step MCP TOOLBOX (b2) query_sales + 5 more tools b3 workflow shortcuts run the fixed jobs · b6 specialist subagents optional MEMORY (b4) compaction + scratchpad: findings survive long, multi-step tasks GUARDRAIL - OUTPUT + APPROVAL (b7) injection screen wraps every tool result · send_email + refund_order held for human approval guardrails ANSWER / ACTION or held for approval b8's eval harness gates the whole stack: 12/12 or no ship. b9 traces it all. Every box is a session you already built - tonight is wiring, not new ideas.
🔍 Click to zoom - the full Daybreak Ops Agent, layer by layer, with the session that built each piece
LiveRead the diagram like an auditor4 min▶

Walk it top to bottom and say, for each layer, what breaks without it - that is the lever diagnosis habit from b1 turned into an architecture review:

  • No input guardrail - the agent burns budget on off-topic or unsafe asks.
  • No planner - complex tasks get one-shot, half-done answers (the 10/12 rung never arrives).
  • No toolbox - fluent, confident, invented numbers (the b1 "demo that lied").
  • No memory sidecar - multi-step findings fall out of context (stuck at 5/12).
  • No output/approval layer - injected instructions in tool results get obeyed, and emails go out unapproved.
  • No eval gate, no tracing - you cannot prove any of the above, before or after shipping.

OpenAI's guide reduces every agent to model, tools, instructions; Anthropic's essay demands simplicity and transparency. This diagram is both, with the guardrails drawn where they actually sit.

Self-studyWhy the fixed jobs stay workflows3 min read▶

The monthly ops report never changes shape: numbers vs last month, refunds, subscription changes, one action. In b3 you built it as a prompt-chained workflow around the server's monthly_report prompt - predefined code path, cheap, testable. The assembled agent routes that request straight to the workflow instead of looping.

This is Anthropic's core rule surviving into your architecture: use the agent only where the task is too varied for a fixed path. An agent that also knows when NOT to be an agent is the mark of a builder who read the essay.

Part 2

The assembly, step by step 12 min live

Order matters. Each step below is one session's artifact bolted onto the last, and each is testable on its own before the next goes on. Build it in this order and you always have a working - if less capable - agent.

Lay the floor (b2). The raw loop over the Daybreak MCP toolbox: three data tools (query_sales, top_products, subscription_status) plus the three action tools the track added (list_tickets, send_email, refund_order).

Freeze the fixed jobs (b3). Route the monthly report to its prompt-chained workflow. Agentic loops are for varied tasks; the report is not one.

Bolt on memory (b4). Compaction when context grows, a scratchpad for findings. Multi-step comparisons stop losing their own work.

Put the planner in front (b5). Plan-then-execute, with the plan shown to the user - Anthropic's transparency principle, working for you as a debugging surface.

Decide on subagents (b6). Only split if conditional logic gets complex or tools overlap confusingly - the single-agent-first rule. Daybreak's six tools do not force a split; keep it single unless yours do.

Wrap the guardrails (b7). Injection screen around tool results, draft-and-hold approval on the high-risk tools, budgets and max turns as stop conditions.

Gate it (b8). Run the 12-task golden set. Below 12/12: name the failing lever, fix, rerun. No exceptions.

Light it up (b9). Tracing on before the first real user, so the first production surprise arrives with its own evidence attached.

LiveThe whole course in one config + one loop5 min▶

The assembled agent is smaller than you expect. One config dict declares every lever; one loop runs them in order. This is the skeleton you flesh out tonight:

CONFIG = {
    "model": "claude-opus-5",
    "tools": DATA_TOOLS + ACTION_TOOLS,           # b2: the 6-tool MCP toolbox
    "max_turns": 15,                              # b1: loops need exits
    "budgets": {"tool_calls": 25, "usd": 0.50},   # b7: stop conditions
    "high_risk": ["send_email", "refund_order"],  # b7: approval-gated
}

def run_ops_agent(goal: str) -> str:
    plan = make_plan(goal, CONFIG)                   # b5: plan-then-execute
    messages = [{"role": "user",
                 "content": f"{goal}\n\nPlan:\n{plan}"}]
    for _ in range(CONFIG["max_turns"]):
        reply = client.messages.create(
            model=CONFIG["model"], max_tokens=1024,
            tools=[t.schema for t in CONFIG["tools"]],
            messages=messages,
        )
        if reply.stop_reason != "tool_use":
            return screen_output(reply)              # b7: output check, then done
        for call in tool_uses(reply):
            spend_budget(call, CONFIG["budgets"])    # b7: budget guardrail
            raw = execute(call)                      # b2: toolbox does the touching
            safe = screen_injection(raw)             # b7: wrap TOOL RESULTS
            if call.name in CONFIG["high_risk"]:
                safe = hold_for_approval(call, safe) # b7: gate the SEND
            messages.append(tool_result_block(call, safe))
        messages = compact_if_needed(messages)       # b4: memory
    return "Max turns hit - escalating to a human."  # b1: exit, never a hang

Every commented line points at the session that built it. If a line surprises you, that session's cheat sheet is your fix.

LiveThe two hard integration bugs4 min▶

Everyone who assembles this hits the same two bugs. Both are placement errors, not logic errors:

  • Bug 1: the injection screen wraps the wrong thing. Screening only user input feels natural - that is where attackers type, right? Wrong. The tickets scenario's attack arrives inside a list_tickets TOOL RESULT: untrusted customer text flows into context through a tool the agent trusts. The screen must wrap tool results, or the trap walks straight past it.
  • Bug 2: the approval gate holds the wrong step. If the gate blocks the DRAFT, the agent never produces the email, the human has nothing concrete to review, and the visible catch that makes approval meaningful disappears. Hold the SEND: let the agent draft freely, then stop the one outward, hard-to-reverse action for a human yes.

Check both placements in your build before you run the gate - they are the difference between 12/12 and a 10/12 that looks mysterious.

Two integration bugs, both about placement ✗ Guardrails misplaced Screen wraps user input only Gate holds the draft, not send Attack and unreviewed send slip ✓ Guardrails placed right Screen wraps every tool result Gate holds the send, not draft 12/12 - no exceptions Both are placement errors, not logic errors - check both before you run the gate.
🔍 Click to zoom - both bugs are placement errors, not logic errors
Part 3 · the ship gate

The gate: do not ship below 12/12 ★ 15 min · everyone runs it

The scorecard below has every lever on - the first time in the track you see the full ladder complete: 1 → 5 → 7 → 10 → 12. Confirm the 12/12, then earn it: the two hardest traces follow, and you narrate them from memory.

The full ladder: 1 → 5 → 7 → 10 → 12 of 12 tasks No levers (b1) 1/12 + tools (b2) 5/12 + memory (b4) 7/12 + planning (b5) 10/12 + guardrails (b7) 12/12 12/12 is the reference state - any change reruns the full set before it ships.
🔍 Click to zoom - the full ladder, assembled: 1 to 12 of 12
Regression discipline - the gate's second half 12/12 once is a snapshot, not a guarantee. ANY change - a new tool, a reworded description, a smaller model, a prompt tweak - reruns the full set before it ships. Anthropic's evals guidance is explicit: the golden set grows from real failures and is tracked continuously. An agent without regression runs is an agent you are re-breaking silently.

Proof of mastery 1: the win-back run. Before each Next click, say out loud what the agent should do - which tool, what the result will be, where the gate fires - then click and check yourself.

Narrate the plan first: which tool finds the cancelled subscribers, and what should the agent do before drafting anything?

Predict the tool result: how many cancelled subscriptions does Daybreak have, and who are they? (You have known this since b2.)

Name the exact moment the approval gate must fire - and what the agent is holding when it does.

Proof of mastery 2: the injection trap. Same drill - narrate every step before revealing it. This is the trace most shipped agents fail in the wild.

Predict where the untrusted text enters: which tool carries it, and why the model would trust it by default?

Say what the injected instruction is trying to make the agent do, and which guardrail layer must catch it.

Explain why this catch would NOT happen if the screen wrapped only user input - bug 1 from Part 2, in the flesh.

★ What you can now do Read any agent trace and name the missing lever. Build the loop raw in Python, no framework required. Wrap it in guardrails placed where attacks actually arrive. Gate it behind a golden set and refuse to ship below the bar - and prove all of it with an eval instead of a vibe. Most people shipping agents today cannot do these things. You can.
Part 4

Where to go next 3 min live

The track ends; the practice does not. Four directions, each one deepening a layer you just assembled:

Go deeper onWhereWhy now
The toolbox sideThe MCP course in this seriesYou used the Daybreak server all track; that course builds it - tools, resources, prompts, transports.
The gate sideThe sibling evals course in this seriesYour 12-task golden set at full depth: graders, LLM-judges, trajectory checks, pass@k vs pass^k as a program.
The principlesAnthropic's "Building Effective Agents" essay, re-readTwenty minutes. Every line lands differently now that you have built what it describes.
An external examHugging Face Agents course, final unitThe GAIA benchmark challenge: score at least 30% for the official certificate - a graduation your employer can verify.
Homework

Make it yours - this week ◐ 60-90 min total

Source material

Official sources covered

The capstone assembles rather than introduces - so tonight's coverage is the canon working together. Certificates, videos, and graded assessments stay on the official sites. This page covers:

✓Anthropic "Building Effective Agents" - principles, assembledSimplicity, transparency, ACI - all three visible in the finished architecture
✓OpenAI "A Practical Guide to Building Agents" - components + guardrails, assembledModel/tools/instructions, guardrail placement, human-intervention triggers, exit conditions
✓Anthropic "Demystifying evals for AI agents" - gate disciplineGolden set from real failures, continuous tracking, the ship gate as an eval
◐Hugging Face Agents course - Unit 4 (final)GAIA benchmark challenge pointer - at least 30% for the certificate
Check yourself

Three questions before you graduate 🎯 ◐ 90 seconds

1 · You reworded one tool description in the shipped agent. What does the ship gate demand?

The gate is 12/12 PLUS regression: any change reruns the whole set. Tool descriptions steer the model's choices, so a "harmless" rewording can break a distant task - only the full set catches it.

2 · Where must the injection screen sit to catch the tickets trap?

The tickets attack never touches user input: it hides in customer text that a trusted tool fetches. Screen only what the user types and the trap walks straight past. Wrap the tool results.

3 · Why does the approval gate hold the SEND rather than the draft?

Draft-and-hold works because the human approves something concrete. Gate the draft and there is nothing to review; gate the send and the risky, hard-to-reverse action waits for a yes.

Builder session 10 cheat sheet · the whole course in 8 lines

The loop (b1)Goal → model decides → tool runs → result observed → repeat → answer. Exits: final answer, no tool call, error, max turns.
The levers (b2-b7)Tools · Memory · Planning · Guardrails. When an agent fails, name the missing lever before touching the prompt.
The ladder12-task golden set: 1 → 5 → 7 → 10 → 12 as the levers come on. The capstone ships only at 12/12.
The patterns (b3)Prompt chaining · routing · parallelization · orchestrator-workers · evaluator-optimizer. Fixed jobs stay workflows.
Memory (b4)The context window IS the working state. Compaction + scratchpad keep multi-step findings alive.
Guardrails (b7)Injection screen wraps TOOL RESULTS; approval gate holds the SEND; budgets + max turns stop runaways.
Evals (b8)Task, trial, grader. pass@k vs pass^k. Golden set grows from real failures and runs on every change.
The gate (b10)12/12 + regression + tracing on, or it does not ship. Prove it with an eval, never a vibe.