Where we are, and what this session is not
This is not a course on models, evaluation methodology, retrieval-augmented generation or agent frameworks. All four have their own courses on this shelf. Tonight is the architecture question only: where does this component sit, what budget does it get, and what happens to everything else when it misbehaves.
★ What you walk out with tonight. A latency and cost budget for one model-backed feature, a staleness owner for the retrieval index, a bound on the agent loop, and a straight answer to the question your leadership is currently getting wrong about throughput and stability.
The model box is the small box, and it has been for a decade
6 min liveSculley and colleagues made this argument at NeurIPS in 2015 about classical machine learning systems. Nothing about the last decade has weakened it. If anything, the surrounding boxes got bigger.
A service calls the model with the customer's question and returns whatever comes back.
p95 is whatever the provider's p95 is today, and it changed last Thursday without anyone telling you. Cost per answer is whatever the answer's length turned out to be. When the provider is degraded, your support page is degraded, and the failure looks like slowness rather than an outage, so the circuit never opens. Nobody can say whether answer quality moved this month.
A component with a timeout, a token ceiling, a cache, a fallback and a gate.
800 ms timeout with a deterministic fallback to the top three help articles. A token ceiling that makes cost per answer predictable within a band. A cache on the 200 questions that are 60 percent of volume. An eval that runs on every prompt change with a threshold and an owner. When the provider degrades, the page gets slightly worse instead of stopping, and someone is paged.
LiveThree properties that make this component awkward5 min▶
- Non-deterministic latency. Response time depends on output length, which depends on the input, which depends on the user. You cannot load-test this the way you load-test a query. Design with a timeout and a fallback, because a p95 you measured last month is not a promise.
- Non-deterministic cost. A per-request cost that varies by an order of magnitude with input. Without a token ceiling, one unusual input pattern can move a monthly bill more than a traffic spike would.
- Non-deterministic output. The same input can produce different output, which breaks the assumption every cache, test and diff you own is built on. This is why evaluation exists as a component rather than as a test suite.
Little's Law from session 1 does real work here. L = λW, and W is the part that is unstable. At 5 requests a second with an 800 ms response, 4 are in flight. When the provider degrades to 12 seconds, 60 are in flight. If your connection pool is 20, you now have an outage in a component that is technically still working.
The index is a table, and nobody is watching it
6 min liveA retrieval index is a derived asset built from source data on a schedule. That is the exact definition of everything in session 4's graph, and yet it is almost never in the graph.
LiveBounding the agent loop4 min▶
An agent is a model in a loop with tools. Architecturally that is a component that decides how much work to do, which is a thing no other component in your platform is allowed to do.
- Bound the iterations. A hard cap on tool calls per request. Not because the loop is dangerous, but because an unbounded loop has no cost estimate, and a component with no cost estimate cannot be in a budget.
- Bound the wall clock. A deadline for the whole request, separate from the per-call timeout. Ten calls of 800 ms is eight seconds, and every one of them was individually inside its budget.
- Bound the blast radius of the tools. This is session 2 and session 6 arriving at the same time: whatever the agent can call, it can call wrongly and repeatedly. Read-only tools by default, write tools behind an explicit boundary with its own authorisation.
- Make the loop observable. Log what it did, not just that it finished. "The request succeeded" is not a useful statement about a component that chose its own path.
LiveEvals and guardrails are infrastructure, not a notebook4 min▶
Treat the eval exactly the way session 2 treats a contract: it is a gate in a pipeline with an owner, a threshold and a schedule, or it is a document.
| Property | Notebook version | Infrastructure version |
|---|---|---|
| When it runs | when someone remembers | every prompt, model or index change |
| What happens on failure | a conversation | the deploy stops |
| Who owns the threshold | unclear | a named team, written down |
| Where the result lives | a screenshot in chat | a tracked metric with history |
The method - which metrics, which judges, how to calibrate them - is learn-model-evaluation-with-phoebe. What this session insists on is the shape: a gate on the path, with a name on it.
Budget Daybreak's support answer
★ 22 min · everyone buildsOne feature: a customer asks "where is my order" and gets an answer. You are going to give that feature four budgets and one fallback, and then find out what an unbounded loop does to the bill.
Latency. p95 under 1,200 ms end to end, of which the model gets 800 ms. Past the timeout you serve the deterministic fallback, which is the top three help articles and the order status from the database.
Cost. A token ceiling on input and output. Cost per answer inside a band you can multiply by volume. If you cannot write "$X per thousand answers" you do not have a budget, you have a bill arriving later.
Freshness. The index refreshes nightly and appears in the session 4 graph as a node, with an owner and a row-count band. A refresh that silently produces 40 percent fewer chunks should stop, loudly.
Failure. Provider degraded: circuit opens, fallback serves, someone is paged. Index stale: answers degrade quietly, which is why it needs a freshness check rather than an error handler.
Gate. An eval set that runs on every prompt, model or index change, with a threshold, an owner and a deploy that stops. Not a notebook run before launch and never again.
Write the latency budget end to end. Not just the model call - retrieval, model, guardrail, render. Then decide what the user sees when it is exceeded. A design without a fallback has chosen "the page hangs" as its fallback.
Do the Little's Law check. Your request rate times the degraded response time. Compare against your connection pool or worker count. This is the failure that presents as an outage in a component that is still technically working.
Now the cost meter below. It starts at $40 per thousand answers. Add the things that make answers better and watch what each one costs. Then find the two that make it cheaper.
Switch on the unbounded loop last. "Let the agent decide when it is done, up to 20 tool calls." It more than doubles the bill on its own, and the reason it keeps getting shipped is that it demos beautifully on the five questions somebody tried.
Put the index in your session 4 graph. As a node, with an owner, a refresh schedule and a row-count band. This one move converts a silent failure class into a loud one, and costs nothing.
Write the gate. One sentence: "On any change to the prompt, the model version or the index, the eval set runs and the deploy stops below X, and Y team owns X." If you cannot name Y, the gate does not exist yet.
If you take one thing from tonight. Give the model component the same four things every other component gets - a budget, a boundary, an owner and a failure plan. There is nothing about it that earns an exemption, and the exemption is where every one of these projects goes wrong.
The finding your leadership is currently half-quoting
4 min liveDORA's 2025 State of DevOps report is the largest recent piece of evidence on what AI adoption does to a software organisation, and it says two things at once. Almost everyone quotes one of them.
LiveThroughput up, stability down, in the same dataset4 min▶
- Throughput went up, substantially. Epics completed per developer up 66.2 percent, with a positive relationship between AI adoption and both software delivery throughput and product performance.
- Stability went down. Higher AI adoption is associated with higher software delivery instability. The report's own reading is that AI increases the rate of change faster than review and deployment infrastructure can absorb it.
- The value is attributed to the surroundings. The report's central argument is that AI amplifies whatever practices you already have. Strong testing, mature version control and fast feedback turn the extra volume into delivery. Their absence turns the extra volume into instability.
This is the same conclusion as sessions 2 and 5, arriving from a completely different direction. More change flowing through the same boundaries produces more failures at those boundaries. If your change failure rate is already uncomfortable, adding a throughput multiplier makes it worse before it makes anything better - and the fix is the boundary, not the model.
The report also replaced its long-standing four performance tiers with seven team profiles, on the argument that a single ranking was hiding too much about culture and constraint. Worth knowing before someone quotes "elite" at you.
Before session 8 - the capstone
35 min- Find one model call in your organisation and ask what its budget is. Latency, cost, timeout, fallback. If any of the four is missing, you have found the session's point in the wild.
- Find a derived index or embedding store and ask when it last refreshed. Then ask who would know if it stopped. These two questions take four minutes and are frequently uncomfortable.
- Do the Little's Law arithmetic on one AI-backed request path. Rate times degraded latency against pool size.
- Write the gate sentence. One line, with a team name in it.
- Bring your design doc, complete, to session 8. All eight blocks. Session 8 is a review, and the thing being reviewed is yours.
Three questions
Clear all three and session 7 gets stamped in your passport.
1 · Why does a stale retrieval index belong in the same category as session 5's most expensive failures?
An index is a derived asset built on a schedule, exactly like every node in the session 4 graph, but it is usually not in the graph, has no freshness SLA and no row-count band. Adding it as a node with an owner converts a silent failure into a loud one and costs nothing.
2 · At 5 requests a second, the provider degrades from 800 ms to 12 seconds. Your connection pool is 20. What have you got?
L = λW. 5 x 12 = 60. The dependency never returned an error and never went down, and your service is unavailable. This is why a timeout with a deterministic fallback is a design requirement rather than a nicety.
3 · DORA 2025 found AI adoption associated with higher throughput and higher delivery instability. What is the design conclusion?
The report's own framing is amplification: AI multiplies whatever practices exist. Strong testing, mature version control and fast feedback convert the extra volume into delivery; their absence converts it into instability. That is sessions 2 and 5 arriving from a different direction.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map. Certificates, videos and assessments stay with the official providers.
Session 7 cheat sheet · pin this
Everything is decided. Session 8 is the part nobody teaches: writing it down so it survives, and defending it in a room. ADRs in four parts, one-way and two-way doors, the four questions a good reviewer asks - and a capstone where you take apart a deliberately broken version of Daybreak's architecture and produce the decision records that fix it.