Where you are in the leader track
Session a3 decoded the quality rows of an LLM report and taught you to distrust unmeasured judges. Tonight covers the rows that usually are not on the report at all: latency and cost. These are not infrastructure trivia - they are the metrics users vote on with their feet, and the ones that quietly decide whether an AI initiative's economics work. The session ends with the move that ties the whole leader track together: every metric on one sheet, one number being optimized, and everything else - including speed and price - holding a declared floor or ceiling.
Latency, decoded for decision-makers 15 min live
An LLM answer is not one wait - it is two experiences in sequence. First the silence: from the moment the user submits to the moment the first token appears. That is TTFT, time to first token - the "is it thinking?" moment, and the number users judge hardest. Then the stream: tokens arriving one by one, whose smoothness is set by inter-token latency (ITL) - the average gap between tokens. Add them up and you get total time. Three numbers, three different experiences, three different engineering levers.
| Metric | Plain-English question | What the user feels | The trap |
|---|---|---|---|
| TTFT (time to first token) | "How long is the silence between submit and the first word?" | The "is it thinking?" moment - the wait users judge hardest and abandon over. | Averages hide the tail. Always ask at p95 - one user in twenty lives in the tail. |
| ITL (inter-token latency) | "Once it starts, how smoothly do the words arrive?" | The stream's rhythm - choppy streaming reads as a struggling product even when total time is fine. | System-wide tokens per second is NOT this number - under batching the two diverge. |
| Total time (e2e) | "Submit to done, how long altogether?" | The full wait - what a stopwatch and an impatient user both measure. | One blended number hiding two experiences. Decompose it with the identity below. |
| Cost per query | "What does one answer cost at OUR volume?" | Nothing - which is why it slips off the eval sheet until the invoice arrives. | Rate cards are not budgets. Pennies per query is a budget line at a million queries a month. |
LiveThe identity, and the batching tradeoff6 min▶
The diagram's box is worth memorizing as a leader's tool: e2e = TTFT + (tokens - 1) x ITL. End-to-end time is the first-token wait plus one inter-token gap for every remaining token. When a vendor quotes you a TTFT, a tokens-per-second figure, and a total time, run the one-line check - numbers that do not obey the identity mean someone is quoting different scenarios in the same table.
Then the tradeoff every deployment hides: batching. Serving many users' requests together raises SYSTEM throughput - more total tokens per second out of the same GPUs, so a cheaper service. But it stretches each user's ITL - every individual stream gets choppier. System throughput up means each user's smoothness down:
- You cannot maximize both. This is a curve, and someone chooses the operating point - exactly like the classification threshold in a2.
- The infra team's dashboard (system tokens per second) and the user's experience (per-user tokens per second) diverge under batching. Ask which one a report shows.
- The choice is a business decision wearing an infrastructure costume: cost per query versus experience per user. It should be made by someone who knows it is being made.
LiveCost per query - the other axis4 min▶
Every answer has a price: tokens in plus tokens out, times the model's rates - or GPU time, if you host. The leader habits that follow:
- Always evaluate at YOUR volume. A metric that looks like pennies per query becomes a budget line at a million queries a month. Quality reports that omit cost per query are comparing free things that are not free.
- Quality gains have prices. The common upgrade decision - a bigger model scores 2% higher on the quality metric at 5x the cost per query - is usually a loss once you price it. Not always: a 2% win on a high-stakes task can be worth 50x. The point is that the arithmetic must be done, and it usually is not.
- Speed and cost trade too. Faster serving often means more hardware or smaller models - the three-way tension (quality, latency, cost) is the real decision space, and any report showing only one axis is hiding the tradeoffs.
Self-studyWhere these numbers come from3 min read▶
The definitions behind Part 1, as the benchmarking docs (NVIDIA's NIM metrics guide) state them - useful when a vendor's glossary gets creative:
- TTFT covers everything before the first token: queueing, prefill (processing your prompt), and the network. Long prompts raise it - which is why a RAG system that stuffs huge contexts pays for that choice in first-word silence.
- ITL is the average gap between tokens after the first: (e2e minus TTFT) divided by (tokens minus 1) - the same identity from the diagram, rearranged.
- System tokens per second vs per-user tokens per second are different quantities that diverge under batching - the vendor's throughput slide usually shows the first, your user feels the second.
You will never compute these yourself - the practitioner track (b8) does. You only need to recognize which one a slide is showing, and ask for the one it is not.
The satisficing architecture 18 min live
Three axes, all trading against each other - so which one wins? The answer is an architecture, and two very different sources agree on it. Andrew Ng's doctrine: pick ONE optimizing metric, and make everything else a satisficing metric - a threshold to satisfy, not a score to maximize. Optimize quality SUBJECT TO latency at most X and cost at most Y. Anthropic's guidance lands in the same place: its success-criteria list includes latency AND price as first-class categories alongside quality, and its worked example sets a quality floor AND a toxicity ceiling AND a latency ceiling - one sheet, several constraints, one thing being maximized.
LiveFloors, ceilings, and the one number you optimize8 min▶
Why one optimizing metric? Because teams iterate fast only when every experiment gets an unambiguous verdict - better or worse. Two "optimize both" metrics produce arguments; one optimizing metric plus hard constraints produces decisions. The architecture in practice:
- Optimize: the metric closest to the business outcome - answer quality, resolution rate, conversion lift. One number.
- Floors: minimums that must hold - quality never below the current baseline, completeness above an agreed bar.
- Ceilings: maximums that must hold - latency (TTFT and total, at p95), cost per query, harmful-output rate.
The leader's move that makes this real: every AI initiative declares its latency and cost budgets ON THE EVAL SHEET - the same document that carries the quality metrics - not in an infrastructure ticket nobody reviews. A constraint that lives outside the eval sheet does not constrain anything: the team optimizes what the sheet measures, and ships whatever the sheet ignores.
The assistant users abandoned. A support assistant's quality scores climbed for three straight months - the team tuned prompts, upgraded models, and the judge scores duly rose. Usage fell the whole time. The upgraded models took around 8 seconds to start answering, users clicked away, and churn did the judging that the eval sheet did not: nobody had made latency a success criterion, so nobody was watching it. The postmortem finding was one sentence - the eval sheet had five quality rows and zero speed rows - and the fix was one line on a document, declared before the next initiative started.
The questions for any LLM vendor or team self-study
Three questions, in the order to ask them. Each has a follow-up that separates teams who measured from teams who assumed.
Self-studyThe three questions, with their follow-ups5 min read▶
- "What is TTFT at p95 - not the average?" Averages hide the tail, and the tail is where users churn: a 1-second average with an 8-second p95 means one user in twenty gets the abandoned-assistant experience. A team that answers with an average has not looked at the tail.
- "What is cost per query at OUR volume?" Demo pricing and production pricing are different worlds - long contexts, retries, and peak-hour batching all move the real number. Ask for the arithmetic at your monthly query count, not the per-token rate card.
- "Which quality metric degrades when you cut latency - and by how much?" Speed is bought with something: a smaller model, tighter retrieval, shorter answers. A team that says "nothing degrades" has not measured the tradeoff; a team that names the metric and the size of the hit has an eval sheet you can trust.
Notice what the three questions share: none asks for a better number - each asks whether the number was measured where it matters (the tail, your volume, the tradeoff). That is the leader track's whole method applied to speed and cost.
Try it yourself - this week ◐ 20 min total
- Open the eval sheet (or the closest thing to one) for your most visible AI initiative. Count the quality rows, then the speed rows, then the cost rows. If the last two are zero, you have found the abandoned-assistant risk in your own portfolio.
- Ask the owning team for TTFT at p95 on the main user-facing flow. Time how long the answer takes to arrive - that delay is itself a finding.
- Run the 2%-at-5x arithmetic on the next model-upgrade proposal you see: quality delta, cost multiple, monthly volume. Two lines of multiplication, one clear decision.
- Draft the satisficing sheet for one initiative: one optimizing metric, one quality floor, one latency ceiling (p95), one cost ceiling. It fits on an index card - the value is that it now exists.
Official sources covered
The leader track distills the canonical sources into decisions - benchmark mechanics and measurement details stay with the linked originals, and the practitioner track works them hands-on in b8. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Users complain the assistant "takes forever to start answering," but the vendor's total-time numbers look fine. Which metric do you ask for?
The complaint names the silence, not the stream - that is TTFT's phase of the timeline. Total time blends both phases together, which is exactly why the identity e2e = TTFT + (tokens - 1) x ITL matters: it lets you take one blended number apart into the two experiences users actually feel.
2 · The infra team proposes heavier batching: "system throughput doubles, same hardware." What is the catch?
Batching trades the individual stream's smoothness for total system output - you cannot maximize both. It is a business choice (cost per query vs experience per user) wearing an infrastructure costume, and the leader's job is to make sure it is chosen consciously, not defaulted.
3 · In the satisficing architecture, where does latency belong?
One optimizing metric, everything else a floor or ceiling - Ng's doctrine, and the shape of Anthropic's own worked example (quality floor, toxicity ceiling, latency ceiling). Optimizing latency itself ships a fast wrong answer; leaving it in an infra ticket ships the abandoned assistant. The ceiling on the eval sheet is the placement that constrains.