The serving math 8 min live
The NVIDIA benchmarking definitions, exactly - these are the industry's shared vocabulary, so learn them as written. TTFT (time to first token) = submit -> first token; it contains the queue wait, the prefill of your prompt, and the network hop, and it owns the "is it thinking?" UX moment. ITL (inter-token latency, also called TPOT) = (e2e - TTFT) / (output_tokens - 1) - the decode smoothness, how fluid the streaming feels. And the identity that ties them: e2e = TTFT + (tokens - 1) x ITL. Sanity-check any vendor claim with it - the three numbers must reconcile, and when they do not, that is your first question.
LiveSystem TPS vs per-user TPS: the batching tradeoff4 min▶
Two throughput numbers that vendors love to blur: system TPS = total output tokens / wall time, across ALL concurrent requests - the infra bill's metric. Per-user TPS - what one person actually experiences in their stream. Under batching they diverge, by design: batching more requests together raises system throughput while degrading each user's ITL.
- Big batches: cheaper per token, choppier for every individual user.
- Small batches: smoother streams, a larger serving bill.
- There is no correct setting - you choose, per product. A batch-overnight summarizer and a live chat assistant should not share a batching config, or a latency budget.
When a vendor quotes one glorious TPS number, ask which one it is, and at what concurrency. The identity from the figure catches the rest.
LivePercentile discipline: budget p95, never averages3 min▶
Latency distributions have long right tails: most requests are fine, and a few are terrible. An average blends the terrible into the fine and reports "fine." Users do not experience your average - the one 9-second answer is what they remember and what they quote in the churn survey.
So the budget is written in percentiles: p95 TTFT <= 800ms means 95% of requests see a first token inside 800ms, and you are watching the slowest 5% explicitly. For anything user-facing, budget p95 or p99; use the average only for capacity planning. One slow tail kills perceived quality - and the average will never tell you it is happening.
Self-studyThe identity as three lines of Python2 min read▶
Worth keeping in a scratch notebook for vendor meetings:
# e2e = TTFT + (tokens - 1) x ITL - reconcile any latency claim
ttft, itl, out_tokens = 0.42, 0.03, 120 # seconds, seconds, tokens
e2e = ttft + (out_tokens - 1) * itl
print(f"implied e2e: {e2e:.2f}s") # 3.99s - does the claimed e2e match?If a datasheet claims sub-second answers with those components, the numbers do not reconcile - someone is quoting TTFT as e2e, or averaging away the tail. Either way, you now have the question.
Cost per query, and the ROI arithmetic 8 min live
The cost of one query is not mysterious: tokens in x input price + tokens out x output price + retries. What teams skip is multiplying it by volume and putting it next to the quality number - which is the only place it means anything. The levers, in rough order of cheapness: model tier per step (not every step needs the big model), prompt caching (stop paying full price for the same system prompt), output-length discipline (you pay per output token; rambling is a line item), and max-turn caps (an agent that loops is a bill that loops).
LiveBuild-along: budget the Lumen assistant14 min▶
The Lumen support assistant, choosing between two models. Put quality and cost on the same sheet and let the arithmetic speak:
| Model A | Model B | |
|---|---|---|
| Quality (judge score, b7-calibrated) | 0.86 | 0.88 |
| Cost per query | $0.004 | $0.021 |
| Volume | 40,000 queries/mo | 40,000 queries/mo |
| Monthly cost | $160 | $840 |
Model B buys 2 quality points for $680/mo more - roughly 5x the cost. Is 2 points worth $680 a month? Maybe! If those points sit exactly where escalations happen, possibly yes. The lesson is not the verdict - it is that this is a product decision, and it belongs on the sheet, made by someone who can see both columns. The general rule of thumb: a 2% quality win at 5x cost is usually a loss.
Compute your own version: your assistant's average tokens in/out, your models' prices, your monthly volume. One spreadsheet row per candidate model.
Add the retry rate - a 10% retry rate is a 10% surcharge on everything, and it hides in logs, not in price sheets.
Now apply one lever (caching, shorter outputs, or a cheaper model on the easy step) and recompute. The cheapest quality point is usually the one you did not have to buy back.
Latency and price go inside the eval sheet 6 min live
Anthropic's own define-success guidance lists latency and price among its success-criteria categories - they are not infra footnotes, they are eval criteria with the same standing as quality. The instrument for holding them together is Ng's satisficing-vs-optimizing pattern from b5, now completed for an LLM system: one metric to maximize, everything else a hard constraint.
LiveThe satisficing architecture, completed5 min▶
The Lumen assistant's eval sheet, in one sentence:
maximize task success on the golden set
subject to faithfulness >= 0.9
p95 TTFT <= 800ms
cost <= $0.02/queryThree rules that make it real rather than decorative:
- Every constraint is owner-signed. Support owns the faithfulness floor, product owns the latency budget, finance owns the cost cap. A constraint nobody signed is a suggestion.
- Constraints are pass/fail, not tradeable. A model that scores 0.95 on quality but blows the latency budget does not ship - that is the entire point of writing it down.
- The optimizing metric stays singular. One number to climb keeps iteration fast (Ng's single-number doctrine); the constraints keep the climb from wandering somewhere expensive.
The assistant that got better and lost its users. A team shipped an assistant and watched judge scores rise release after release - richer answers, better citations, longer reasoning. Answers also crept from 2 seconds to 8. Users churned to the old search box. The postmortem's one line: latency was never a success criterion, so nobody was failing - every dashboard was green while the product died. One row on the eval sheet would have caught it in week one.
Self-studyWhere the sheet goes next2 min read▶
You now hold every column of the eval sheet: the classic metrics and their costs (b1-b5), LLM quality metrics (b6), a calibrated grader to compute them (b7), and tonight's latency and price constraints. b9 makes the whole thing continuous - drift, regression evals, the two-hop proxy chain - and b10 assembles the full sheet for Lumen's classic model and a RAG assistant, every number defended.
Try it yourself - this week ◐ 20-30 min total
- Take any LLM feature you can measure and compute its real cost per query from logs: tokens in, tokens out, retries. Multiply by monthly volume. Show someone the number.
- Find one vendor latency claim and reconcile it with the identity: e2e = TTFT + (tokens - 1) x ITL. Write down what is missing from the claim (percentile? concurrency? which TPS?).
- Draft the satisficing sentence for a system you own: one metric to maximize, two or three constraints with numbers - then find the owner for each constraint and get the sign-off. That conversation IS the deliverable.
- Check one latency dashboard you rely on: does it show p95/p99 or an average? If average, you have this week's cheapest fix.
Official sources covered
This session teaches the NVIDIA serving vocabulary as the industry standard, then wires latency and price into the eval sheet using Anthropic's success-criteria categories and Ng's satisficing pattern. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Users complain the assistant "sits there thinking forever" before anything appears, but once text starts it flows fine. Which metric owns that complaint?
The dead air before the first character is TTFT's territory - queue, prefill, network. ITL owns the opposite complaint: text that starts fast but stutters as it streams. Matching the user's complaint to the right metric is what makes the fix findable.
2 · A vendor claims: e2e 1.5s, TTFT 400ms, 100 output tokens, ITL 30ms. What does the identity say?
e2e = TTFT + (tokens - 1) x ITL = 0.4 + 99 x 0.03 = 3.37s - more than double the claimed 1.5s. Maybe the token count is smaller, maybe the ITL is a best case, maybe e2e is a p50 while the rest are means. The identity does not tell you which - it tells you to ask.
3 · Why does the latency budget say "p95 TTFT <= 800ms" instead of "average TTFT <= 800ms"?
Most requests are fine and a few are terrible; the average blends them and reports "fine." A p95 budget watches the slowest 5% explicitly - and one slow tail kills perceived quality faster than a mediocre median ever will. Averages are for capacity planning, percentiles are for users.