Two gates that nobody argues about until launch week
Quality gets all the meeting time. Then the bot ships, the p99 turns out to be eleven seconds, finance sees the first full month of inference spend, and both of those become somebody's emergency. Operational metrics are unglamorous, cheap to instrument and almost always measured too late - and unlike quality metrics, they have the decency to be objective.
Users feel the first token, not the last 6 min live
When an answer streams, the perceived wait ends at the first token and the reading begins. Total response time is still real - it caps how fast a conversation can move, and it is the whole story for anything non-streaming - but it is not the number the user is judging you on.
LiveWhy TTFT and TPOT need separate gates4 min▶
The two halves of the timeline are produced by different work, so they fail independently and they get fixed differently.
- TTFT is queueing plus reading the whole prompt before a single token comes back. It grows with prompt length, with retrieval that runs before the model, with any pre-flight guardrail call, and with load. Every one of those is a lever you added in b2 for quality reasons.
- Inter-token latency (TPOT) is the per-token cost of generation. It is mostly a property of the model and the serving stack, and it barely cares how long your prompt was. It sets the ceiling on how fast the answer can finish.
A useful starting SLO if you have nothing today: p95 TTFT under one second and p95 inter-token latency under 50 ms. That is deliberately loose - it is a floor to detect regressions against, not a target to be proud of. Well-tuned interactive systems land in the 200 to 500 ms TTFT range, and p95 TTFT under 500 ms is a common target for anything that feels like a chat.
Conversational voice is a different regime entirely, because there is no partial output to look at while you wait. The working budget there is roughly 800 ms of round-trip for the whole loop, which has to cover speech recognition, the model and speech synthesis. If Beacon ever puts its support bot on a phone line, none of today's numbers survive the move.
Self-studyWhat to instrument, and the units that make it comparable3 min read▶
Record five fields per request and you can compute everything on this page later: ttft_ms, total_ms, output_tokens, input_tokens, and an attempt number so retries are visible rather than folded silently into a single latency figure.
Inter-token latency is then derived, not measured: (total_ms - ttft_ms) / max(1, output_tokens - 1). Deriving it rather than logging per-token timestamps keeps the instrumentation cheap and gives you a number that is comparable across models with different tokenizers, as long as you say which tokenizer produced the count.
One trap worth naming now: log one row per request, not one per user session. A percentile computed over sessions weights a chatty user the same as a quiet one and quietly flatters your tail. Session a4 makes the same point about golden sets, where it shows up as the cluster standard error gotcha - and it can widen a real confidence interval by up to about three times.
The average is the summary that flatters you most 7 min live
Latency distributions are not symmetric. They have a long right tail made of cache misses, cold starts, retries, long prompts and unlucky queueing - and the mean sits well to the left of that tail, in a place where nothing bad is visible.
lat = sorted(ttft_ms) # one row per request, not per session
def pct(q): return lat[max(0, ceil(q * len(lat)) - 1)]
print(mean(lat), pct(.50), pct(.95), pct(.99)) # report all four| Metric | Starting SLO | Well-tuned target | Notes |
|---|---|---|---|
| TTFT, streamed chat | p95 under 1 s | p95 under 500 ms; 200-500 ms achievable | The number users actually feel. Grows with prompt length and pre-flight guardrails |
| Inter-token latency | p95 under 50 ms | Model and serving stack dependent | Sets the ceiling on how fast an answer can finish. Derive it, do not log per token |
| Full response time | Report, do not gate | Falls when the answer gets shorter | Dominated by output length, so it is really a verbosity metric in disguise |
| Voice round trip | ~800 ms end to end for conversational voice | A different regime. No partial output to look at, so the whole loop is the wait | |
Cost per task, and the shift to cost per outcome 6 min live
A cost number is meaningless without a denominator and a comparison. The denominator is the unit somebody actually cares about - a resolved conversation, a completed analysis, a query answered - and the comparison is what that unit costs when a human does it.
Label the sourcing, because these numbers do a lot of work in a business case. The $13.50 median per US assisted-channel contact is Gartner-attributed and reaches us through third parties, and the same is true of the ~$1.50 Sierra figure. Intercom's $0.99 is different in kind: it is a published price on Fin's own billing documentation, which also spells out exactly what counts as a resolution. Treat the first two as order-of-magnitude anchors for an ROI conversation and never as a line item, and be the person in the meeting who says which is which.
cost_turn = (in_tok * price_in + out_tok * price_out) / 1e6
cost_conv = cost_turn * turns * retry_factor # retries are not free
cost_res = cost_conv / resolution_rate # the billable unitThe third line is the one that surprises people. If a conversation costs 30 cents and the bot resolves 50% of them, the cost per resolution is 60 cents, not 30 - you paid for the failures too, and then a human handled them anyway. Improving the resolution rate is therefore a cost lever as much as a quality lever, and halving your unresolved traffic does more for cost per outcome than any amount of prompt trimming.
LiveRetry amplification, and why reliability is a budget line4 min▶
Retries are the most common way a cost model goes wrong, because they are usually invisible in the instrumentation. If your agent retries a failed tool call up to three times, your worst-case cost for that task is three times the happy path, and your average sits somewhere between depending on a failure rate you probably have not measured.
The metric that governs this is the one b8 builds properly. pass@k asks whether at least one of k attempts succeeds, and it is how retries buy you reliability - you pay k times for a better chance of success. pass^k asks whether all k attempts succeed, and it is the metric that matches what production actually demands, because the same user asks the same thing again on Tuesday. They pull in opposite directions, and the arithmetic is unforgiving: a task the bot passes 75% of the time succeeds three times running only about 42% of the time.
What this means for today's gate: your cost number must include the retry factor, and your retry factor comes from your reliability, not from your config file. A budget built on the happy path is a budget that assumes a bot you do not have.
Self-studyR-VES: the one published metric that prices correctness by runtime3 min read▶
Almost every benchmark scores correctness and ignores what the correct answer cost to produce. The text-to-SQL world is the exception: BIRD's official R-VES weights execution accuracy by the runtime of the generated query relative to the gold query. A correct query that takes forty seconds where the reference takes two does not score the same as a correct query that takes two.
That is worth borrowing even if you never touch SQL, because it encodes the honest position on this whole page: a right answer that is too slow or too expensive is a partially wrong answer. Most teams hold quality and efficiency in two separate documents owned by two separate people, and R-VES is the field's clearest example of putting them in one number. Beacon's Analyst Bot inherits it directly in b6; for the other two bots you get the same effect the cruder way, by running a quality gate and a cost gate side by side and refusing to trade them.
Build Beacon's operational table ★ 10 min · everyone builds
Same idea as the golden set in b1: one sheet you will keep. Rows are your bots or your traffic slices, columns are the numbers you will gate on in b9. Fill it with real logs if you have them and with honest blanks if you do not - the blanks are the useful part.
Make eight columns: slice · requests · TTFT p50 · TTFT p95 · TTFT p99 · inter-token p95 · cost per task · cost per outcome. One row per bot to start, then split the busiest bot by the categories you defined in b1.
Pull one hour of production logs, one row per request. If your logs only have averages, write "average only" in the cell and stop - you have found the most valuable finding of this session and it costs a day of instrumentation to fix.
Compute the three TTFT percentiles. Then put the average next to them, in a column you will delete before anyone else sees the sheet. Look at the gap once. On most real traffic the average sits closer to the p50 than to the p95, which is why it is such a comfortable number to report.
For cost per task, use the three-line formula above and be honest about retry_factor. If you do not know it, put 1.0 and write "unmeasured" beside it. An unmeasured retry factor is a known unknown in your budget rather than a hidden one.
Now add the row that makes the table an argument rather than a report: the human baseline. For Beacon's support bot that is the assisted-channel contact cost. Your cost per outcome is only interesting relative to what the outcome costs today.
Finally, write the SLO you intend to gate on into the header of each column, before you look at your numbers. Same discipline as the threshold in b1 - nobody ever failed a latency SLO they set after seeing the p95.
Price every quality lever from b2 ★ 8 min · everyone builds
This is the exercise that makes b9 make sense. Take the levers you switched on last session for quality reasons, and put a millisecond figure and a cent figure next to each one. Nothing here is free, and most of it is paid for in TTFT.
| Quality lever from b2 | What it costs in latency | What it costs in money |
|---|---|---|
| Retrieve more context for grounding | TTFT only - a longer prompt to read before the first token | Input tokens on every single call, including the easy ones |
| Strict claim-level faithfulness checking | Offline in eval; online it is a second model call on the critical path | Roughly a second inference per checked response |
| Pre-flight injection or policy guardrail | Pure TTFT, paid on 100% of traffic to catch a small slice | Small per call, large in aggregate because it never skips |
| Citations attached to every claim | Total response time - more output tokens to stream | Output tokens, the expensive side of the price sheet |
| Retries for reliability | Only on the retried requests, which is exactly where p99 lives | Multiplies the whole task cost by the retry factor |
| A larger model for the hard slice | Both TTFT and inter-token latency get worse together | The single biggest line, and the easiest to route around |
Copy that table and fill the two right-hand columns with your own measured numbers, by running your eval once with each lever off and once with it on. Ten minutes of work per lever, and the result is the only version of this table anyone should believe.
Rank the levers by cost per point of quality gained. The ordering is almost never what the team expected, and there is usually one lever that is buying nothing measurable and paying for it on every request.
Find the levers that only cost you on part of the traffic. Routing the easy 70% of turns to a smaller model, or running the strict grounding check only on the policy slice, buys most of the quality for a fraction of the bill. This is the fix behind almost every red cost gate in b9.
Now switch every quality lever on at once and re-measure p95 TTFT and cost per outcome. Watch both gates go red together. That is the coupling this session exists to teach, and in b9 you will see the same thing happen on the scorecard.
Try it yourself - this week ◐ 45-60 min total
- Instrument the five fields if you do not already have them: ttft_ms, total_ms, input_tokens, output_tokens and attempt number. One row per request. Everything else on this page is derivable from those five, and none of it is recoverable without them.
- Compute p50, p95 and p99 TTFT for your busiest bot, then compute them again sliced by your golden-set categories. Find the slice with the worst p95 and check whether it is also the slice with the worst pass rate. It usually is, and that overlap is worth a slide of its own.
- Measure your retry factor for one week. Count attempts, not requests. Then recompute your cost per outcome with the real number and see whether your budget still holds.
- Take your most verbose bot and cut the average answer in half through prompting alone, then re-measure. Total response time falls, output token cost falls, and if you scored verbosity in b2 that number improves too - one change, three gates. It is the only lever on this page that is free.
- Write down your latency SLO and your cost-per-outcome budget, today, before b9. Seal them alongside the quality threshold you sealed in b1. In session b9 you find out how many of your three sealed numbers you would have quietly moved.
Official sources covered
Every figure on this page traces to a primary source or is labelled as secondary where it reaches us through a third party. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your dashboard reports an average TTFT of 548 ms against a 1 second SLO, so the gate is green. Why is that not good enough to launch on?
The mean is dragged only slightly by a thin tail, so it sits close to the median and reports the experience of your typical user. Your unhappiest users are the tail, and they are the ones who churn, escalate and complain. Report p50, p95 and p99 and gate on p95 - and if you slice by category you will usually find the slow slice is also the slice your bot answers worst.
2 · You add a retrieval step that pulls twice as much context, purely to improve grounding. Which operational number moves most?
The two halves of the timeline fail for different reasons, which is why they need separate gates. Prompt length lands almost entirely on time-to-first-token, while inter-token latency is mostly a property of the model and the serving stack. That makes this a particularly expensive quality lever in UX terms: you pay for it in the one phase where the user has nothing to look at, and you pay it on every request rather than only on the hard ones.
3 · A conversation costs 30 cents to run and the bot resolves half of them. What is your cost per resolution, and why does it matter?
Cost per outcome divides total spend by successful outcomes, so every failure is paid for twice: once in inference and once by the human who picks it up. This is exactly why the pricing model is shifting from per-interaction to per-outcome - Fin bills $0.99 per resolved conversation and publishes a detailed definition of what resolved means, including reversing a resolution if the customer comes back. If you are going to be billed on an outcome, you had better be measuring the same outcome.