Get your teammates to read this

Everything in the previous five sessions is a decision rule, and a decision rule only works if the people making decisions know it. If the ranking rule lives with one leader, then every ranking has to pass through that leader, which is a queue. If the dev set rule lives with one engineer, a new joiner will build a convenient dev set in their first month and nobody will catch it for a quarter.

The failure is not that people disagree. It is that they never had the same information to disagree about. A sponsor who has seen the sheet and still wants their project funded is having a useful argument. A sponsor who has never seen it is having a different conversation entirely, and no amount of authority makes that conversation productive.

• The engineer who joins next month. Needs the dev set sentence and the metric, because those two decide whether their first experiment is measured against the right thing.
• The sponsor whose project came fifth. Needs the sorted sheet and the start week beside their row, so that a no reads as a queue position rather than a verdict on them.
• Whoever sells the product. Needs the current number and the expected trajectory, phrased so it cannot be repeated to a customer as a guarantee.
• The person who funds the team. Needs the diagnosis and the one funded project, because that is the pairing that says the money followed the evidence.
• You, in three months. Needs the date, because the only way to notice that a page has gone stale is to see when it was written.

The book is 58 chapters and each one is short enough to read standing up. That is a deliberate design, and the reason it gives is practical: a colleague who needs one specific idea can be handed exactly the pages carrying it, rather than a book with a recommendation to start at the beginning. A one-page idea gets read. A fifty-page argument gets acknowledged.

The consequence for a leader is that the unit of communication is not the document, it is the page somebody can be handed. If the team needs to agree on how the dev set is sized, the artefact is the paragraph about sizing, not a data strategy. If a sponsor needs to understand why their project is fifth, the artefact is the sorted sheet with their row in it, not an invitation to a review.

• Short means specific. A page can only hold one decision, which forces the decision to be named.
• Short means datable. A page is cheap enough to rewrite when a number moves, and a long document never gets rewritten, only appended to.
• Short means arguable. Somebody who read the whole thing can push back on the whole thing, and the pushback lands on the numbers rather than on the fact that they did not read it.

One model, one page, five sections, in this order. The order matters because each section is only meaningful once the one above it is fixed.

• The dev set sentence. What the model is measured on, in one sentence, including where the examples came from and how many there are. Lanternfish: 1,000 frames sampled from live camera traffic, night and day in the proportions the cameras produce, of which 120 are currently wrong.
• The single metric, with its satisficing bars. One number the team optimises, and the constraints that only have to be cleared rather than maximised. Accuracy is the number; runtime under 100 milliseconds per frame is a bar. Session 2 chose these; the memo records them so nobody re-litigates them mid-sprint.
• The sorted spreadsheet, with points per week. Every candidate project with its ceiling, its cost in weeks, and the division of the two, sorted by the third column. Five rows is usually the whole quarter.
• The three-number bias and variance diagnosis. Training error, dev error, human-level error, and the verdict they produce. Lanternfish: 3, 12 and 2, which is 2 points unavoidable, 1 point of avoidable bias and 9 points of variance, so the work goes to variance.
• The one project funded this sprint, and the four that are not. The funded one with its cost. The other four listed with their points per week and the week each would start under the current order.

Nothing else goes on the page. Not the architecture, not the infrastructure plan, not the hiring case. Each of those is a different page for a different reader, and putting them here is what turns a memo back into a document.

A memo listing four unfunded projects is walking into a room with four disappointed sponsors in it. What survives that room is not the ranking. It is the evidence that the ranking costs somebody something real, and that the somebody is the customer rather than the sponsor.

• In the sheet's order (re-label, motion filter, denoise, occlusion, specialist) the Blackbox error goes 11.2 at the start, 10.2 after week 1, 9.0 by week 3, 6.2 percent by week 6, 5.6 at week 10, and 3.6 at week 18.
• In the order the team argued for, with the specialist model first, the Blackbox error is still 11.2 percent at week 6, 9.2 at week 8, 8.6 at 12, 5.8 at 15, 4.6 at 17, and 3.6 at week 18.
• Both orders finish in the same place. That is what makes the argument winnable. Nobody's project is being called worthless, and nobody has to be told their judgement was poor. The difference is eight weeks that the field spends above 11 percent error, for the same destination.

Put those three lines in the memo verbatim. They are the only part of the page that answers the question a sponsor is actually asking, which is not "is my project good" but "why not mine first".

Once the page exists, most conversations become a pointer to one of its sections rather than a meeting. A new engineer gets sections one and two before writing any code. A sponsor gets section five with their row highlighted. A customer-facing colleague gets the current number and the trajectory, and explicitly not the sheet, because a queue of internal projects is not a thing to share with a customer.

The reason this works is that the sections were written to stand alone. If section five cannot be understood without reading section three, it is too compressed, and the fix is a clause rather than a longer memo. Read each section on its own before sending it and ask whether a person who saw only that section would take a correct action.

Hamel Husain's account of evaluating a language-model product, in "Your AI Product Needs Evals", sorts the work into three levels that rise in cost and in trustworthiness.

• Level 1: unit tests and assertions. Cheap checks that run on every change. Did the output parse, did it include the required field, did it avoid the phrase it must never use. These are the fast half of the loop, and they are the ones teams skip because they feel too simple to matter.
• Level 2: human and model eval. A person reads outputs, and once the criteria are stable a model helps apply them at scale. This is where most of the learning happens, and where the previous five sessions' habits reappear almost unchanged.
• Level 3: A/B testing. Real users, real traffic, the slowest and most expensive signal, and the only one that settles whether the change was worth anything commercially.

The shape is familiar: a cheap fast loop, a considered middle loop, and an expensive final reading. What changes between a classifier and a language model is the instrument at each level, not the structure of the ladder.

Underneath all three levels is one act, and Husain is direct about it: "You must remove all friction from the process of looking at data". Every session in this track has been a variation on that sentence. Look at 100 mistakes. Split the dev set so half of it stays honest. Walk one error through the boxes by hand. The instrument changes and the act does not.

His stopping rule is the same shape as the error-analysis spreadsheet's: "keep reading logs until you feel like you aren't learning anything new". That is the language-model version of stopping the tally when the categories stop changing. It is a judgement, not a threshold, and it is a judgement that only the person doing the reading can make.

• The error sheet becomes a log sheet. Same columns, same principle: a row per failure, a category per column, a share at the bottom, and one row can tick several columns.
• The ceiling still prices a project. If 20 percent of bad outputs come from retrieval returning the wrong document, that share is the most a retrieval project could fix.
• Error analysis by parts still applies, and applies harder, because a language-model product is almost always a pipeline: retrieve, then prompt, then parse. Hand one stage a perfect input and see whether the failure survives.
• The one funded project and the four not still fits on a page. The memo does not change shape at all.

The last section of the memo is where somebody senior asks for a number and a date. What can honestly be promised is not a target error rate. Nobody knows in advance which idea will work on a real dataset, which is the premise the whole track is built on, and a promised accuracy figure is a guess wearing a commitment's clothes.

What can be promised, and kept:

• A dev set drawn from the traffic the product will be judged on, with a stated size, by a stated date.
• A single metric with its satisficing bars written down, so that "better" means one thing across the team.
• A sorted sheet of candidate projects with ceilings and costs, rebuilt each time the errors are re-counted.
• A sprint cadence that says how often the loop closes and the numbers are re-read.

Those four are entirely within the team's control, which is what makes them promises rather than forecasts. A leader who commits to them and reports them on time buys far more credibility than one who commits to 95 percent and arrives at 94.

One tool from session 4 breaks on the move and it should be retired rather than approximated. The bias and variance decomposition needs a training error to compare against: avoidable bias is training error minus the optimal error rate, and variance is dev error minus training error. With a language model called through an API and prompted rather than trained, there is no training error belonging to your team. Two of the three numbers are missing.

• What you keep is the idea that the total error splits into parts with different remedies, and that the split decides which project to fund.
• What you need instead of training error is a different yardstick for the floor: how well the strongest available configuration does on the same examples, or how well a careful person does with the same context.
• What to avoid is quoting an avoidable-bias number computed from a number you do not have. A diagnosis with an invented input is worse than no diagnosis, because it is persuasive.

Say this out loud when the memo covers a language-model product. The section is shorter and it says which yardstick was used instead. An honest gap on the page is much cheaper than a confident number that nobody can reproduce.

• Golden sets, judges and regression suites belong to the AI evals course. It owns how a language-model eval is actually built and run, how a model-as-judge is validated, and how an eval culture is kept alive once the novelty wears off.
• Metric definitions belong to the model-evaluation course. What precision is, where a threshold goes, whether a score is calibrated, what a cost matrix does to a decision boundary. Every session so far has taken a metric as given and never once defined one.
• A/B testing itself belongs to the experimentation course. Husain's third level is named here as the top of the ladder and is not taught.

The reason to draw these lines explicitly is the same reason the memo is one page. A track that tried to own all three would be a track nobody finishes, and the decisions it exists to teach would be buried in the middle of it.