learn-ship-playbook-with-phoebe / Session 5 of 6
Learn Ship Playbook with Phoebe · Session 5 of 6

The revenue model decides

Session 4 left you with a list that has passed the data gate: cases the business could actually build with what it holds today. That list is still in no particular order, and the order is where the money is. Tonight is the second gate, and it is simpler than it sounds: the same candidate is worth a different amount at two businesses that make their money differently. A churn score sits near the top at a recurring-membership business and a long way down at one that sells single transactions to people it will never see again. Then the harder half of the evening, which is taking apart the failure-rate statistics people reach for when they want a roadmap approved.

🟠 Advanced Bring your gated list No code required 45 min
0-3 · Welcome 3-16 · Why the same case is worth different amounts 16-38 · Build-along: score fit 38-45 · Q&A
Part 0

Where we start

Two things happen tonight and they pull in opposite directions, which is deliberate. The first is a rule that puts your gated list in order. The second is a demolition of the evidence most people use to argue for a list at all. Holding both at once is the actual skill: you can have a strong argument and still refuse to decorate it with a number you cannot source.

🎫 Clear all three quiz questions to stamp your passport Live = presented in the 45 minutes Self-study = the full depth, after class

★ What you walk out with tonight. One sentence describing how your business actually makes its money, a fit score against it for every candidate that cleared the data gate, the name of the candidate whose score would swing hardest at a different kind of business, and a business case with every unsourced statistic deleted from it.

Part 1 · covers the sequencing rule

The same case, worth two different amounts

6 min live

The rule is one line. Which use cases pay depends on how the business makes its money, so a candidate that belongs at the top of one list belongs a long way down another, with nothing about the technology having changed.

Read this before you use the rule anywhere. The sequencing rule is this playbook's own heuristic and it carries no citation. A deliberate search for published work establishing that a firm's revenue model determines which AI or automation investments pay off found nothing peer-reviewed. There is no paper behind it, no study, no dataset. It is a working rule that has held up in practice, offered as exactly that, and you should hold it more loosely than anything cited on these pages. Session 6 asks you to catch people dressing their own heuristics as findings, so it would be absurd to do it here.

ONE CANDIDATE, SCORED AT TWO BUSINESSES revenue fit, 0 to 3, a judgement not a measurement THE CANDIDATE RECURRING MEMBERSHIP the renewal is the revenue event SINGLE TRANSACTION no repeat relationship at all Churn score per customer flag who is about to stop paying 3 saving one renewal saves every renewal after it 0 there is no next renewal for the score to protect Upsell at the point of sale recommend the next item in the basket 1 raises the basket, but the renewal is what pays here 3 the transaction is the only revenue event there is Fraud screening on payment block the charge that gets reversed 2 the same loss per event at either kind of business 2 the one that barely moves when the model changes Nothing about the technology changed. Nothing about the candidate changed. Two of the three scores moved by three points and the third did not move at all. Fit is a judgement about a business, not a property of a use case. Every score above is a judgement made against one sentence about how the money arrives. Shown so you can disagree with it.
🔍 Click to zoom - the third row is the useful control: a candidate whose score barely moves is one you can sequence without knowing the revenue model at all
LiveWhy the session 3 bench scores would all move at your business4 min▶

The bench in session 3 scored revenue fit against a recurring-membership model, and that choice was shown on screen rather than hidden inside the arithmetic. It is worth being blunt about what follows from it: every single revenue-fit score on that bench would move at a business with a different model. Not some of them. All of them are readings taken against one sentence, and changing the sentence changes the readings.

That is not a weakness in the bench, it is the whole reason the second gate exists. The data gate in session 4 is close to objective: either the business holds the entity in full or it does not, and two reasonable people looking at the same field will usually agree. The revenue gate is a judgement, and it is a judgement about the business rather than about the use case. Which means it is the one gate you cannot outsource, cannot copy from another company's roadmap, and cannot buy from a vendor who does not know how your money arrives.

The practical consequence is small and annoying: you have to write the sentence first. Not the mission, not the strategy deck, the sentence. "We are paid a monthly fee and we keep about 70 percent of members past their first year." "We are paid once, per job, and about one customer in twenty ever comes back." Those two sentences produce different roadmaps out of an identical candidate list, and everybody in the room can check a fit score against them in about four seconds.

Self-studyThe four sentences, and the candidates that swap places between them4 min▶

Paid a recurring fee for continued access. Everything that protects the next renewal is near the top: churn scores, health scores, reasons for cancelling, onboarding friction. Anything that raises the size of a single purchase is a long way down, because a bigger basket does not extend the relationship.

Paid once per transaction, with no repeat relationship. Exactly reversed. Basket size, conversion on the page, pricing at the moment of sale, fraud at the moment of sale. A churn model has nothing to hold on to, which does not stop people asking for one, and that is the subject of the fourth build-along pass.

Paid on volume you do not control, such as usage or claims. The money moves with throughput and with loss, so forecasting demand and detecting the expensive tail matter most. Retention often matters less than it feels like it should, because the customer is locked in by something other than satisfaction.

Paid on an asset being available. Utilities, rentals, fleets, equipment. Availability is revenue, so failure prediction and maintenance scheduling go to the top of a list where they would be mid-table almost anywhere else, and the same equipment-sensor case that is an expensive curiosity at a gym chain is the core of the business here.

Write yours in one of those four shapes if you can. If your business genuinely has two, score twice and keep both columns, because a candidate that scores high on one and zero on the other is a real strategic argument you have just made visible rather than a scoring problem.

Part 2 · covers the nearest real evidence

The closest published work, and why it is next door rather than underneath

3 min live

There is a serious, well-measured paper in the neighbourhood of the sequencing rule. It does not support it. Being clear about the difference between adjacent and supporting is the teaching in this part, and it is a habit worth more than the paper itself.

ADJACENT IS NOT SUPPORTING one of these was measured; the other is a working rule with no paper behind it MEASURED, READ DIRECTLY Brynjolfsson, Hitt and Yang, 2002 "Intangible Assets: Computers and Organizational Capital", Brookings Papers on Economic Activity 1,216 firms, 1987 to 1997 matched survey subsample of 272 $10 to $12 of market value per dollar of computer capital, against about $1 for other physical capital NOT THE SAME CLAIM no citation bridges these UNCITED HEURISTIC The sequencing rule in part 1 Which use cases pay depends on how the business makes its money, so the same candidate is worth different amounts at different businesses. A deliberate search found no peer-reviewed source establishing it. This playbook's own rule, offered as exactly that. WHAT THE 2002 PAPER DOES SUPPORT the spend alone does not pay: the premium held only alongside organisational investment decentralisation, teams, training. That is a claim about complements to an IT investment, not a claim about revenue models, and it says nothing at all about which use case should go first. Cited here for the adjacent point only.
🔍 Click to zoom - the crossed link is the point of the diagram: a real finding sitting next to a rule is not evidence for the rule
LiveWhat the 2002 paper actually measured, in its own terms3 min▶

Erik Brynjolfsson, Lorin Hitt and Shinkyu Yang, "Intangible Assets: Computers and Organizational Capital", Brookings Papers on Economic Activity, 2002. It works across 1,216 firms from 1987 to 1997, with a matched survey subsample of 272 used to measure organisational practice.

The headline association: each dollar of computer capital was associated with roughly $10 to $12 of market value, against about $1 for other physical capital. On its own that number gets quoted as a reason to spend on technology, which is a misreading, because the paper's substantive finding is the condition attached to it. The premium was strongly conditional on complementary organisational investment: decentralised decision rights, team structures, training. Firms with the spend and without the organisational work did not show the same premium.

So the honest way to use it is as an argument about complements. The gate check in session 4 and the fit score tonight are both, in that light, part of the organisational work rather than a delay before the real project. Nobody builds a decentralisation programme out of a business case, but the habit of deciding what a case is worth before building it is the same family of thing.

Self-studyThe mismatch, said plainly, and why naming it is worth more than the citation3 min▶

The mismatch. That paper is about organisational complements to IT investment. The sequencing rule is about revenue models deciding which case pays. Those are different claims about different objects. The paper does not measure revenue models, does not rank use cases, and could not support a statement about sequencing even in principle. Attaching it to the rule as a citation would be a false footnote, and false footnotes are worse than none because they survive being challenged for a while.

The temptation to attach it anyway is strong and worth examining, because it is the same temptation that produces the failure-rate numbers in part 3. A rule that sounds right feels safer with a paper next to it. A reader skims the citation and reads the rule as established. Nobody lied, and the page is now making a claim it cannot defend.

The test that catches it. Ask what the source would have had to measure for the claim to follow. For the sequencing rule, that would be something like: firms sorted by revenue model, comparable use cases deployed at each, returns compared. If no source in your hand did that, no source in your hand supports the rule, however adjacent the subject. Say "this is our own rule" and move on. It costs one sentence and buys you the standing to challenge everybody else's citations.

Part 3 · covers the failure-rate numbers

The three numbers on the first slide, and what is behind them

4 min live

Nearly every business case for a data or AI roadmap opens with a failure rate. Eighty-five percent, eighty-seven percent, ninety percent. All three circulate as common knowledge, all three get quoted in board packs, and not one of them survives being looked up. This is the part of the evening people remember.

TRACE EACH NUMBER BACK BEFORE YOU QUOTE IT none of these three is usable, and the trace takes about ten minutes each QUOTED AS A LIVE RATE WHAT IS ACTUALLY BEHIND IT STATUS 85% of AI projects will deliver erroneous outcomes quoted as if it were current A 2018 vendor press release. The release itself could not be opened for this build, so nothing in it is stated here as fact. Even at face value, the forecast was scoped "through 2022". EXPIRED BY ITS OWN TERMS the window it forecast closed years ago 87% of data science projects never reach production the most quoted of the three A 2019 article reporting an unattributed remark made on a conference panel. No study, no survey, no sample, no methodology of any kind sits underneath it. NO SOURCE AT ALL a remark somebody made out loud, requoted until it looked measured 90% of models never reach production the one with a real correction A 2022 interview study went and read where the number came from. Its footnoted origins are a non-peer-reviewed monograph and a blog post. The study calls the statistic misguided. CALLED MISGUIDED many versions not shipping is what experimentation looks like You do not need any of them. The argument from the two gates stands on its own, and reaching for an unsound statistic to strengthen a case that was already sound makes the whole thing easier for one sceptic to dismiss in a single sentence.
🔍 Click to zoom - three traces, three different kinds of rot: an expiry nobody noticed, a remark with nothing behind it, and a number a study went and corrected
The claim, as it gets quotedWhat is actually behind itWhat you can say instead
"85 percent of AI projects will deliver erroneous outcomes"A vendor press release from 2018, which could not be opened for this build, so nothing inside it is stated here. Even taking the most-repeated wording at face value, it was a forecast scoped "through 2022""That number was a forecast about a window that closed. Quoting it in 2026 as a live rate means nobody checked. Here is what our own gated list says instead."
"87 percent of data science projects never reach production"A 2019 article reporting an unattributed remark from a conference panel. No study, no survey, no sample, no method. The claim has no source, only a trace"I went looking for the study behind that and there isn't one. It is a remark somebody made on a panel, requoted until it looked measured."
"90 percent of models never reach production"A 2022 interview study read its footnoted origins - a non-peer-reviewed monograph and a blog post - and states that the statistic is misguided, because constant experimentation is expected to produce many versions that never ship"A high non-deployment rate is not evidence of failure. If most of our experiments shipped, we would not be experimenting."
"Most AI projects fail" (no number attached)Nothing sound. The three circulating figures above are the ones people mean, and none of them establishes a rate. No defensible published failure rate was found for this build"Nothing in our case depends on how often other people fail. It depends on what we hold and what moves our money."
"It takes months to get a model into production"The only readable thing near this is an anecdote from one named participant in that 2022 study: P19 reported that "around 40 or 50% will make it to initial launch" and that staged deployment "took over three months". The paper does not claim it generalises"One engineer in one interview study reported that. Treat it as a single anecdote, not a benchmark, and go measure our own cycle."
LiveThe 2022 study, and the only figures it licenses you to repeat3 min▶

Shankar, Garcia, Hellerstein and Parameswaran, "Operationalizing Machine Learning: An Interview Study", 2022. 18 machine learning engineers, semi-structured interviews, IRB-reviewed, grounded-theory coding over 1,766 coded segments. It is a qualitative study and it says so, which is precisely why it is useful here: it is careful about what it can and cannot claim.

On the 90 percent figure it is unusually direct, and this is the sentence worth carrying around: "various articles claim that it is a problem for 90% of models to never make it to production... but we find that this statistic is misguided". The reasoning matters more than the verdict. In real practice, constant experimentation is expected to produce many model versions that never ship. That is what experimenting is. So a high non-deployment rate is not itself evidence of failure, and treating it as one gets the causality backwards: the teams doing the most careful work will have the worst-looking ratio.

On time to production, the study gives you one thing and only one thing: an attributed anecdote. Participant P19 reported that "around 40 or 50% will make it to initial launch", and that staged deployment "took over three months". That is one engineer describing one setting. The paper does not claim it generalises, and neither does this page, so if you repeat it, repeat the attribution with it.

Self-studyHow to trace a number in ten minutes, and the four rot patterns4 min▶

The trace. Find the earliest thing that states the number rather than repeating it. Follow the citation chain backwards, one hop at a time, and read what each hop actually says. You are looking for the point where the chain stops, and the interesting part is what is sitting there when it does: a survey with a method, or a sentence somebody said out loud.

Rot pattern one: the expiry nobody noticed. The number was a forecast with a stated horizon. The horizon passed. Nobody reread the original, so it now circulates as a description of the present. Tell: the original wording contains a date or a phrase like "through" or "by", and the requoters have dropped it.

Rot pattern two: the remark that grew a method. Somebody says a figure on a panel or in an interview. An article reports it faithfully. The next citation drops the "said on a panel" and keeps the number. Five hops later it is "research shows". Tell: nobody can name the sample size.

Rot pattern three: the misread denominator. The number is real but counts something other than what it is used for. Non-deployment counted as failure is the clean example: the denominator is every experiment, and experiments are supposed to not ship.

Rot pattern four: the round number. 90 percent, 80 percent, two thirds. Genuine measurements are rarely round, and a suspiciously round figure with no confidence interval anywhere near it has usually been rounded by a requoter rather than measured by anybody.

None of this is about being pedantic in meetings. It is about what happens when one person in the room has done the trace and you have not. Your whole case gets re-examined on the strength of the weakest number in it, and the weakest number was the one you added for colour.

Self-studyWhat to put on the first slide instead3 min▶

The failure rate was doing a job on that slide: it was establishing that this is hard and that a method is therefore warranted. You can do that job with your own material, and your own material is far harder to argue with because it is about the business in the room.

Put the blocked count on it. From session 4: how many of your candidates cannot be built today with the data you hold. That is a measured fact about your business, produced in an afternoon, and it makes the case for a method more forcefully than any industry percentage.

Put the zero-fit favourite on it. The most-requested item on the list, next to its revenue fit. If that fit is low, you have demonstrated the need for a second gate using the room's own enthusiasm as the evidence.

Put the two definitions on it. If two teams define your most-used metric differently, that screenshot is the strongest slide in the deck and it cost you ten minutes.

All three share a property the borrowed statistic does not have: nobody can dispute them by looking something up, because they are about you. The worst anybody can do is disagree with your judgement, and that conversation is the one you actually wanted.

Build-along

Score your gated list, then strip your own case

★ 22 min · bring the list from session 4

Five passes. You should end with an ordered list, one candidate you now understand differently, one popular candidate you can explain the popularity of, and a business case with nothing unsourced left in it.

Write in one sentence how the business actually makes its money, including whether the same customer pays again. Not the mission. Not the strategy. The mechanism by which cash arrives. If you cannot write it in one sentence, that is tonight's real finding and it outranks everything else on the list.

Score every candidate that cleared the data gate, 0 to 3, for fit against that sentence. 3 means it moves the thing in the sentence directly. 0 means it does not touch it. Do it fast and out loud, because the speed is what makes people willing to disagree with a score, and the disagreements are the useful part.

Find the candidate whose score would change most at a different kind of business. Rewrite your sentence as if you were paid once per transaction, or paid on availability, and rescore just that one. A big swing tells you the case is tightly coupled to your model, which is good news and a strong argument. A candidate that barely moves is one you can sequence without knowing anything about the business.

Find a candidate with a fit of zero and say why it is still popular. There is always one, and the reason is usually real: it is visible, it is what a competitor announced, it is what somebody was promised, it demos well in four weeks. Write the actual reason down. You are not going to win that argument by pointing at the score alone.

Open your own draft business case and strike every statistic you cannot trace to something you have read. Then read what is left and check the argument still stands. It will, because the argument was always the two gates. If the case collapses without the borrowed numbers, the numbers were carrying it, and you have just found that out cheaply rather than in front of a board.

If everything scores 2. It means the sentence is too vague to score against, not that every candidate is equally valuable. "We make money by serving customers well" cannot discriminate between anything. Go back and write the mechanism: what event causes cash to arrive, and how often does the same customer cause it again. A sentence that can produce a 0 and a 3 on the same list is a sentence that is doing work.

Homework

Before session 6

35 min
  1. Get your revenue sentence checked by somebody in finance or commercial. Read it to them and ask what is wrong with it. This takes five minutes and it is the single highest-leverage correction available to you, because every fit score on your list hangs off that one sentence.
  2. Rescore your top three candidates against a second revenue sentence, written as if the business were paid a different way. Keep both columns. Any candidate that scores high in both is unusually safe to start, and any candidate that only scores high in one is a bet on the business staying as it is.
  3. Trace one number you have quoted yourself. Pick a statistic you have used in a deck and follow it back until the chain stops. Write down what you find at the end. Everybody discovers at least one thing they were repeating without knowing, and doing it on your own material first is much more comfortable than having it done to you.
  4. Bring the ordered list. Session 6 turns it into the one-page artefact, so arrive with the gated candidates, their fit scores, your revenue sentence, and the blocked list from session 4.
Check yourself

Three questions

Clear all three and session 5 gets stamped in your passport.

1 · Why does the 2002 Brookings paper on intangible assets not support the sequencing rule in part 1?

The test is to ask what a source would have had to measure for the claim to follow: firms sorted by revenue model, comparable cases deployed at each, returns compared. Nothing like that is in the paper. A and C are the reflex objections and both are wrong in shape - the paper is careful and well measured, it just measures something else. Attaching it as a citation would be a false footnote, which is worse than no footnote because it survives a challenge for a while.

2 · A colleague opens the business case with "87 percent of data science projects never reach production". What is the accurate response?

B is the trap and it is the opposite of true: the 90 percent figure is the one a 2022 interview study called misguided after reading its footnoted origins, which were a non-peer-reviewed monograph and a blog post. C is the most common real-world answer and the weakest - a caveat on a claim with no source is still a claim with no source. Consistency between three unsound numbers is not corroboration.

3 · Why does the 2022 interview study argue that a high non-deployment rate is not evidence of failure?

The study's own words are that the statistic "is misguided", and the reasoning is about the denominator: it counts every experiment, and experiments are supposed to not ship. The one thing that study licenses you to repeat about timing is an attributed anecdote from participant P19, who reported around 40 or 50 percent reaching initial launch and staged deployment taking over three months. The paper does not claim that generalises, so neither should you.

Sources covered

What this session teaches, and where it came from

◐The sequencing rule: the revenue model decides which cases payThis playbook's own heuristic, uncited, and labelled as such on the page itself rather than in a footnote. A deliberate search found no peer-reviewed source establishing that a firm's revenue model determines which AI or automation investments pay off
✓The 2002 Brookings intangible-assets paperPrimary source read directly. The 1,216 firms from 1987 to 1997, the matched subsample of 272, the $10 to $12 against about $1, and the organisational-complements condition are all from it. The mismatch with the sequencing rule is stated rather than glossed
✓The 2022 correction of the 90 percent figurePrimary source read directly: 18 engineers, semi-structured interviews, IRB-reviewed, 1,766 coded segments. The "misguided" verdict is quoted verbatim, as is the P19 anecdote, which is given with its attribution because the paper does not claim it generalises
◐The 85 percent forecast and the 87 percent claimNeither is stated as fact. The 2018 release behind the 85 percent could not be opened for this build, so it appears only as an example of a number expired by its own stated horizon; the 87 percent appears only as a trace to an unattributed panel remark
◐Any AI or data project failure rate, and any generalisable time to productionNot claimed anywhere. No sound published rate was found, and the only readable timing figure is one named participant's anecdote. A consultancy discussion paper in this area could not be fetched and is not cited

Every fact on this page, and its verification tier, is recorded in the course's source map.

Session 5 cheat sheet · pin this

Write the sentence firstHow cash arrives, and whether the same customer pays again.
Fit is about the businessNot a property of the use case. Same candidate, different score.
The rule is ours and uncitedNo paper behind it. Say so before somebody asks.
Adjacent is not supportingAsk what the source would have had to measure.
85, 87, 90 all fail the traceExpired forecast, panel remark, corrected number.
Non-deployment is not failureExperiments are supposed to not ship.