Where we start
Two things happen tonight and they pull in opposite directions, which is deliberate. The first is a rule that puts your gated list in order. The second is a demolition of the evidence most people use to argue for a list at all. Holding both at once is the actual skill: you can have a strong argument and still refuse to decorate it with a number you cannot source.
★ What you walk out with tonight. One sentence describing how your business actually makes its money, a fit score against it for every candidate that cleared the data gate, the name of the candidate whose score would swing hardest at a different kind of business, and a business case with every unsourced statistic deleted from it.
The same case, worth two different amounts
6 min liveThe rule is one line. Which use cases pay depends on how the business makes its money, so a candidate that belongs at the top of one list belongs a long way down another, with nothing about the technology having changed.
Read this before you use the rule anywhere. The sequencing rule is this playbook's own heuristic and it carries no citation. A deliberate search for published work establishing that a firm's revenue model determines which AI or automation investments pay off found nothing peer-reviewed. There is no paper behind it, no study, no dataset. It is a working rule that has held up in practice, offered as exactly that, and you should hold it more loosely than anything cited on these pages. Session 6 asks you to catch people dressing their own heuristics as findings, so it would be absurd to do it here.
LiveWhy the session 3 bench scores would all move at your business4 min▶
The bench in session 3 scored revenue fit against a recurring-membership model, and that choice was shown on screen rather than hidden inside the arithmetic. It is worth being blunt about what follows from it: every single revenue-fit score on that bench would move at a business with a different model. Not some of them. All of them are readings taken against one sentence, and changing the sentence changes the readings.
That is not a weakness in the bench, it is the whole reason the second gate exists. The data gate in session 4 is close to objective: either the business holds the entity in full or it does not, and two reasonable people looking at the same field will usually agree. The revenue gate is a judgement, and it is a judgement about the business rather than about the use case. Which means it is the one gate you cannot outsource, cannot copy from another company's roadmap, and cannot buy from a vendor who does not know how your money arrives.
The practical consequence is small and annoying: you have to write the sentence first. Not the mission, not the strategy deck, the sentence. "We are paid a monthly fee and we keep about 70 percent of members past their first year." "We are paid once, per job, and about one customer in twenty ever comes back." Those two sentences produce different roadmaps out of an identical candidate list, and everybody in the room can check a fit score against them in about four seconds.
Self-studyThe four sentences, and the candidates that swap places between them4 min▶
Paid a recurring fee for continued access. Everything that protects the next renewal is near the top: churn scores, health scores, reasons for cancelling, onboarding friction. Anything that raises the size of a single purchase is a long way down, because a bigger basket does not extend the relationship.
Paid once per transaction, with no repeat relationship. Exactly reversed. Basket size, conversion on the page, pricing at the moment of sale, fraud at the moment of sale. A churn model has nothing to hold on to, which does not stop people asking for one, and that is the subject of the fourth build-along pass.
Paid on volume you do not control, such as usage or claims. The money moves with throughput and with loss, so forecasting demand and detecting the expensive tail matter most. Retention often matters less than it feels like it should, because the customer is locked in by something other than satisfaction.
Paid on an asset being available. Utilities, rentals, fleets, equipment. Availability is revenue, so failure prediction and maintenance scheduling go to the top of a list where they would be mid-table almost anywhere else, and the same equipment-sensor case that is an expensive curiosity at a gym chain is the core of the business here.
Write yours in one of those four shapes if you can. If your business genuinely has two, score twice and keep both columns, because a candidate that scores high on one and zero on the other is a real strategic argument you have just made visible rather than a scoring problem.
The closest published work, and why it is next door rather than underneath
3 min liveThere is a serious, well-measured paper in the neighbourhood of the sequencing rule. It does not support it. Being clear about the difference between adjacent and supporting is the teaching in this part, and it is a habit worth more than the paper itself.
LiveWhat the 2002 paper actually measured, in its own terms3 min▶
Erik Brynjolfsson, Lorin Hitt and Shinkyu Yang, "Intangible Assets: Computers and Organizational Capital", Brookings Papers on Economic Activity, 2002. It works across 1,216 firms from 1987 to 1997, with a matched survey subsample of 272 used to measure organisational practice.
The headline association: each dollar of computer capital was associated with roughly $10 to $12 of market value, against about $1 for other physical capital. On its own that number gets quoted as a reason to spend on technology, which is a misreading, because the paper's substantive finding is the condition attached to it. The premium was strongly conditional on complementary organisational investment: decentralised decision rights, team structures, training. Firms with the spend and without the organisational work did not show the same premium.
So the honest way to use it is as an argument about complements. The gate check in session 4 and the fit score tonight are both, in that light, part of the organisational work rather than a delay before the real project. Nobody builds a decentralisation programme out of a business case, but the habit of deciding what a case is worth before building it is the same family of thing.
Self-studyThe mismatch, said plainly, and why naming it is worth more than the citation3 min▶
The mismatch. That paper is about organisational complements to IT investment. The sequencing rule is about revenue models deciding which case pays. Those are different claims about different objects. The paper does not measure revenue models, does not rank use cases, and could not support a statement about sequencing even in principle. Attaching it to the rule as a citation would be a false footnote, and false footnotes are worse than none because they survive being challenged for a while.
The temptation to attach it anyway is strong and worth examining, because it is the same temptation that produces the failure-rate numbers in part 3. A rule that sounds right feels safer with a paper next to it. A reader skims the citation and reads the rule as established. Nobody lied, and the page is now making a claim it cannot defend.
The test that catches it. Ask what the source would have had to measure for the claim to follow. For the sequencing rule, that would be something like: firms sorted by revenue model, comparable use cases deployed at each, returns compared. If no source in your hand did that, no source in your hand supports the rule, however adjacent the subject. Say "this is our own rule" and move on. It costs one sentence and buys you the standing to challenge everybody else's citations.
The three numbers on the first slide, and what is behind them
4 min liveNearly every business case for a data or AI roadmap opens with a failure rate. Eighty-five percent, eighty-seven percent, ninety percent. All three circulate as common knowledge, all three get quoted in board packs, and not one of them survives being looked up. This is the part of the evening people remember.
| The claim, as it gets quoted | What is actually behind it | What you can say instead |
|---|---|---|
| "85 percent of AI projects will deliver erroneous outcomes" | A vendor press release from 2018, which could not be opened for this build, so nothing inside it is stated here. Even taking the most-repeated wording at face value, it was a forecast scoped "through 2022" | "That number was a forecast about a window that closed. Quoting it in 2026 as a live rate means nobody checked. Here is what our own gated list says instead." |
| "87 percent of data science projects never reach production" | A 2019 article reporting an unattributed remark from a conference panel. No study, no survey, no sample, no method. The claim has no source, only a trace | "I went looking for the study behind that and there isn't one. It is a remark somebody made on a panel, requoted until it looked measured." |
| "90 percent of models never reach production" | A 2022 interview study read its footnoted origins - a non-peer-reviewed monograph and a blog post - and states that the statistic is misguided, because constant experimentation is expected to produce many versions that never ship | "A high non-deployment rate is not evidence of failure. If most of our experiments shipped, we would not be experimenting." |
| "Most AI projects fail" (no number attached) | Nothing sound. The three circulating figures above are the ones people mean, and none of them establishes a rate. No defensible published failure rate was found for this build | "Nothing in our case depends on how often other people fail. It depends on what we hold and what moves our money." |
| "It takes months to get a model into production" | The only readable thing near this is an anecdote from one named participant in that 2022 study: P19 reported that "around 40 or 50% will make it to initial launch" and that staged deployment "took over three months". The paper does not claim it generalises | "One engineer in one interview study reported that. Treat it as a single anecdote, not a benchmark, and go measure our own cycle." |
LiveThe 2022 study, and the only figures it licenses you to repeat3 min▶
Shankar, Garcia, Hellerstein and Parameswaran, "Operationalizing Machine Learning: An Interview Study", 2022. 18 machine learning engineers, semi-structured interviews, IRB-reviewed, grounded-theory coding over 1,766 coded segments. It is a qualitative study and it says so, which is precisely why it is useful here: it is careful about what it can and cannot claim.
On the 90 percent figure it is unusually direct, and this is the sentence worth carrying around: "various articles claim that it is a problem for 90% of models to never make it to production... but we find that this statistic is misguided". The reasoning matters more than the verdict. In real practice, constant experimentation is expected to produce many model versions that never ship. That is what experimenting is. So a high non-deployment rate is not itself evidence of failure, and treating it as one gets the causality backwards: the teams doing the most careful work will have the worst-looking ratio.
On time to production, the study gives you one thing and only one thing: an attributed anecdote. Participant P19 reported that "around 40 or 50% will make it to initial launch", and that staged deployment "took over three months". That is one engineer describing one setting. The paper does not claim it generalises, and neither does this page, so if you repeat it, repeat the attribution with it.
Self-studyHow to trace a number in ten minutes, and the four rot patterns4 min▶
The trace. Find the earliest thing that states the number rather than repeating it. Follow the citation chain backwards, one hop at a time, and read what each hop actually says. You are looking for the point where the chain stops, and the interesting part is what is sitting there when it does: a survey with a method, or a sentence somebody said out loud.
Rot pattern one: the expiry nobody noticed. The number was a forecast with a stated horizon. The horizon passed. Nobody reread the original, so it now circulates as a description of the present. Tell: the original wording contains a date or a phrase like "through" or "by", and the requoters have dropped it.
Rot pattern two: the remark that grew a method. Somebody says a figure on a panel or in an interview. An article reports it faithfully. The next citation drops the "said on a panel" and keeps the number. Five hops later it is "research shows". Tell: nobody can name the sample size.
Rot pattern three: the misread denominator. The number is real but counts something other than what it is used for. Non-deployment counted as failure is the clean example: the denominator is every experiment, and experiments are supposed to not ship.
Rot pattern four: the round number. 90 percent, 80 percent, two thirds. Genuine measurements are rarely round, and a suspiciously round figure with no confidence interval anywhere near it has usually been rounded by a requoter rather than measured by anybody.
None of this is about being pedantic in meetings. It is about what happens when one person in the room has done the trace and you have not. Your whole case gets re-examined on the strength of the weakest number in it, and the weakest number was the one you added for colour.
Self-studyWhat to put on the first slide instead3 min▶
The failure rate was doing a job on that slide: it was establishing that this is hard and that a method is therefore warranted. You can do that job with your own material, and your own material is far harder to argue with because it is about the business in the room.
Put the blocked count on it. From session 4: how many of your candidates cannot be built today with the data you hold. That is a measured fact about your business, produced in an afternoon, and it makes the case for a method more forcefully than any industry percentage.
Put the zero-fit favourite on it. The most-requested item on the list, next to its revenue fit. If that fit is low, you have demonstrated the need for a second gate using the room's own enthusiasm as the evidence.
Put the two definitions on it. If two teams define your most-used metric differently, that screenshot is the strongest slide in the deck and it cost you ten minutes.
All three share a property the borrowed statistic does not have: nobody can dispute them by looking something up, because they are about you. The worst anybody can do is disagree with your judgement, and that conversation is the one you actually wanted.
Score your gated list, then strip your own case
★ 22 min · bring the list from session 4Five passes. You should end with an ordered list, one candidate you now understand differently, one popular candidate you can explain the popularity of, and a business case with nothing unsourced left in it.
Write in one sentence how the business actually makes its money, including whether the same customer pays again. Not the mission. Not the strategy. The mechanism by which cash arrives. If you cannot write it in one sentence, that is tonight's real finding and it outranks everything else on the list.
Score every candidate that cleared the data gate, 0 to 3, for fit against that sentence. 3 means it moves the thing in the sentence directly. 0 means it does not touch it. Do it fast and out loud, because the speed is what makes people willing to disagree with a score, and the disagreements are the useful part.
Find the candidate whose score would change most at a different kind of business. Rewrite your sentence as if you were paid once per transaction, or paid on availability, and rescore just that one. A big swing tells you the case is tightly coupled to your model, which is good news and a strong argument. A candidate that barely moves is one you can sequence without knowing anything about the business.
Find a candidate with a fit of zero and say why it is still popular. There is always one, and the reason is usually real: it is visible, it is what a competitor announced, it is what somebody was promised, it demos well in four weeks. Write the actual reason down. You are not going to win that argument by pointing at the score alone.
Open your own draft business case and strike every statistic you cannot trace to something you have read. Then read what is left and check the argument still stands. It will, because the argument was always the two gates. If the case collapses without the borrowed numbers, the numbers were carrying it, and you have just found that out cheaply rather than in front of a board.
If everything scores 2. It means the sentence is too vague to score against, not that every candidate is equally valuable. "We make money by serving customers well" cannot discriminate between anything. Go back and write the mechanism: what event causes cash to arrive, and how often does the same customer cause it again. A sentence that can produce a 0 and a 3 on the same list is a sentence that is doing work.
Before session 6
35 min- Get your revenue sentence checked by somebody in finance or commercial. Read it to them and ask what is wrong with it. This takes five minutes and it is the single highest-leverage correction available to you, because every fit score on your list hangs off that one sentence.
- Rescore your top three candidates against a second revenue sentence, written as if the business were paid a different way. Keep both columns. Any candidate that scores high in both is unusually safe to start, and any candidate that only scores high in one is a bet on the business staying as it is.
- Trace one number you have quoted yourself. Pick a statistic you have used in a deck and follow it back until the chain stops. Write down what you find at the end. Everybody discovers at least one thing they were repeating without knowing, and doing it on your own material first is much more comfortable than having it done to you.
- Bring the ordered list. Session 6 turns it into the one-page artefact, so arrive with the gated candidates, their fit scores, your revenue sentence, and the blocked list from session 4.
Three questions
Clear all three and session 5 gets stamped in your passport.
1 · Why does the 2002 Brookings paper on intangible assets not support the sequencing rule in part 1?
The test is to ask what a source would have had to measure for the claim to follow: firms sorted by revenue model, comparable cases deployed at each, returns compared. Nothing like that is in the paper. A and C are the reflex objections and both are wrong in shape - the paper is careful and well measured, it just measures something else. Attaching it as a citation would be a false footnote, which is worse than no footnote because it survives a challenge for a while.
2 · A colleague opens the business case with "87 percent of data science projects never reach production". What is the accurate response?
B is the trap and it is the opposite of true: the 90 percent figure is the one a 2022 interview study called misguided after reading its footnoted origins, which were a non-peer-reviewed monograph and a blog post. C is the most common real-world answer and the weakest - a caveat on a claim with no source is still a claim with no source. Consistency between three unsound numbers is not corroboration.
3 · Why does the 2022 interview study argue that a high non-deployment rate is not evidence of failure?
The study's own words are that the statistic "is misguided", and the reasoning is about the denominator: it counts every experiment, and experiments are supposed to not ship. The one thing that study licenses you to repeat about timing is an attributed anecdote from participant P19, who reported around 40 or 50 percent reaching initial launch and staged deployment taking over three months. The paper does not claim that generalises, so neither should you.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 5 cheat sheet · pin this
You now have all four moves: the candidates sorted, a verdict on each, a gate against the data you hold, and an order against the way the money arrives. What you do not have is one thing anybody else can pick up and use. Session 6 turns the whole set into the single-page artefact a vertical roadmap can import, with a final scorecard across the four moves and where to go next once the page exists. Bring the ordered list, the fit scores, the revenue sentence and the blocked pile, because session 6 is assembly rather than new material.