Signal versus vendor claim 15 min live
Almost every number in a sales-AI deck is true about something. The question is never "is this a lie?" - it is "what exactly was measured, and would it still be true about my team?" Four times out of five the answer is no, and it takes four questions to find out. Here is the test, and here is what dies at each filter.
LiveThe five-question test, and how to ask it politely6 min▶
You are not trying to catch anyone out. You are trying to find out whether the number transfers to your team, and a good vendor will enjoy the conversation. Ask in this order, because the cheap questions kill the most claims.
| Question | What a weak answer sounds like |
|---|---|
| 1. Who was measured? | "Our customers." How many, in what segment, selling what, at what price point, and were they already your best performers before they bought? |
| 2. Against what baseline? | "Compared to before." Before what - the tool, the new manager, the new list, or the quarter their biggest competitor imploded? |
| 3. Over how long? | "In the pilot." Three good weeks is a mood. Sales cycles are longer than most pilots, which is how a lift in activity gets reported as a lift in revenue. |
| 4. Who paid for it? | "An independent study." Commissioned by whom, and did the commissioning party see the results before deciding to publish them? |
| 5. Did anyone repeat it? | "It is proprietary." A finding nobody else has reproduced is a data point, not a fact, and pricing a three-year contract off it is a bet you did not know you placed. |
LiveThe same claim, written two ways4 min▶
Here is a real-shaped claim in the form you will be shown it, and the form you would need in order to act on it. Nothing in the second version is hostile. It is simply the first version with the missing four questions answered.
"Teams using our platform see a 312% increase in qualified pipeline."
"Across 34 mid-market SaaS teams (8-25 reps, ASP $20k-$80k), qualified pipeline per rep rose 312% over two quarters versus their own prior two quarters, on their own stage definitions. Caveats we will state out loud: 19 of the 34 also changed their ICP in the same period, the study was run by us, and teams that churned before quarter two are not in the sample."
- "Their own stage definitions" is doing enormous work. If qualified means whatever the CRM says, a 312% lift can be a field-hygiene change.
- "19 of the 34 also changed their ICP" is the confound that would have been invisible in the deck version. It does not make the claim worthless - it makes it a claim about ICP discipline plus a tool.
- "Teams that churned are not in the sample" is survivorship, and it is present in nearly every vendor number you will ever see.
Compare that to a number that survives the test. Gartner's 3.7x quota finding names the population (1,026 B2B sellers), the window (January to March 2024), the comparison (sellers who effectively partner with AI versus those who do not) and the funder (Gartner, selling research to buyers rather than software to sellers). You can still argue about what "partner" means. You cannot argue that you were not told what was measured, which is the entire difference.
What the good research actually supports 12 min live
Now the useful half. A small number of findings survive the five filters, and they are enough to build a plan on. Everything to the right of the line below may still turn out to be true - it simply is not evidence yet, and the honest leadership move is to say which column a number is in before you spend against it.
LiveThe five Gartner findings, and the three things they do not say5 min▶
The left column of the board holds up. Here it is in full, with the boundary of each claim marked, because a finding used past its boundary becomes a vendor claim in your own mouth.
- 3.7x quota attainment for sellers who effectively partner with AI, from a survey of 1,026 B2B sellers. Boundary: this is a correlation in a survey. Excellent sellers may partner well because they are excellent. Treat it as evidence that the habit is worth coaching, not proof that the tool caused the quota.
- Nearly 5 hours a week returned per seller. Boundary: saved, not redeployed. The hours are real; their destination is a management decision.
- 72% of sales organisations reinvest none of it in high-value activities. Boundary: this is the honest headline of the whole category, and it is the one number no vendor will lead with.
- 2.6x commercial growth for organisations providing AI-enabled next best actions. Boundary: "next best action" is a design commitment, not a feature toggle. Buying a product that has the phrase in it is not the intervention.
- 2.4x good revenue growth where organisations emphasise upskilling sellers on AI. Boundary: emphasise means curriculum, coaching and manager time, not a lunch-and-learn.
And the three things this body of evidence genuinely does not support, however often you will hear them:
- That AI closes deals. Nothing credible shows autonomous agents closing B2B deals. Gartner's own agent forecast has fewer than 40% of sellers even reporting a productivity gain by 2028.
- That more sends produce more pipeline. The benchmark data points the opposite way, and it has for two years running.
- That adoption predicts outcome. Adoption is near-universal and the 72% figure exists anyway. If usage predicted results, that number could not be true.
LiveThe benchmark picture, caveats attached4 min▶
Published 2025-2026 cold-email benchmark reports, read the way you would read a vendor number, because that is largely what they are.
| Finding | How much weight to put on it |
|---|---|
| Average reply about 3.4%, down from about 5.1% | Directionally solid - the decline shows up across independent samples and it matches the mechanism. Do not treat the decimal as precise. |
| Top quartile 15-25% | The most useful number here. The spread between average and good got wider, which means discipline is being rewarded, not commoditised. |
| Timeline hooks about 2.3x problem hooks | Strong and consistent, and it is the one finding your reps can act on tomorrow morning. It is why the trigger event is the heaviest lever in the rep track. |
| Under 80 words performs best | Reliable across samples, and free to implement. The instinct to explain the product is what kills the reply. |
| Most replies by around day 10 | Solid enough to set cadence policy on. One-and-done sending throws away the majority of the work already done. |
The caveat that applies to every row above. These samples come from sending platforms, which means they describe the behaviour of people who bought a sending platform. That population is not your population, and it skews toward high-volume outbound. Use the reports for the shape of the curve - decline, spread, what lifts a reply - and use your own instance for the level. Anyone quoting a benchmark to two decimal places about your team is selling you something.
Reading the evidence into a plan self-study · four cards
Four consequences of the two parts above. One is about headcount, two are about the metrics on your weekly dashboard, and the last is the thing that makes you independent of every benchmark in this session.
Self-studyThe 2030 buyer prediction and your headcount plan3 min read▶
Gartner predicts that by 2030, 75% of B2B buyers will prefer sales experiences that prioritise human interaction over AI. Sit with that next to the agent forecast for a moment, because most planning decks quote one and ignore the other.
- It is a demand-side prediction. It says nothing about how many sellers you need. It says a lot about what those sellers must be free to do, which is spend unhurried time with people.
- The planning consequence is mix, not headcount. If human interaction becomes the preferred experience, capacity for it becomes the constraint. That argues for automating the preparation aggressively and protecting seller hours for contact - which is precisely the 2.6x next-best-action finding read from the buyer's side.
- The risk is cutting the wrong side. A plan that reduces seller capacity because "AI covers it" is betting against the only buyer-side prediction in this evidence base. If you are going to make that bet, make it explicitly and write down what would falsify it.
- Segment it. The preference is strongest where deals are large and consequential. Cutting human contact in your enterprise segment to fund automation in SMB is a very different decision from the reverse, and the deck usually will not distinguish them.
Self-studyActivity metrics that lie, outcome metrics that do not3 min read▶
Every metric on this list was fine when the underlying action was expensive. AI made the actions cheap, and a metric that measures a cheap action becomes a target your team can hit while going backwards.
| Lies now | Why | Use instead |
|---|---|---|
| Emails sent | Cost per send collapsed to roughly nothing, so the number only measures willingness | Replies per 100 sends, split by whether a real trigger was used |
| Calls logged | Auto-logging inflates it without anyone speaking to a human | Conversations over two minutes, and next steps agreed |
| Drafts generated | An input that gets cheaper every quarter. Rewarding it rewards volume | Emails sent after a human edit, and what the edit rate is |
| Meetings booked | Still useful, but gameable at the top of the funnel by lowering the bar | Meetings that reach a second meeting |
| Hours saved | Real, but incomplete - it is the 72% figure hiding in your own dashboard | Hours saved with a named destination you can point at |
The test for any metric on your weekly review: could a rep move this number by doing more of something that costs nothing? If yes, it is now an activity metric regardless of what it used to be, and a3 shows what happens when you attach one to a forecast.
Self-studyYour reply rate is a targeting metric3 min read▶
This is the single most useful reframe in the leader track, and it is the reason your team's "our messaging is not landing" conversation keeps failing to fix anything.
- The rep-track demonstration. In b1 the same email with every lever turned on reaches 19.2% predicted reply for one well-targeted prospect, while the same treatment across an eight-prospect week averages 10.1%. The copy is identical. The gap is entirely the list.
- Three rows never move. Wrong seat, under the ICP price floor, and a prior opt-out. No sentence anyone writes raises those, and the opt-out row must score zero forever - that one is a compliance rail, not a performance question (a5).
- So when reply rate drops, look at the list first. Ask what changed in targeting before you approve a messaging project. Most messaging projects are targeting problems wearing a copywriting costume, and they are expensive, slow and morale-draining ways to not fix a list.
- Segment your own number. One blended reply rate hides everything. Split by segment, by seat seniority, and by whether a genuine trigger event was used, and the picture usually resolves in an afternoon.
Self-studyRun your own six-week test instead4 min read▶
Everything in this session is someone else's data. Six weeks of your own beats all of it, and it is cheap enough that there is no honest reason not to.
Pick one claim, not five. "Timeline hooks beat problem hooks for us." One variable, one segment, one team. A test with three changes in it answers nothing.
Split by list, not by rep. Randomly assign accounts, not people. Splitting by rep measures your reps, which you already knew how to do.
Fix the sample size before you start. At a 3-4% base rate you need real volume to see a difference. Write down the number of sends per arm in advance, and write down what result would make you stop.
Run six weeks, not three. Most replies land by around day 10, so a three-week test truncates the follow-up curve and flatters whichever arm sent first.
Measure to the second meeting. Reply rate is the cheap metric. Meetings that reach a second meeting is the one that pays rent.
Write the result down either way. Including when it says nothing. A file of your own null results is the most expensive thing to fake and the cheapest thing to own.
Do this twice a year and you stop being a market for benchmark decks. You will also find that your own numbers make a far better slide in front of a board than anyone else's, because you can answer all five filter questions about them without flinching.
Before session 3 ◐ 40-50 min total
- Run the five-question test in writing. Take one vendor claim you are currently being sold, and write the answers to all five questions on one page. Where you cannot answer, write "not stated" rather than guessing - that page is your negotiating position and it took twenty minutes.
- Send the questions you could not answer. Straight to the vendor, before the next call. How they respond to question 4 is information you cannot get any other way.
- Pull your own reply rate. Then segment it by whether a real, dated trigger event was used in the first line. Most teams have never split it this way, and the split is usually larger than any tool decision you are considering.
- Find one activity metric on your weekly dashboard that a rep could move by doing more of something free. Decide whether to replace it or retire it, and bring that decision to a3.
- Sketch one six-week test you could start on Monday. One claim, one segment, sample size written down in advance.
Official sources covered
This session is the evidence base of the whole course, read with its caveats visible. The Gartner findings are named with their populations and windows; the benchmark reports are treated as directional rather than precise, because their samples come from sending platforms. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · You have thirty seconds and one vendor number in front of you. Which two questions kill the most weak claims?
Baseline and funding are the two filters most deck numbers die at. A lift with no stated baseline could be a field-hygiene change or a good quarter, and a study the vendor commissioned and reviewed before publishing has a survivorship problem built in. Both questions are perfectly friendly to ask in a full room.
2 · Your team's reply rate dropped this quarter. What should you look at first?
In the rep-track demonstration, identical copy scores 19.2% for one well-targeted prospect and averages 10.1% across a real week, because three rows are capped by targeting: wrong seat, under the ICP price floor, and a prior opt-out. No sentence lifts those, and the opt-out row is a compliance rail. Segment your own rate by whether a real trigger was used before you approve a messaging project.
3 · Why is "hours saved" an incomplete metric even though the saved hours are real?
The five hours a week are well supported. What is not automatic is where they go, and 72% of sales organisations reinvest none of it in high-value activities. The metric that survives is hours saved and redeployed into something you can point at, which is also the shape of the 2.6x next-best-action finding.