learn-marketing-with-phoebe / Session 4 of 6
Learn Marketing with Phoebe · Session 4 of 6

The funnel bench

Session 3 chose channels by asking where the customers already are. Tonight the question gets a price. A Brazilian marketplace published its real sales funnel: every lead that asked to sell on it, the channel each lead came from, and which ones became paying sellers. Those rows sit in this page. You split a month of budget across six channels, the page counts what the split buys in leads, deals, cost per customer and months to earn it back, and one button breaks the data on purpose to prove the bench measures what it says it does.

🟡 Bench night Bring your session 3 channel shortlist No code tonight, all clicking 45 min
0-3 · Welcome 3-12 · Leads, deals, CAC 12-38 · The bench, preset by preset 38-45 · Q&A
Part 0

Where we start

The data is the public Marketing Funnel by Olist: 8,000 marketing-qualified leads, sellers who filled in a request to sell on the Olist Store between June 2017 and June 2018, and the 842 of them that closed. Each lead carries the channel it arrived through. That is everything a channel decision needs on the conversion side, and nothing on the cost side, because the dataset has no spend. So the bench splits the job honestly: what happened is measured from the rows; what things cost is an assumption panel you can see and change.

🎫 Clear all three quiz questions to stamp your passport Live = presented in the 45 minutes Self-study = the full depth, after class ★ Build-along = click the bench yourself

★ What you walk out with tonight. A budget split you can defend with a number, the one sentence that separates a lead from a deal in any channel meeting, and the habit of asking two questions before trusting any close rate: how many deals is it built on, and could the data be broken in a way that produces it?

Part 1 · covers leads, close rate and CAC

A lead is a cost, a deal is a customer

9 min live

Money buys leads at some cost per lead. Leads become customers at some close rate. Customer acquisition cost, CAC, is the money divided by the customers, and nothing else. Two multipliers sit between the budget and the CAC, and a channel can win the first and lose the second.

ONE MONTH, THE "MOST LEADS" SPLIT, DEFAULT ASSUMPTIONS Budget $20,000 a month, assumed Leads 488.8 bought at a cost per lead Deals 50.5 new sellers CAC $396 budget / deals ÷ cost per lead × close rate measured on real rows assumed, and rising cap = 2x real volume spend on one channel the next lead costs more than the last one Every split pays the same budget; only the deals it buys differ.
🔍 Click to zoom - the close rate is the multiplier the lead count never shows you
LiveTwo multipliers, one division4 min▶

Cost per lead turns money into leads. It is never flat: the first leads from a channel come from the people most ready to respond, and each extra dollar reaches people slightly less ready. The bench models that with a curve that starts at the stated cost per lead and flattens toward a cap, set by default at twice the channel's real monthly lead volume. That curve is constructed and labelled; the shape is the standard one, the numbers are not anybody's measurements.

Close rate turns leads into customers. It is measured, per channel, from which of the 5,998 leads in the 2018 cohort became sellers. It varies by more than three to one across the channels: paid search 14.8 percent, social 6.3 percent, email 4.0 percent.

CAC is the budget divided by the deals. A channel with cheap leads and a poor close rate can have a worse CAC than one with expensive leads that close. Social's assumed cost per lead is $25 against paid search's $60; at the first dollar, social's CAC is about $25 / 0.063, roughly $400, and paid search's is about $60 / 0.148, roughly $405. Nearly a tie, from very different routes.

LiveWhy the bench starts in January 20183 min▶

The closed-deal records begin on 5 December 2017. Leads that arrived in 2017 had their deals logged, if at all, only from that date on, so the 2,002 leads from 2017 show just 61 closes, 3.0 percent, against 13.0 percent for the 5,998 leads from January to May 2018. The 2017 leads did not get worse. The deal log started late.

The bench defaults to the 2018 cohort for that reason, and gives you a checkbox to include 2017 anyway. Tick it and every CAC on the ladder rises, the even split from $524 to $742, because a third of the leads now look as if they never close. The ordering of the presets survives. The dollar figures do not, which is the general rule for any funnel number taken from a log whose start date you have not checked.

Real world

The channel nobody can buy converts best. 813 of the 2018 leads have origin "unknown", and they close at 20.7 percent, the highest rate of any large group in the table. Untracked does not mean bad; it often means the lead came through a route the tracking never saw, a conversation, a forwarded link, a salesperson's own contact. The bench cannot spend on "unknown", and neither can you. What you can do is fix the tracking, which is the subject of the attribution course rather than this one.

Build-along · the bench

Six channels, one budget, computed in front of you

★ 26 min · everyone builds

Every count in the table is computed from the real rows when the page loads. Every dollar comes from the assumptions panel. Work down the steps below the bench in order; each one prints a number this page quotes, and if your bench prints something different, check whether you changed an assumption.

CAC BY PRESET, 2018 COHORT, DEFAULT ASSUMPTIONS (lower is better) $300 $630 $368 best split $380 deals $396 most leads $524 even split $627 cheapest leads SHUFFLED: most leads $316 = deals $316 The anti-lever loses $16 a customer to conversion weighting, and the gap vanishes when conversion is shuffled away.
🔍 Click to zoom - the bigger trap on this data is the cheapest lead, not the biggest channel

Read the real table before touching a dollar. Leave every assumption at its default. Find the two channels that bring the most leads, and the two that close best. On this data they are the same two, organic and paid search, which matters in step 3.

★ Step 1 · what the bench printsOrganic search 1,736 leads 255 closed 14.7% (13.1% to 16.4%) Paid search 1,182 leads 175 closed 14.8% (12.9% to 16.9%) Social 1,051 leads 66 closed 6.3% (5.0% to 7.9%) Email 347 leads 14 closed 4.0% ONLY 14 DEALS Referral 211 leads 24 closed 11.4% ONLY 24 DEALS Display 75 leads 5 closed 6.7% ONLY 5 DEALS All 5,998 leads, 781 closed, overall 13.0%

Click "Even split". One sixth each. It looks fair and spends a sixth of the money on display, a channel that can only absorb so much before it saturates.

★ Step 2 · even splitExpected leads 431.4 · deals 38.1 · CAC $524 · payback 10.1 months CAC 42.5% above the best split the bench found ($368)

Click "Spend where the most leads come from". Budget in proportion to each channel's real lead count. It buys more leads than any other preset, and that is exactly why it feels right in a meeting.

★ Step 3 · the obvious moveExpected leads 488.8 · deals 50.5 · CAC $396 · payback 7.6 months CAC 7.5% above the best split the bench found ($368)

Click "Spend where the deals come from". Same idea, weighted by real closed deals instead of real leads. Fewer leads, more customers.

★ Step 4 · conversion-weightedExpected leads 436.9 · deals 52.7 · CAC $380 · payback 7.3 months CAC 3.1% above the best split the bench found ($368)

Click "Next dollar to the next best deal". The bench hands out the budget $100 at a time, each time to the channel where the next $100 buys the most deals. Read the per-channel table: display gets nothing, organic gets most.

★ Step 5 · the best split under these assumptionsExpected leads 419.9 · deals 54.3 · CAC $368 · payback 7.1 months Organic $14,400 · Paid $3,500 · Social $1,500 · Email $400 · Referral $200 · Display $0

Click "Cheapest leads first". Budget in proportion to how cheap each channel's leads are. Email and social get the most money, and they are the two worst closers.

★ Step 6 · the expensive bargainExpected leads 412.1 · deals 31.9 · CAC $627 · payback 12.1 months CAC 70.4% above the best split the bench found ($368)

Press "Break it". The closed flags are dealt back out across all 5,998 leads at random, so a channel's close rate now says nothing about the channel. Click "most leads" and "deals" again.

★ Step 7 · the break buttonSocial 13.3% · Email 10.7% · Display 10.7% (all near the overall 13.0%) Most leads: CAC $316 · payback 6.1 Deals: CAC $316 · payback 6.1 The advantage of conversion weighting is gone, because the conversion information is gone.

Restore the flags, then change one assumption. Raise social's cost per lead from $25 to $40, or headroom from 2 to 3. Watch which preset ranks first. Write down the assumption that would have to be true for "most leads" to win; that sentence is what to check before anyone spends the money.

★ Step 8 · your own assumptionsYour numbers will differ. Record: the assumption you changed, the preset that now ranks first, and its CAC.
Part 2 · covers the anti-lever

Most leads, fewer customers

5 min live

"Put the money where the leads come from" is the most natural sentence in a budget meeting, and on this data it is wrong, though by less than you might hope. It buys 488.8 leads, more than any other preset, and 50.5 customers, fewer than weighting by deals. The gap is $16 a customer, 4.3 percent. Small, because the two biggest lead sources here are also the two best closers. All of the damage comes from one channel.

TWO PRESETS, SAME $20,000 SOCIAL, 2018 COHORT Leads, most leads Leads, deals-weighted Deals, most leads Deals, deals-weighted CAC, most leads CAC, deals-weighted 488.8 436.9 50.5 52.7 more $396 $380 lower 17.5% 8.5% of leads of deals More leads, fewer deals: the extra leads came from a channel that closes at 6.3 percent.
🔍 Click to zoom - volume and conversion agree on the top two channels here, and part ways on social
LiveWhy the gap is small here, and when it is not3 min▶

Weighting by leads and weighting by deals give the same answer wherever a channel's share of leads equals its share of deals. Organic search is 28.9 percent of the 2018 leads and 32.7 percent of the deals; paid search 19.7 and 22.4. Close. Social is 17.5 and 8.5. Not close. The lead-weighted split sends social roughly twice the money its deals justify, and that is the whole $16.

So the honest lesson is not "volume is always a trap". It is that volume is the right signal only to the extent that conversion is uniform, and you cannot know whether it is uniform without looking. On a funnel where the biggest lead source is also the worst closer, the same sentence would cost far more than 4.3 percent. Check the second column before the first one wins the meeting.

Self-studyThe bigger trap: cheapest leads first3 min▶

Cost per lead is the metric ad platforms report first, and the cheapest-leads preset follows it: email at an assumed $15 and social at $25 get the most money. They are the two worst closers in the table, so the split buys 412.1 leads and only 31.9 customers, CAC $627, 70.4 percent above the best split. A lower cost per lead bought a higher cost per customer.

The fix is the same in both traps: rank channels by what a customer costs, never by what a lead costs, and treat cost per lead as an input to CAC rather than a result.

Real world

Social, in this funnel, is a lead machine and a weak closer. 1,051 real 2018 leads, the third biggest source, and 66 deals. Whether those leads were poorly targeted, arrived earlier in their decision, or were handled differently by sales, the rows cannot say. What they can say is that a budget meeting looking only at the lead column would have scaled the channel that converts at under half the rate of the two above it.

Part 3 · covers small samples, missing revenue and the break button

What is measured, what is assumed, what is too thin to trust

5 min live

Every close rate on the bench is a fact about real rows, and some of those facts rest on a handful of deals. Display's 6.7 percent is five deals out of 75 leads; the 95 percent range around it runs from 2.9 to 14.7 percent, wide enough to contain both the worst and the best channel in the table. A bench that printed 6.7 without the range would be telling you something it does not know.

CLOSE RATE AND ITS 95% RANGE, 2018 COHORT overall 13.0% Organic · 255 deals Paid · 175 deals Social · 66 deals Email · 14 deals Referral · 24 deals Display · 5 deals 14.7% 14.8% 6.3% 4.0% · thin 11.4% · thin 6.7% · very thin 0% 5% 10% 20% Pale bars rest on fewer than 30 deals. Display's range covers almost the whole table.
🔍 Click to zoom - a close rate is a dot with a width, and the width is the number of deals
LiveThe revenue column that is mostly zeros3 min▶

The dataset has a field called declared monthly revenue on each closed deal. It would be the obvious source for payback. It cannot be used: 797 of the 842 closed deals declare 0, and the 45 that declare something range from 6 to 50,000,000 reais. The median of all closed deals is zero. The table on the bench shows the counts per channel, "12 of 255 declared" for organic, so you can see why.

So payback uses one stated assumption for every channel: $80 a month of revenue per new seller at a 65 percent gross margin, $52 a month of gross profit, and payback is CAC divided by 52. Because the assumption is the same everywhere, payback ranks the presets exactly as CAC does. Session 5 is where revenue per customer becomes the subject rather than a constant.

Self-studyWhy a break button, and how it works3 min▶

A bench that can only ever report the answer you hoped for measures nothing. The break button takes the 781 closed flags and deals them back out across the 5,998 leads with a fixed random seed, so the number of deals stays 781 but which channel they land in is now arbitrary. If the bench's ranking came from the channels, the ranking should collapse. It does: lead-weighted and deal-weighted tie at $316, because with conversion uniform, volume is the right signal after all.

It also moves the "best split" toward the cheap-lead channels, social and email, which is what you would expect once every channel converts alike: then cost per lead is the only difference left. That is the mirror image of step 6, and the reason step 6 loses on the real data.

Real world

The 30-deal flag is a convention, not a law. The bench marks any channel with fewer than 30 closed deals and prints the 95 percent range beside every rate. There is nothing magic about 30; what matters is that the range is printed. Referral's 11.4 percent could be anywhere from 7.8 to 16.4. Before moving a budget on it, run the cheap test session 3 described and collect more deals.

Homework

Before session 5

30 min
  1. Pull your own lead table: leads and closed deals by channel for one cohort whose deals have had time to close. Check when your deal log starts before you trust any rate.
  2. Compute each channel's share of leads and share of deals. Circle the channel where the two differ most. That channel is where "most leads" costs you money.
  3. Print the 95 percent range for every rate built on fewer than 30 deals, and write "too thin to move budget on" beside it.
  4. Write down your real cost per lead for two channels, or your best guess with its source. Session 5 turns CAC into payback and lifetime value, and it needs your numbers, not the bench's.
Check yourself

Three questions

Clear all three and session 4 gets stamped in your passport.

1 · The "most leads" preset buys 488.8 leads and the deals-weighted preset 436.9. Why does the second still have the lower CAC?

Both presets spend the same $20,000. CAC is budget over deals, so the preset with more deals wins. Leads are an input; a split that buys more of them from a weak closer can end with fewer customers.

2 · After pressing "Break it", most leads and deals-weighted both print $316. What does that prove?

A break button is a test of the instrument. If shuffling the very thing a preset relies on removes its advantage, the advantage was coming from that thing and not from an artifact of the bench.

3 · Display closes at 6.7 percent on 5 deals, range 2.9 to 14.7 percent. What is the right budget decision?

Five deals cannot rank a channel. The range is the honest statement of what the data knows, and it spans most of the table. The decision it supports is to buy information cheaply, not to move a budget either way.

Sources covered

What this session teaches, and where it came from

✓Marketing Funnel by Olist (Kaggle, CC BY-NC-SA 4.0)All 8,000 leads and 842 closed deals shipped in the page; every count, rate and range computed in your browser and matched to a Python reference run
✓The deal log's start date and the 2018 cohortMeasured: closed-deal records begin 5 December 2017; 2017 leads close at 3.0 percent against 13.0
◐Cost per lead, saturation, revenue and marginConstructed, printed on the widget and editable; the pages claim the ordering of the presets, not their dollar values
✓The 95 percent range on a close rateWilson score interval, the standard interval for a proportion; the 30-deal flag is a convention
◐The field definitionsKaggle publishes no description of "origin" or "declared monthly revenue"; the page says only what the values show

Every fact on this page, and its verification tier, is recorded in the course's source map.

Session 4 cheat sheet · pin this

CAC = budget / dealsNever budget over leads. Cost per lead is an input, not a result.
Two multipliersCost per lead (rises as you push) and close rate (measured per channel).
Share of leads vs share of dealsWhere they differ, "most leads" costs money. Social here: 17.5 vs 8.5.
Cheapest lead is the bigger trap$627 against $368 on this data: 70.4 percent above best.
Print the rangeUnder 30 deals, a rate is a guess. Display: 2.9 to 14.7 percent.
Check the log's start dateA deal log that starts late makes early leads look like they never close.