LIVE Commerce Analytics / Builder session 5 of 8
Learn LIVE Commerce Analytics with Phoebe · Builder session 5 of 8

Score the content

Sessions 3 and 4 gave every minute a picture and a label. Tonight every label gets a number, and the number has to survive the smallest denominator in the table. Click rate and click-to-order rate are honest at the stream grain and liars at the minute grain, because a minute has a handful of clicks. The fix is a denominator that is never small: the room. Orders per thousand viewer-minutes, summed the right way, gives you a nine-row table a leader can read aloud, a rule for every ratio you will ever compute on minute rows, and two features (how long a segment has been running, and what happens one minute later) that session 6 cannot do without.

🟠 Advanced Data scientists Anyone who has averaged a ratio and regretted it 45 min
0-3 · Welcome 3-20 · Rates at the minute grain 20-40 · Build-along: the segment table 40-45 · Q&A
Part 0

Where session 5 sits in the pipeline

The minute table is complete: 7,200 rows of Lumen Glow, one per stream and minute, with what was said, what was shown and what was sold. Tonight is the first session that reads it instead of building it. Session 6 will ask which of tonight's numbers survive holding the traffic still; session 7 will forecast the next five minutes from them; session 8 will cue a host with them. So the numbers built tonight are the ones the rest of the track stands on, and the point of the session is that they stand on the right denominator.

Live = presented in the 45 minutes Self-study = the full depth, after class 🔍 Every diagram zooms

★ What you walk out with tonight. The segment table: orders per 1,000 viewer-minutes for all nine labels, from CTA at 11.57 down to dead air at 0.71, with the minutes share and the click and order rates beside each. A rule for every ratio you compute at the minute grain, and the two answers it separates. A run-length column and a lagged-orders column ready for session 6. And a clear reading of what the AI-host rows show at this grain, before session 6 says what they mean.

LiveWhy per-minute CTR and CTOR lie3 min▶

At the stream grain the funnel ratios are fine: an average stream has 3,997 clicks and 484 orders, and 12.1 percent is a stable number. Divide the same stream into 120 minutes and each minute has a few dozen clicks and a few orders. A minute with 3 clicks and 2 orders has a click-to-order rate of 67 percent. A minute with 1 click and 0 orders has a rate of zero. Neither number means anything, and averaging 7,200 of them means nothing with more decimals.

The cure is not a bigger sample; it is a bigger denominator. Every minute has a room, and the room is never small: even a quiet stream holds hundreds of viewers a minute. So the rate that is honest at the minute grain is orders divided by the room, scaled to orders per 1,000 viewer-minutes. Across the whole table it averages 5.80. Its numerator is still small, but its denominator is stable enough that averaging over a segment's minutes lands in the same place whichever way you sum it, which is the property the click-based ratios lack.

Real world

A Lumen Glow weekly report once ranked hosts by their "average minute conversion", the mean of orders over clicks across each host's minutes. The winner was the host with the most minutes of two clicks and one order. The leaderboard flipped when the ratio was rebuilt as orders per thousand viewer-minutes, and flipped back to the stream-level GPM ranking everyone already trusted, which was the first sign the new number was right.

Self-studyThe vocabulary tonight uses3 min▶
  • Viewer-minute - one viewer watching for one minute. The sum of the room column over a stream is its viewer-minutes; dividing by views gives average watch time, as session 1 showed.
  • Weighted mean - a ratio's average across rows computed as the sum of numerators over the sum of denominators. Rows with a bigger denominator count for more, which is the point.
  • Small-denominator bias - what an unweighted mean of per-row ratios does: the rows with the fewest clicks contribute the most extreme values and the same weight as everyone else.
  • Lag - a column shifted in time within each stream. orders_next1 at minute 40 is the orders at minute 41. Session 6's regressions need it because an order takes a moment to place.
  • Run length - how many consecutive minutes a segment has been running, counting the current one. The fomo_run column session 4 built.
Part 1 · covers the segment table and the weighting rule

The segment table

8 min live

Nine labels, one row each, three rates and a minutes share. It is the most-quoted table in the course: the leader track's script card is read off it, and session 6's regressions start from it. Read it with the denominator in mind.

Orders per 1,000 viewer-minutes by segment, 60 streams, 7,200 minutes bar = orders per 1,000 viewer-minutes · table mean 5.80 · the three right-hand columns are the same minutes on other denominators Segment Orders per 1,000 viewer-minutes Minutes CTR CTOR CTA 11.57 9.9% 9.61 16.73 demo 7.72 20.9% 9.41 11.38 price 7.69 6.6% 7.85 13.62 urgency 5.83 16.4% 6.20 13.06 build 4.51 19.0% 5.83 10.39 Q&A 3.68 9.1% 4.35 11.76 hook 1.98 12.8% 3.86 7.11 filler 1.40 4.6% 2.76 7.06 dead 0.71 0.8% 1.93 5.07 A CTA minute is worth about six hook minutes per viewer. The hook is 12.8% of the show; it is where the room is built.
🔍 Click to zoom - the nine segments on the denominator that is never small, with the click-based rates beside them.
LiveReading it3 min▶

Three things the table says, in the order a leader will hear them. CTA minutes sell. 11.57 orders per thousand viewer-minutes, against a table mean of 5.80: twice the average minute, and about six times a hook minute at 1.98. Price and demo are the next tier, 7.69 and 7.72, close enough that the difference is not worth a script line. The bottom three are not failures. The hook orders at 1.98 because nobody buys in the first minutes of a show; they arrive. It is 12.8 percent of all minutes and the room curve is built there. Filler at 1.40 and dead air at 0.71 are the two rows a host can actually remove.

Now read the right-hand columns with the denominator in mind. CTOR for CTA is 16.73 and for dead air 5.07; the ranking mostly agrees with the bar, which is reassuring, but the click-based rates are computed on a few clicks a minute and they wobble. The bar does not. When a per-segment CTOR and a per-segment orders-per-viewer-minute disagree on a small segment, believe the bar.

SegmentOrders per 1,000 viewer-minTimes the mean (5.80)What it is for
CTA11.572.0Converting the room that is already there
price7.691.4Removing the last objection
demo7.721.3Showing, which also moves clicks (session 3)
urgency5.831.0Two good minutes, then a cliff (Part 2)
hook1.980.3Building the room, not selling to it
filler, dead1.40, 0.710.2, 0.1Nothing. The removable minutes
LiveThe weighting rule4 min▶

To average a ratio across minutes, sum the numerators and sum the denominators, then divide. Never average the per-minute ratio. That is the whole rule, and it is the difference between a number the dashboard would agree with and one it would not.

Watch the two answers separate on a toy pair. Minute A: 3 clicks, 2 orders, rate 67 percent. Minute B: 300 clicks, 30 orders, rate 10 percent. The mean of the two rates is 38.5 percent. The pooled rate is 32 orders over 303 clicks, 10.6 percent. The dashboard, which counts orders and clicks for the whole stream, reports the second. The first is a number about minute A's three clicks, dressed up as a number about the stream.

The rule holds for every ratio in the table. It is why orders per thousand viewer-minutes is computed as sum(orders) over sum(room) within a segment, and why CTR and CTOR in the segment table should be read as pooled clicks over pooled impressions and pooled orders over pooled clicks. When the denominator is the room, the pooled and the averaged versions land within a tenth of each other, because the room is never small; that closeness is the test that the denominator is safe. When the denominator is clicks, they can separate by a factor of three on a small segment, and the pooled one is right.

Keep numerator and denominator as columns until the last line. A DataFrame with orders and room can be re-grouped by segment, by host, by run length, by anything, and every regrouping is correct. A DataFrame with only opv can be regrouped only wrongly.

LiveThe example that made it stick2 min▶
Real world

Maya's stream 17, a Wednesday at 21:00 with a giveaway opener: 16,404 views, peak room 1,579, 6.86 CTR, 11.58 CTOR, GMV $29,799. Priya's stream 16, a Monday at 18:00 with no giveaway: 4,858 views, peak room 498, 6.90 CTR, 14.01 CTOR, GMV $11,217. Per minute, Priya's stream had a fraction of the clicks, so its per-minute CTOR was all over the place and its unweighted mean looked worse than Maya's. Pooled, Priya converted better: 14.01 to 11.58, and her GPM was 2,309 to Maya's 1,817. The stream with a third of the room sold the room it had better. Only the weighted number could say so.

Self-studyThe chat is where the room talks back3 min▶

Add a fourth column to the table: comments per 100 viewers, computed the same weighted way, sum of comments over sum of room. Q&A minutes run at 3.6; every other segment sits between 1.6 and 2.0, with filler and dead air below one. The Q&A minute is the only kind in which the room speaks at nearly twice its normal rate, and it is also the segment with plus 0.96 points of click-to-order within a stream: not a selling minute, but the minute the room decides to stay.

Two things follow. The comment column is a denominator problem too: comments per minute is a count that grows with the room, and only per-viewer is comparable across segments. And the AI-host rows, coming up in Part 2, have no Q&A minutes at all, which this column predicts will show up in the room curve before it shows up in orders.

Part 2 · covers run length, the one-minute lag and the AI-host rows at this grain

Runs, lags and the urgency cliff

9 min live

A segment's value is not a constant. It depends on how long the segment has already been running, and it arrives a minute late. Both facts become columns tonight, because session 6 will regress on them and session 8 will cue a host with them.

CTOR by consecutive urgency minute, pooled over 60 streams x = how many urgency minutes in a row so far (fomo_run) · y = click-to-order rate in that minute, percent 0 10 20 15.12 run 1 15.44 run 2 8.30 run 3 8.33 run 4 8.43 run 5 9.09 run 6 the cliff: minute 3 soft flag here, policy risk from 4 Scarcity is worth two minutes. From the third the rate nearly halves and never recovers within the run.
🔍 Click to zoom - the urgency cliff at the third consecutive minute, and why run length is a feature and not a footnote.
LiveRun length as a feature3 min▶

Urgency minutes as one row in the segment table score 5.83 per thousand viewer-minutes, a hair above the mean. That single number hides two very different minutes. In the first and second minute of a run the click-to-order rate is 15.12 and 15.44 percent, better than every segment except CTA. From the third it is 8.30, then 8.33, 8.43, 9.09: the room has heard "almost gone" three times in three minutes and stopped believing it. Session 6 will price the two halves separately, plus 4.93 points within a stream for short urgency and minus 1.84 for stretched, and the sign flips only because the run length was a column.

So the fomo_run column session 4 built is not a diagnostic; it is a feature. A segment's value depends on how long it has already been running, and the only way a model can learn that is if the table says how long. The same logic applies more gently to CTA (a fourth consecutive CTA minute is a host who has run out of things to say) and to Q&A, but urgency is where the cliff is sharp enough to draw.

LiveThe one-minute lag2 min▶

A CTA at minute 40 sells at minute 40 and at minute 41. Placing an order takes a tap, a size, a checkout, and for a fair share of viewers that crosses the minute boundary. If the regression in session 6 only looks at same-minute orders it under-credits every segment and mis-credits whatever segment came next. So tonight builds orders_next1, each minute's orders shifted back by one within the stream, and session 6 uses the sum of the current and the next minute as its outcome.

The shift must be within the stream. A plain shift(-1) on the whole table hands the last minute of stream 16 the first minute of stream 17, and the first minute of a stream is a hook with a room of near nothing. The last minute of every stream gets a missing value, which is correct: nothing was observed after it.

LiveWhat the AI-host rows show at this grain3 min▶

Two of the six hosts are AI avatars, Nova and Kai. At the minute grain their rows are unusual in three ways that the stream-level comparison (GMV 8,132 versus 19,850 for humans) cannot see. There are no Q&A minutes. The tagger never fires the name-plus-question rule because the avatar never reads a name. Replies are zero, every minute of every AI stream, against 95.6 per human stream. And the room fades by script cycle: the average room across the eight cycles of the script runs 216, 442, 484, 468, 441, 413, 383, 366 for the AI hosts and 419, 856, 961, 961, 935, 880, 847, 790 for the humans. The AI room peaks lower, peaks earlier, and gives back more of its peak.

What the table cannot say is why. The AI streams also sit mostly outside the evening (26 percent of them in the evening against 95 percent of human streams), and traffic follows the hour. Session 6 holds the hour still and finds the gap survives it. Before that, one fact sits outside the data entirely: TikTok Shop's requirements for LIVEs, in force since 2026-09-17, say "Do not use non-real-time verbal interaction such as AI-generated voices, audio recordings, or radio", so on that platform a fully AI-voiced avatar host is a compliance stop before it is an analytics question. The dataset's AI rows show what the trade would be on a platform that allows it.

At the minute grainHuman hostsAI hosts
Q&A minutes9.1% of all minutes are Q&A across the tablenone
Host replies per stream95.60
Room by script cycle 1 to 8419, 856, 961, 961, 935, 880, 847, 790216, 442, 484, 468, 441, 413, 383, 366
Average watch9.80 min7.28 min
Self-studyWhat this table cannot say yet3 min▶

The segment table is pooled across 60 streams. That is its strength for a leader and its weakness for a scientist. A segment that appears more often in high-traffic streams borrows those streams' room, and orders per viewer-minute is a rate, so it borrows their conversion too: a segment that good hosts use more will look better than it is, and one that weak hosts lean on will look worse. Urgency is the obvious case: the hosts who stretch it are not a random sample of hosts.

The fix is to compare minutes within a stream, subtracting each stream's own mean before asking what a segment does. Session 6 does exactly that, with stream fixed effects and clustered standard errors, and its numbers (CTA plus 6.50 points of CTOR within a stream, short urgency plus 4.93, stretched minus 1.84) are the ones that survive the confound. Tonight's table is the map; session 6 checks the map against the territory. Read tonight's numbers as "what the minutes did", never yet as "what the segment causes".

Build-along

Build the segment table

★ 20 min · everyone builds

pandas only, the 7,200-row minute table in, a nine-row table and two new columns out. You compute the rate the right way, then the wrong way on purpose so you recognise it in someone else's notebook, then the run-length and lag features session 6 needs.

Orders per viewer-minute, then both ways of averaging it. Per minute first, so the column exists for session 7's nowcast. Then per segment as the mean of the per-minute rate, and as the pooled sum over sum. Keep both; the gap between them is your denominator test.

★ Numerator and denominator stay as columns import numpy as np import pandas as pd m = pd.read_csv("data/streams.csv") m["opv"] = np.where(m.room > 0, 1000 * m.orders / m.room, np.nan) # per minute g = m.groupby("segment") seg = pd.DataFrame({ "minutes": g.size(), "share": 100 * g.size() / len(m), "opv_mean": g.opv.mean(), # mean of per-minute rates "opv_pool": 1000 * g.orders.sum() / g.room.sum(), # sum over sum "ctr": 100 * g.clicks.sum() / g.impressions.sum(), "ctor": 100 * g.orders.sum() / g.clicks.sum(), }).sort_values("opv_pool", ascending=False) seg.round(2)

Reproduce the segment table and assert the CTA row. The published table is the pooled version, sum over sum. The mean-of-rates version lands within a tenth of it on every row here, because the room is never small; on a table where it is, the two diverge, and that gap is the actual lesson. Assert the CTA value to two decimals, and assert the gap.

★ Two asserts: the number, and the reason the number is safe assert round(float(seg.loc["cta", "opv_pool"]), 2) == 11.57 assert (seg.opv_mean - seg.opv_pool).abs().max() < 0.2 # a stable denominator: both ways agree order = ["cta", "demo", "price", "fomo", "build", "qna", "hook", "filler", "dead"] seg.loc[order, ["share", "opv_pool", "ctr", "ctor"]].round(2)

Now the wrong way on purpose. The same table with click-to-order rate as the mean of the per-minute ratio, minutes with no clicks dropped, beside the pooled version. Pooled over 7,200 minutes the two are close; group by stream and segment, where a segment can be a handful of minutes, and watch the columns separate on the small rows.

✗ The mean of a per-minute ratio, so you know it when you see it m["cvr_min"] = np.where(m.clicks > 0, 100 * m.orders / m.clicks, np.nan) # 67% on 3 clicks, NaN on 0 by_ss = m.groupby(["stream_id", "segment"]) wrong = by_ss.cvr_min.mean() # unweighted mean of a noisy ratio right = 100 * by_ss.orders.sum() / by_ss.clicks.sum() # pooled cmp = pd.DataFrame({"n_min": by_ss.size(), "mean_of_ratio": wrong, "pooled": right}) cmp["gap"] = cmp.mean_of_ratio - cmp.pooled cmp.sort_values("n_min").head(12).round(1) # the smallest groups, the widest gaps
★ The rule, as a function you reuse for every ratio def pooled_rate(df, num, den, by, scale=100): g = df.groupby(by) return scale * g[num].sum() / g[den].sum() pooled_rate(m, "orders", "clicks", "segment") # CTOR by segment pooled_rate(m, "orders", "room", "segment", 1000) # orders per 1,000 viewer-minutes pooled_rate(m, "comments", "room", "segment") # comments per 100 viewers: qna 3.6

The urgency run table. Click-to-order rate by fomo_run for urgency minutes only, with the minutes count beside it so the reader knows how much data sits under each step. Run 7 and beyond is a handful of minutes; do not draw it.

★ fomo_run by run length fomo = m[m.segment == "fomo"] runs = pd.DataFrame({ "minutes": fomo.groupby("fomo_run").size(), "ctor": pooled_rate(fomo, "orders", "clicks", "fomo_run"), "opv": pooled_rate(fomo, "orders", "room", "fomo_run", 1000), }) runs.loc[1:6].round(2) # 15.12, 15.44, then the cliff at 3

Lagged columns, within the stream. Orders and clicks one minute ahead, grouped by stream so the last minute of one stream never sees the first minute of the next. Session 6's outcome is the current plus the next minute.

★ orders_next1, clicks_next1 m = m.sort_values(["stream_id", "t"]) m["orders_next1"] = m.groupby("stream_id").orders.shift(-1) m["clicks_next1"] = m.groupby("stream_id").clicks.shift(-1) m["orders_2min"] = m.orders + m.orders_next1 assert m.orders_next1.isna().sum() == 60 # exactly one missing per stream: its last minute m.groupby("segment").orders_2min.mean().sort_values(ascending=False).round(2)

Save the derived tables. Next to the raw file, never over it. The segment table is the leader track's script card material; the minute table with its new columns is session 6's input.

★ Derived, never raw seg.loc[order].round(2).to_csv("data/derived/segment_table.csv") runs.loc[1:6].round(2).to_csv("data/derived/fomo_by_run.csv") m.to_csv("data/derived/minutes_with_lags.csv", index=False)

Test the rule on your own data before trusting the table. The assert in step 2 is not decoration. On a table where the room is sometimes tiny (a stream that started before anyone arrived, a feed that dropped for a minute) the mean of the per-minute rate and the pooled rate will separate, and the assert will tell you before a leader reads the wrong row aloud. When it fails, the pooled column is the one to publish.

Homework

Before session 6

30 min
  1. Rebuild the segment table for human hosts and AI hosts separately, both ways. Write one sentence about which segment's rank changes between the two groups and whether the mean-of-ratio or the pooled version would have told you.
  2. Compute the segment table per stream and count, for each segment, how many streams have fewer than five minutes of it. Those are the rows where a per-minute ratio is at its most dangerous.
  3. Build a CTA run table like the urgency one. Say at which run length, if any, a CTA stops paying, and how many minutes sit under that step.
  4. Regress orders_2min on the segment dummies, pooled, with plain OLS. Keep the coefficients. Session 6 runs the same regression within streams and you will compare the two sets.
Check yourself

Three questions

Why is the segment table built on orders per 1,000 viewer-minutes rather than on per-minute click-to-order rate?

The room is the denominator that is never small. The test is that the mean of the per-minute rate and the pooled rate agree within a tenth; on clicks they can differ by a factor of three on a small segment.

Across 60 streams, what happens to click-to-order rate at the third consecutive urgency minute?

15.12, 15.44, then 8.30, 8.33, 8.43, 9.09. The single-row figure of 5.83 hides it, which is why run length is a column and why session 4's checker raises a flag at run 3.

Demo minutes score 7.72 per thousand viewer-minutes in the pooled table. Why can that number still mislead about what a demo does?

Pooling mixes "what this segment does" with "who does this segment". Session 6 subtracts each stream's own mean first, and its within-stream numbers are the ones a leader can act on.

Sources covered

What tonight maps to

✓ Orders per 1,000 viewer-minutes by segment All nine labels from CTA 11.57 to dead air 0.71, with minutes share, CTR and CTOR beside each, reproduced in the build-along and asserted.
✓ The weighting rule for ratios at the minute grain Sum numerators, sum denominators, divide. The wrong way computed on purpose, and the agreement test that shows when a denominator is safe.
✓ Run length and lag as features The urgency cliff by fomo_run, and orders_next1 shifted within the stream.
◐ The confound: pooled versus within-stream Named tonight as the reason the table cannot yet say "causes". Session 6 runs the within-stream regression with stream fixed effects.
◐ The leader's script card Leader session 3 reads the segment table as a running order for a host: hook, build, price, CTA, two minutes of urgency at most.
· Order-level attribution Not covered by design. The table credits minutes, not viewers; which viewer saw which segment before buying needs a buyer-level log the recording does not contain.

Builder session 5 cheat sheet

The unitOrders per 1,000 viewer-minutes: 1000 x orders / room. Table mean 5.80. The denominator is never small.
The ladderCTA 11.57 · price 7.69 · demo 7.72 · urgency 5.83 · build 4.51 · Q&A 3.68 · hook 1.98 · filler 1.40 · dead 0.71.
The weighting ruleSum numerators, sum denominators, divide. Never average the per-minute ratio. Keep both columns until the last line.
The safety testMean-of-ratio and pooled agree within a tenth on the room; they can split threefold on clicks. When they split, publish the pooled one.
The urgency cliffCTOR by run: 15.12, 15.44, 8.30, 8.33, 8.43, 9.09. Two minutes, then stop.
The lagorders_next1 = groupby(stream).orders.shift(-1). Sixty missing values, one per last minute. Outcome for session 6 is this minute plus the next.
AI rows at this grainNo Q&A minutes, zero replies, room by cycle 216 to 484 to 366 against 419 to 961 to 790 for humans. Compliance stop on TikTok Shop first.
What it cannot sayPooled across streams, so a segment borrows the rooms of the hosts who use it. Session 6 compares within a stream.