The whole track, in one paragraph
Session 1 chose the minute as the grain and built a 7,200-row table. Session 2 put four recordings on one clock. Sessions 3 and 4 filled the content columns from the camera and the transcript, and session 4's claims checker counted the stretched urgency runs. Session 5 scored every minute in orders per thousand viewer-minutes, and session 6 separated what moves the money from what only rides the traffic, with pooled and within-stream models side by side. Session 7 built the nowcast and backtested it over 6,720 windows. Tonight the nowcast goes into a policy, the policy goes onto a bench, and the bench result goes into a test design. The brand is still Lumen Glow, the stream is Jules on a Friday at 20:00, and every number below is computed in this tab.
★ What you walk out with tonight. A bench that runs seven coaching policies over forty seeded nights in the browser, from the same engine that made the 60 streams, and prints a ladder: the host's own plan at $18,198, the playbook at $23,166, and two traps that the room asks for, one worth $92 and 3.18 policy-risk flags a night, the other costing $2,033. The shape of a policy you can write yourself in twelve lines and bench in one call. And a switchback design, with blocks, washout, metric and clustering, that takes the bench's best rule into a real TikTok Shop room without confounding it with the feed.
LiveWhat a policy is here, and why that shape is the honest one3 min▶
In the engine a policy is fn(t, st, m): the minute t, the state so far st (every row up to t-1, the current urgency run), and m, the minute the host's own script was about to play (its segment, its shot). It returns an override object, such as {seg: "cta", inHand: 1, priceOverlay: 1}, or null. The engine merges the override into the minute before it draws the room's response.
That shape is not a convenience. It is the constraint that makes the bench mean something. A policy sees the past and the plan, never the future and never the dice. It cannot know that minute 47 would have converted anyway; it can only see that minutes 44 to 46 came in under the nowcast. A real producer with an earpiece has exactly the same information, so a policy that wins on the bench is a policy a producer could actually run. Any analysis that peeks (choosing the cue minute after seeing which minutes sold) would win on the bench and lose in the room.
The first version of Lumen Glow's coaching rule was written from the session 6 regression: "cue a CTA in the minutes with the highest predicted orders". It benched beautifully. It was also unusable, because the prediction used the room size at the end of the minute, which the producer does not have until the minute is over. Rewriting it to use only rows up to t-1 cost about a third of the lift and made it real.
Self-studyThe vocabulary tonight adds4 min▶
- Policy - a function of the minute, the room so far and the host's plan that returns an override or nothing. Seven live in
LIVE_ENGINE.POLICIES. - Override - the fields a cue changes in the minute about to play: segment, product in hand, close-up ratio, price overlay, replies, giveaway.
- Seeded night - one run of tonight's stream from a fixed seed. Same seed, same dice, so two policies on one seed differ only by the cues.
- Ladder - every policy benched over the same nights, sorted by GMV, read against the host's own plan.
- Carryover - the effect of a minute leaking into the next: the one-minute ledger lag from session 2, and the urgency hangover that lowers conversion for three minutes after a stretched run.
- Washout - minutes at the start of each block dropped from the analysis so carryover from the previous block does not land in this one.
- Switchback - an experiment that alternates treatment on and off in time blocks within one unit (here one stream), so the comparison is within the same night and the same feed.
- Bandit - an algorithm that chooses among cues online, spending some minutes exploring and most exploiting the current best. Named tonight, not built.
Seven policies, forty nights
9 min liveThe engine ships seven policies: the host's own plan, three cues the leader track recommends, two the room asks for when it gets nervous, and the three good cues combined. Each is benched over forty seeded nights of the same stream. The ladder below is the canon; the widget in the build-along recomputes it in this tab.
LiveHow each policy is written4 min▶
Every policy in LIVE_ENGINE.POLICIES is a few lines. Reading them is the fastest way to know exactly what the ladder measured, because the label is a summary and the function is the spec.
| Policy | The logic, as written | Override returned |
|---|---|---|
none | fn is null. Jules runs the script as written. | nothing |
ctaOnDip | From minute 12. Take the three-minute nowcast from rows up to t-1. If the last 3 minutes ordered under 70 percent of it, and the last CTA is 6+ minutes back: fire. | seg is price if the last price line is 6+ minutes back, else cta; inHand: 1, priceOverlay: 1 |
showOnCta | Whenever the host's own minute is a cta or a price minute. | inHand: 1, closeUp: 0.8, priceOverlay: 1 |
replyEvery10 | Every tenth minute, unless the host's own minute is a CTA. | seg: "qna", replies: 6 |
stretchFomo | After minute 10. If the last minute was urgency and the run is under 6, keep going while the room sits below 97 percent of its 8-minute high. Otherwise start a run on three consecutive shrinking minutes. | seg: "fomo" |
giveawayOnDrop | After minute 10, on three consecutive shrinking minutes. | seg: "hook", giveaway: true |
playbook | replyEvery10, else ctaOnDip, else showOnCta. First non-null wins. | whichever fired |
Two details that matter for the numbers. The dip rule uses a three-minute nowcast horizon for the comparison and the widget's tile shows the five-minute one; both come from the same function. And stretchFomo is capped at six consecutive minutes, which is about as long as a host can say "almost gone" before even they stop believing it, so its 3.18 flags a night is a floor for what a real host in a real panic would produce, not a ceiling.
LiveWhy means over forty nights3 min▶
The bench seeds are written down: 424242 + s * 101 for s from 0 to 39. Night 0 is the one the replay panel shows; nights 1 to 39 differ only in the dice. Every policy is run on all forty and the summary means are reported. That is the whole method, and it is what LIVE_ENGINE.benchPolicy(name, 40) does.
One night is drawn. On seed 424242 a CTA at minute 43 lands on whatever the room happened to be at minute 43. The difference against the host's own plan on that seed is a real number and it is a story: it could be twice the mean or negative. Forty nights is measured, because the same rule fires at whatever minutes its trigger fires on across forty rooms, and the mean of the difference is the rule's value with the dice averaged out. The widget shows both and labels which is which: the replay's final line says "on this seed", the forty-night button says "mean of 40 seeded nights", and the ladder says "means" in its header.
The seeds are the reproducibility. Anyone with the engine file can recompute the ladder to the dollar. That is why the canon in data/canon.json carries builtFrom: assets/live-engine.js and the night count, and why a number quoted from this page can be checked rather than trusted.
LiveReading the traps3 min▶
Stretched urgency moves the number the host watches. Watch time goes from 9.63 to 9.84 minutes, the room holds, and GMV moves from $18,198 to $18,290: $92 a night, well inside the spread of a Friday. What it also buys is 3.18 policy-risk flags a night, the session 4 counter firing on every fourth consecutive urgency minute. On the ladder it is the row where the metric a nervous host optimises and the metric the business is paid in point in different directions.
The giveaway on a drop moves views up, 8,732 from 8,519, and GMV down by $2,033 to $16,165. The engine's giveaway minutes multiply arrivals and engagement and cut the conversion rate during them and for the rest of the stream after them; that is the session 2 finding placed mid-stream, where it lands on people who were about to buy. Orders fall from 544 to 482. The ladder row reads as a bigger room with an emptier till, which is exactly what the seller dashboard would show the next morning, as views up and GPM down.
The first time the ladder was shown to Lumen Glow's hosts, one of them asked why the two red rows were on it at all. The answer was that they were the two rules the hosts were already running, unwritten. Writing them down as functions was the only way to bench them, and benching them was the only way the conversation stopped being about feelings.
Self-studyWhat the bench cannot measure4 min▶
Two things are outside the engine by construction. The first is the feed's reaction to a flagged account. The bench counts flags; it does not model what the platform does with them. A LIVE stamped Risk on the diagnosis page loses future traffic, and the size of that loss is the platform's and not in any recording. So the stretched-urgency row's $92 is an overstatement: the true value of that rule is $92 minus whatever three flags a night cost in next week's rooms, and the sign of that is not in doubt.
The second is a real room's memory of last week. The engine draws each night fresh from the seed. A real audience that was promised "last chance" on Friday and saw the same serum at the same price on Monday remembers, and its conversion on the next urgency line is lower than the engine's. Followers who came back for a giveaway that did not repeat are also not modelled. Both effects make the traps worse in reality than on the ladder, and neither makes the good cues better, which is the direction of error you want a rehearsal to have.
What the bench also cannot do is prove anything about a room it has not seen. That is the point of Part 2.
From the bench to a real room
8 min liveThe bench says the playbook is worth 27 percent on a generator whose truth is known. A real TikTok Shop room's truth is not known, and a cue's effect there is confounded with the feed, the hour and the host, exactly the way the giveaway opener was in session 6. The fix is a design that compares on and off within the same night: the switchback.
LiveThe design3 min▶
A switchback alternates treatment on and off in time within one unit, so the unit is its own control. For a LIVE the unit is the stream and the time is blocks of minutes. The design, in the order you have to decide it:
- Blocks. 10 minutes. Long enough that a CTA cued in the block converts in the block, given the one-minute ledger lag and the fact that the dip rule needs three minutes of evidence before it fires.
- Randomisation. Each block is on or off by a coin flip made before the stream, sealed, and read by the producer at the block boundary. Not alternating, because a host learns an alternating pattern by the third block.
- Washout. The first two minutes of every block are dropped from the analysis: one for the ledger lag from session 2, one for the room to settle. If the on arm includes any urgency line, three, because the engine's urgency hangover lowers conversion for three minutes after a stretched run, and a real room's is not shorter.
- Metric. Orders per 1,000 viewer-minutes in the block, the session 5 rate, because it puts the room in the denominator and a block with a small room does not get a 67 percent conversion from two orders.
- Clustering. Blocks within a night are not independent. Standard errors cluster by night, so the effective sample size is the number of nights, not the number of blocks.
- How many nights. Enough that the clustered interval excludes zero if the bench's effect size is real. Compute it from the within-stream residual variance from session 6, not from a rule of thumb.
The design and its analysis are laid out in Bojinov, Simchi-Levi and Zhao (2023), "Design and Analysis of Switchback Experiments", Management Science 69(7), which is where the carryover and block-length reasoning comes from. The randomisation theory behind all of it lives in a separate course: learn-experimentation-with-phoebe.
LiveThe analysis3 min▶
The analysis is the session 6 within-stream regression with one new column. Stream fixed effects absorb the night, the host and the feed; the treatment column carries the block assignment; standard errors cluster by stream. The coefficient on treated is the lift in orders per 1,000 viewer-minutes.
Weighting by the room is what stops a two-order block in a room of forty from counting like a twenty-order block in a room of eight hundred. Why the block length must clear carryover: if a stretched urgency run in an off block leaves a three-minute hangover and the next on block is only five minutes long with two of washout, three of the on block's five analysed minutes are still paying for the previous block's pitch, and the on arm is penalised for a cue it did not give. Ten-minute blocks with two or three minutes of washout leave seven clean minutes, which is why ten and not five.
LiveBandits, as the next step2 min▶
The switchback answers one question: is the playbook better than no playbook. The next question is which cue to fire when several qualify at the same minute, and that is an online choice, not an offline test. The tool for it is a bandit: an algorithm that plays each cue, reads the response, and shifts its choices toward the one that is paying, while still trying the others often enough to notice if the room changes.
- MABWiser 2.7.4 - context-free bandits with UCB1 and Thompson sampling, a clean fit for "which of three cues at a dip". The arms are the cues, the reward is the block's orders per 1,000 viewer-minutes.
- vowpalwabbit 9.11.6 - contextual bandits, where the context is the room so far (the nowcast gap, minutes since the last CTA, the urgency run). The policy becomes a learned function rather than a hand-written one.
Named, not built. A bandit run before the switchback has proven the cues work is a bandit optimising among options nobody has shown to be positive, and its exploration minutes are real minutes in a real room. Prove the playbook first.
Self-studyReading the POLICIES block as the spec of the test4 min▶
Open assets/live-engine.js at the POLICIES object and read it as the protocol of the switchback. Every decision the producer has to make in an on block is already written there: what counts as a dip (three-minute orders under 70 percent of the three-minute nowcast), how long since the last CTA before another is allowed (six minutes), what the CTA minute looks like (serum in hand, price on screen), when a Q&A minute happens (every tenth minute, never over a CTA). The playbook function's priority order (reply, then dip, then shot) is the tie-break rule when two triggers fire in the same minute.
That is what makes the bench a rehearsal rather than a decoration: the on arm in the real room runs the same function a producer reads off a card, and the off arm is none. If the real-room lift comes in near the bench's, the engine's planted structure was close to the truth. If it does not, the gap is itself a finding about what the engine is missing, and the two lists in "What the bench cannot measure" are where to look first.
The replay bench
★ 20 min · everyone buildsThe bench below is live-bench.js mounted on this page. Top panel: tonight's stream replayed from seed 424242, the nowcast beside the truth, cues at the current minute. Bottom panel: the seven-policy ladder over forty nights, computed when the page loaded. Twenty minutes: run it, then write a policy of your own and bench it from the console.
Run the replay with no cues to the end and note the single-night GMV. Press Play and let it run to minute 120. The final line reads the GMV on this seed and a difference of zero against the host's own plan, because with no cues the coached run is the plan. Write the GMV down. It will not equal $18,198: that is a mean, and this is one night.
Cue a CTA at the first dip and run forty nights. Reset, play, and watch the "nowcast, next 5 min" tile against what the engine actually produces. When orders come in under the forecast for a few minutes, press "Cue a CTA". Play to the end, read the single-night difference, then press "Run this exact plan over 40 nights". The mean delta is the number to keep. Compare it with the ladder's ctaOnDip row, and notice why they differ: the ladder's rule fires at every dip on every night; your cue fires at one minute on all forty.
Write your own policy and bench it. In the browser console, add a policy to the engine and call the bench on it. Anything that reads st.rows only up to the current minute is fair.
Then score it against the published canon in pandas. The canon file carries the ladder means and the seed count, so your policy's row can sit on the same table as the seven shipped ones.
Write the switchback plan as a table. One row per design decision, with the value and the reason, so a producer and a finance director can both read it.
The honesty rail: which numbers here are computed, which are planted. Write it on the page that carries the ladder. Every GMV, order count, watch time and flag count in the widget is computed in this tab by live-engine.js from the seeds. The causal structure those numbers come from (that a stretched urgency run halves conversion, that a giveaway multiplies arrivals and cuts conversion, that the serum in hand lifts clicks) is planted in the engine's TRUTH object, openly. The ladder is therefore a test that the policies recover a known truth, not evidence about a real room. The switchback is where evidence comes from. A number quoted from the bench without that sentence next to it is a number that will be misread.
Read the engine before you trust the ladder. POLICIES is under two hundred lines from TRUTH, and the two together are the complete specification of everything on this page. If a row on the ladder surprises you, the reason is in one of those two blocks, and it is faster to read it than to reason about it.
After the capstone
45 min- Write a policy that beats the playbook's $23,166 over forty nights without adding a flag. Bench it with
benchPolicy, then read its function back and ask whether a producer with an earpiece could run it. If not, it is a leak: it saw something the producer cannot. - Bench
stretchFomowith the six-minute cap raised to twelve, by copying it under a new name. Report GMV, watch time and flags. Write one sentence on what a host in a real panic would produce. - Fill in the switchback plan's last row. From session 6's within-stream residuals and the playbook's bench effect, compute how many nights the clustered interval needs to exclude zero. Show the arithmetic.
- Read the leader track's session 6 scorecard and mark which of its rows the switchback can estimate and which it cannot. Bring the list to whoever owns the hosts.
Three questions
A policy in the engine is fn(t, st, m) and st.rows holds minutes 1 to t-1 only. Why is that restriction the point rather than a limitation?
The bench is a rehearsal for a real earpiece. Choosing the cue minute after seeing which minutes sold is the leak that made Lumen Glow's first rule unusable.
On the ladder the giveaway-on-a-drop policy shows views of 8,732 against 8,519 and GMV of $16,165 against $18,198. How do both happen at once?
It is the session 2 finding placed mid-stream. GPM falls faster than views rise, so GMV, which is views times GPM, falls. Nothing is capped; the till is emptier because the people are different.
In the switchback design, what is the two-minute washout at the start of each block for?
Carryover is the switchback's specific weakness. Washout removes the contaminated minutes; the block length must still be long enough that what is left is clean, which is why ten minutes and not five.