Where session 4 sits in the pipeline
Session 1 fixed the grain, session 2 put every source on one clock, session 3 filled the picture columns: product in hand, close-up ratio, price overlay, scene cuts. Tonight fills the speech columns: segment, wpm and a claims flag. Session 5 scores every label at the minute grain, and session 6 asks which of the labels actually move the money once the traffic is held still. The recordings are still Lumen Glow, 60 streams of a 29 dollar serum, four human hosts and two AI avatars.
★ What you walk out with tonight. A word-timestamped transcript of a two-hour stream, with a start and an end on every word. A nine-label segment column per minute (hook, build, demo, price, CTA, urgency, Q&A, filler, dead air) from a tagger that runs keyword rules first and reaches for a model only on the minutes the rules cannot call. A words-per-minute column. And a claims checker that flags the exact phrases TikTok Shop's guides name, plus a run counter that knows when urgency has gone on too long to be either honest or effective.
LiveWhy the words are the column that moves orders3 min▶
Session 3 ended with a within-stream result on click rate: product in hand plus 1.38 points, close-up ratio plus 1.01, price overlay plus 0.97. Useful, and about the click. Run the same within-stream comparison on the click-to-order rate, in points, and the picture columns go quiet: product in hand plus 0.04, close-up plus 0.46, price overlay minus 0.13, none of them distinguishable from zero. The speech columns are the ones that speak: CTA minutes plus 6.50 points, price minutes plus 3.78, short urgency (its first two minutes) plus 4.93, Q&A plus 0.96. Hook minus 3.18, filler minus 2.64, dead air minus 4.72.
The picture gets the tap. The words get the order. That is the reason a segment label per minute is worth a whole session, and the reason its labels have to be defensible: session 6 will put them in a regression, and a leader will read the result as advice to a host.
Lumen Glow's first tagger was a single prompt to a language model with the whole transcript pasted in. It returned plausible labels, and the review with the host Jules fell apart in ten minutes: nobody could say why minute 61 was "demo" and minute 62 "build" when she had said nearly the same sentence in both. The rewrite put the keyword rules first, wrote the matching phrase into a side column, and only sent the minutes with no match to a model. The next review took a whole hour, because for the first time there was something to argue about.
Self-studyThe vocabulary tonight uses4 min▶
- Word timestamp - a start and an end time for each transcribed word, not just for each sentence. Needed because a minute boundary falls in the middle of sentences all night long.
- Forced alignment - snapping a known transcript onto the audio with a phoneme model so every word gets a precise time. whisperx does this with wav2vec2 after Whisper has produced the text.
- VAD - voice activity detection, the pre-pass that finds where anyone is speaking at all. It is also how you find dead air without a model.
- Diarisation - who spoke when. A co-host, a producer off camera and a played-in customer clip all end up in the audio; the tagger should only read the host.
- Segment label - the dominant thing the host is doing in a minute: hook, build, demo, price, CTA, urgency, Q&A, filler, dead. One per minute, as a column.
- Claims checker - a phrase list, matched by regex, that flags language a platform policy names as exaggerated or misleading. It flags; a person decides.
- VLM - a video-language model, one that takes frames and audio together and answers a question in text. The fallback for minutes where the words alone do not settle the label.
Words with a clock on them
8 min liveA transcript without word times is a document. A transcript with them is a table you can join to the minute grid, and that is the only kind tonight cares about. Three local routes and two managed ones produce it; they differ in licence, timestamp method, the unit the time comes back in, and what two hours costs.
LiveThe timestamp trap2 min▶
Every route hands back a list of words with a start and an end. Deepgram's words[] carry start and end in seconds. AssemblyAI's carry them in milliseconds. The three Whisper routes are in seconds. Nothing in the field name tells you which, and both are plain numbers, so a pipeline written against one vendor runs without error against the other and produces minute numbers a thousand times too big, or a whole stream that lands in minute 1.
The fix is one line at the boundary and a habit: convert to seconds the moment the response arrives, name the column start_s, and never let a raw vendor timestamp touch the minute join. The same discipline session 2 applied to the stream delay applies here: one clock, converted at the door.
Assert the unit. After loading any transcript, check that the last word's start is below the stream length in seconds, which for a two-hour LIVE is 7,200. A millisecond transcript fails that test at the first word past minute 2.
LiveWhat a two-hour stream costs each way3 min▶
Local routes cost GPU minutes and your time. The three Whisper packages are free to use: openai-whisper and faster-whisper under MIT, whisperx under BSD-2. faster-whisper is the same model family run through CTranslate2, which is why it is the one in tonight's build-along: same word_timestamps=True flag, less memory, and a batched pipeline for throughput. whisperx adds the forced-alignment pass, VAD and diarisation, at the cost of a heavier install.
Managed routes cost cents. Deepgram nova-3 is priced at $0.0043 per minute of mono audio, about $0.26 an hour, so a two-hour stream is about 52 cents. AssemblyAI Universal-3.5 Pro is $0.21 an hour, so about 42 cents. Both per-hour figures are course arithmetic from the published per-minute and per-hour prices. Sixty streams is a few tens of dollars either way. The cost that matters is not the transcript; it is the review time when the labels built on it are wrong.
| Route | What you pay | What you get that the others do not |
|---|---|---|
| faster-whisper 1.2.1 | GPU time, MIT | Whisper quality, batched, one flag for word times |
| whisperx 3.8.6 | GPU time, BSD-2 | Forced-aligned word times, VAD, who spoke when |
| Deepgram nova-3 | About 52 cents per 2 h | No GPU, seconds out of the box |
| AssemblyAI Universal-3.5 Pro | About 42 cents per 2 h | No GPU, speaker labels on every word, milliseconds |
Lumen Glow ran the first 20 streams through a managed API and the next 40 through faster-whisper on a rented GPU, because the API's terms did not allow the audio of a co-host who had not signed a release to leave the building. The labels from the two halves were compared on five overlapping streams before anyone joined them. The wpm column agreed; the minute of every CTA agreed to within one word. It is the check you want written down before a second vendor ever enters the pipeline.
Self-studyThe two local snippets, verbatim from the docs4 min▶
faster-whisper: one call, one flag, and each segment carries its words with start, end and a probability.
whisperx: transcribe with batching, then align the segments with a language-specific wav2vec2 phoneme model. Diarisation is a third pass, with DiarizationPipeline and whisperx.assign_word_speakers.
Whisper's own word timings come from the cross-attention pattern and dynamic time warping, and the CLI still marks the flag experimental. whisperx exists because forced alignment is more precise at word edges, which matters when a CTA phrase straddles a minute boundary. For a Mandarin stream, check the wav2vec2 alignment-model coverage before promising word-level accuracy.
From words to a segment label
9 min liveThe tagger is a pipeline, not a prompt. Words fall into minute buckets, keyword rules make the calls they can make and write down why, a model is asked only about the minutes left over, and every minute ends with one dominant label and a confidence beside it. The claims checker runs on the same buckets and never decides anything; it flags.
LiveWhy keyword rules first3 min▶
Three reasons, in the order they matter. Transparent: a rule that fired on "tap the cart" can be shown to the host, and the host can say "I say that in every minute, it is not a CTA when I say it under my breath", and then you have a better rule. Free: seven of the nine labels never need a model call, and the two that do (demo versus build) are settled by a column session 3 already produced, product in hand. Stable: a model prompt drifts with the model version; a keyword list only changes when you change it, and the change is a diff.
The cost is coverage. Rules miss the CTA said in a new way, and they miss sarcasm entirely. So the pipeline keeps the misses honest: a minute with no rule hit gets a low confidence and a model's opinion, and low-confidence minutes are sampled into the weekly review. Over time the review feeds the list. The tagger gets better the way a spam filter does, by argument, not by re-prompting.
LiveThe VLM route for what words miss3 min▶
Demo and build are the two labels the transcript cannot separate: "this is the texture you get after it sinks in" is a demo if the serum is on the back of the host's hand and a build if it is not. The cheap answer is session 3's product_in_hand column. The expensive answer, for minutes where even that is ambiguous, is a video-language model that sees the frames and hears the audio at once.
- Gemini 3.8 Flash through
client.interactions.create, with the video passed asprocessing: {type: "static", fps: 0.5}, one frame every two seconds. About 100 video tokens per second at the default low resolution, which at $0.75 per million input tokens through 2026-12-31 works out to about $0.006 per minute of video (course arithmetic from the published token rate and price). A whole two-hour stream is under a dollar; you will not send a whole stream, only the unmatched minutes. - Qwen3-VL, Apache-2.0, for the on-prem case where the recording cannot leave the building. Same question, your GPU, video sampled at 2 fps by default.
- Twelve Labs Pegasus 1.5 for the search-shaped question ("find every minute where she applies it to her hand") rather than the label-shaped one. Marengo 3.0 is the search and embeddings model beside it.
When is it worth it? When the unmatched minutes are more than a review can absorb, and when the label being settled is one that session 6 will put a number on. A demo minute converts at 11.38 percent and a build minute at 10.39 in the segment table; the gap is real but small, so spend on it after the CTA and urgency rules are right, not before.
LiveThe claims checker3 min▶
A regex list, and every phrase on it comes from a page TikTok Shop publishes. The How to Avoid Misleading Content guide names outright bans: "100% effective", "Guaranteed results", "Works for everyone", "Permanent effect", "Instant results", "No side effects". It also flags urgency wording that has to be true to be said: "While stocks last", "This sale is for selected customers only", "Don't miss these big discounts", "We are having our largest sale!". The Content Policy in force since 2026-09-17 bans "exaggerated promises about a product's performance, effect, or price", and the Misleading Discount guide says "sale ends today" or "only 2 hours left" must be genuine.
The checker returns one row per hit: minute, phrase, category (claim or urgency). It does not judge. A genuine last-day sale can say "ends today"; the checker's job is to make sure someone looks at the minute and the stock sheet together, before a platform reviewer does.
The run counter is the second half. Scarcity works for about two minutes and then it stops working and starts looking like a policy problem. Across the 60 streams, the click-to-order rate in a run of urgency minutes reads 15.12 in the first minute and 15.44 in the second, then 8.30 from the third, 8.33, 8.43, 9.09. So the tagger keeps a fomo_run column, consecutive urgency minutes so far, and the checker raises a soft flag at run 3 (the audience has stopped believing it) and a policy-risk flag at run 4 (the platform's "genuine" test is now hard to pass, and the UK's DMCC Act 2024 names "falsely stating that a product will only be available for a limited time" as a banned practice).
| Category | Phrases | Source |
|---|---|---|
| claim (banned) | 100% effective · Guaranteed results · Works for everyone · Permanent effect · Instant results · No side effects | Misleading Content guide |
| urgency (must be genuine) | While stocks last · This sale is for selected customers only · Don't miss these big discounts · We are having our largest sale! | Misleading Content guide |
| urgency (must be genuine) | sale ends today · only 2 hours left | Misleading Discount guide, eff. 2026-08-31 |
| run 3 soft flag, run 4 policy risk | third and fourth consecutive urgency minute | CTOR by run, 60 streams |
Self-studyAffective versus cognitive CTAs, a tagging refinement3 min▶
Guo, Zhang, Goh and Peng (2025, Production and Operations Management) ran speech-to-text over real live-selling sessions and split the host's calls to action into two kinds: affective ("you are going to love this, grab it with me") and cognitive ("check the ingredients list, compare the price"). In their data the affective kind raised orders and the cognitive kind did not, and cognitive CTAs reduced social engagement. Read from the abstract only; the full text was not fetched, so treat the direction as a hypothesis for your data, not a number for your slide.
As a tagger refinement it is one extra column: split the CTA keyword list into two lists and record which fired. Session 6's within-stream regression can then ask whether the plus 6.50 points on CTA minutes belongs to one kind more than the other. If it does in the Lumen Glow data, that is a line on a host's script card. If it does not, you have learned something about a paper's generalisability, which is also worth an hour.
Build the segment tagger
★ 20 min · everyone buildsOne stream's audio in, three columns out: segment, wpm, claims_flag. faster-whisper 1.2.1 for the words, pandas for everything after. If you have no GPU tonight, load the words table a managed API returned instead, and convert its unit at the door.
Transcribe with word timestamps into a words DataFrame. One row per word: start, end, text, probability. The batched pipeline is the throughput setting; the flag is the same one openai-whisper uses.
Bucket words into minutes and compute words per minute. Minute t is the interval [t-1, t) from the start of the stream, indexed from 1, the same convention as every other column. A word belongs to the minute its start falls in.
The rule-based tagger. One function, one minute in, a label and a matching phrase out. Dead air first (it overrides everything), then the keyword rules, then the two labels the words cannot separate, which read session 3's product_in_hand column. Every rule writes down why it fired.
The claims checker. The two phrase lists from TikTok Shop's guides, matched case-insensitively, one row per hit. It returns rows; it does not change the label.
Write the columns into the minute table. Segment, wpm, the phrase that fired, the confidence, a claims flag, and the urgency run counter. The run counter is consecutive urgency minutes so far, reset to zero on any other label.
Sanity-check the label distribution. Across the 60 Lumen Glow streams the minutes split demo 20.9%, build 19.0%, urgency 16.4%, hook 12.8%, CTA 9.9%, Q&A 9.1%, price 6.6%, filler 4.6%, dead 0.8%. A band to compare against, not a target: a stream with 40 percent CTA minutes is not wrong, but it is a stream you should listen to before believing the tagger.
Keep the phrase column. The why column costs nothing and is the entire review. When a host disputes a label, you look at the phrase, not the model. When a keyword list changes, you can re-tag 7,200 minutes in a second and diff the labels. A tagger without it is a model you cannot argue with, and session 6 is about to put its labels in front of a leader as advice.
Before session 5
30 min- Transcribe one real two-hour recording you are allowed to use, by any of the five routes. Write down the unit the timestamps came back in and the one line that converted it. Assert the last word's start is under 7,200 seconds.
- Run the tagger and list every minute with confidence "low". Watch three of them with the video and decide the label by hand. Write down whether the words, the picture or neither settled it.
- Add two phrases to the CTA list that your host actually says, re-tag, and count how many minutes changed label. Bring the diff.
- Run the claims checker and the run counter over the 60-stream table's
fomo_runcolumn. Count the minutes at run 4 or more, and which hosts they belong to. Session 5 puts a conversion number on each run length.
Three questions
A pipeline written against Deepgram is pointed at an AssemblyAI transcript. It runs without error, and every word of a two-hour stream lands in minutes 1 through 3, then the rest in minutes far past 120. What happened?
Both vendors return plain numbers in fields called start and end. Convert to seconds at the door, name the column so the unit is visible, and assert the last start is under the stream length.
Why do the keyword rules run before any model, when a model would label more minutes?
Rules are less accurate on coverage and more accurate on accountability. Seven of nine labels never need a model; the two that do are mostly settled by session 3's product-in-hand column.
Minute 71's transcript reads: "Only 2 hours left on the bundle, and honestly it works for everyone, tap the cart." What does the claims checker return for it?
"Works for everyone" is on the Misleading Content guide's banned list, and "only 2 hours left" is on the Misleading Discount guide's must-be-genuine list. The checker flags both and changes nothing; a person checks the stock sheet.