Ingest and align

A LIVE is not live. The host speaks, the phone encodes, the platform ingests, transcodes and distributes, and the viewer's player buffers. The words reach the room some seconds after they left the host's mouth. The viewer then taps the card and buys, and the platform logs the purchase on its own clock: the time it received the order.

So a purchase logged at 20:43:07 is an answer to words spoken earlier, and the seller's per-minute ledger is a record of what the room did, shifted late by the delivery delay. Your recording of the host is on the earlier clock. Join the two raw and you are pairing what the host said at minute 43 with what viewers did about minute 42. If the delay straddles a minute boundary, and with a delay of several seconds it often does, a chunk of every minute's sales is credited to the following minute.

• Keyframe - a frame the codec stored whole, not as a difference from its neighbours. A stream-copied file can only be cut cleanly at one.
• Segment - one output chunk of the recording, here one minute of video, so that every downstream job has a natural unit of work that matches the table's grain.
• fps sampling - keeping one frame per chosen interval. 1 fps gives 60 frames a minute, enough for a share-of-frames column and cheap enough to run a detector on.
• PCM 16 kHz mono - uncompressed audio, 16,000 samples a second, one channel. The input format Whisper-family transcribers were trained on.
• Offset - seconds from the start of the recording, the recording's own clock. Every source is converted to it.
• Lag - a shift of one series against another by a number of steps. Scanning lags is how the delay is found.
• Event time vs recording time - when something happened for the viewer versus when the recording says it happened. The ledger is on event time; the video is on recording time; the delay d is the gap.

FFmpeg 9.0.2 ships a segment muxer that splits one input into many output files. The documented example is a ten-second split with a list file:

Change -segment_time 10 to 60 and the file pattern to chunks/min-%03d.mp4 and you have minute chunks. Add -reset_timestamps 1 so every chunk starts at zero, which is what you want when a per-chunk job reports "second 17" and means second 17 of that minute.

The catch is -c copy. Stream copy does not decode the video, so it can only split where the encoder left a keyframe, and a live-stream recording has keyframes wherever the encoder decided, typically every few seconds and not on the minute. The muxer waits for the next keyframe after the 60 second mark, so each chunk runs a little long and the drift accumulates: chunk 43 no longer starts at minute 43. Two documented fixes. Re-encode once with -force_key_frames "expr:gte(t,n_forced*60)", which places a keyframe on every minute boundary, then stream-copy the segments from that file. Or accept -break_non_keyframes 1, which cuts exactly on time and leaves the first frames of each chunk undecodable until the next keyframe. For analysis frames the first is the one to use, because the detectors in session 3 need every frame of the minute.

Two more commands, both one line. The fps filter resamples the frame rate, and pointed at an image-sequence output it writes one JPEG per second: ffmpeg -i stream.mp4 -vf fps=1 frames/%06d.jpg. Frame number n is second n of the recording, which makes the frame-to-minute mapping a floor division and nothing more.

The audio strip drops the video, downmixes to one channel, resamples to 16,000 Hz and writes 16-bit PCM: ffmpeg -i stream.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le audio.wav. Why exactly that: the Whisper-family models session 4 uses were trained on 16 kHz mono and resample anything else to it internally, so handing them that format skips a conversion, removes a source of tiny timestamp drift, and makes the file a fifth of the size of the stereo 48 kHz original. It is also what the hosted transcription APIs bill on, so the smaller file is the cheaper call.

Scene cuts are the cheapest thing a camera tells you. A cut is a hard change between one frame and the next: the host switched from the front camera to the product camera, the operator cut to the price card, the phone was picked up. PySceneDetect 0.7.1, BSD-3 licensed, finds them in one call:

Why AdaptiveDetector and not the plain ContentDetector: ContentDetector compares each frame with the last in HSV space and calls a cut whenever the difference crosses a fixed threshold. A handheld live-commerce camera crosses that threshold constantly, every time the host waves the serum or the phone tilts. AdaptiveDetector runs the same scores and then compares each one with a rolling average of its neighbours, so a sustained wobble is not a cut and a real switch is. Fewer false cuts under camera motion is exactly the property this footage needs.

The output is a list of scenes with start and end timecodes. A cut is where one scene ends; floor its seconds by 60 and count per minute, and scene_cuts becomes a column. In the Lumen Glow table it already is one, and the AI-avatar streams sit at 1 to 3 cuts a minute against 2 to 7 for the human hosts, which session 3 comes back to.

A two-hour stream is 7,200 frames at 1 fps, and the whole 60-stream corpus is 432,000. Every detector in session 3 runs once per frame, so this number is the multiplier on every per-frame cost you take on later. Some rules of thumb for the sampling budget:

• 1 fps is enough for share-of-frames columns. Product in hand, host on camera and price on screen change on a scale of seconds, not frames. Sixty samples a minute give a share with a resolution of about 1.7 points, well below anything the analysis in session 6 can distinguish.
• Scene cuts need more. A cut is a single frame-to-frame event, so PySceneDetect reads the full-rate video, not the 1 fps sample. Run it on the MP4, not on the JPEGs.
• Sample down, never up. If a detector is slow, run it on every other frame and mark the minute's usable-frame count. Session 3 nulls any minute with fewer than 30 usable frames rather than reporting a share built on a handful.
• Keep the chunks. When a detector improves, you re-run on the minute chunks that changed, not on the two-hour file. That is what the segment muxer bought you.

You rarely get told the delay. The platform does not publish it, it varies by region and by the viewer's connection, and nobody on the production side is measuring it. But it is sitting in your data, because the host's words cause the room's actions and causes precede effects. Whatever the host does that reliably produces a response (a call to action is the strongest one in this data, at +6.50 points of CTOR within a stream) will produce that response in the ledger after the delay, not before.

So build two per-minute series on the recording clock: a count of CTA minutes from the transcript, and the ledger's orders. Shift the CTA series forward by 0, 1, 2, 3 and 4 minutes and correlate each with orders. The lag with the highest correlation is the delay, rounded to the grain. Most stream delays are a handful of seconds, so the answer is usually 0 or 1, and the value of the scan is not the number but the certainty: you know which one, and you can show your working.

Then verify against one event you know the time of on both clocks. A LIVE Flash Sale has a start timestamp in LIVE Manager and a moment in the recording when the host says "the flash sale is live". The gap between them is the delay in seconds, independently of the correlation scan. If the two methods agree, apply the shift. If they do not, the scan found a coincidence and the known event wins.

Nothing errors. The join succeeds, the row counts match, the totals reconcile to the cent because a shift moves values between minutes without changing their sum. And every content-to-sales number in sessions 5 and 6 is wrong in the same direction.

• Effects attenuate toward zero. If part of each CTA minute's orders is credited to the following minute, the CTA minute looks weaker than it was and the estimate of its effect shrinks. The within-stream +6.50 points of CTOR for a CTA is what you see when the clocks agree. Misalign them and it comes out smaller, and the confidence interval widens because the signal is smeared over two minutes.
• Credit moves to the following segment. The minute after a CTA is often filler or a breath, and it now carries the CTA's orders. So a CTA looks worthless and the filler after it looks golden. In the aligned Lumen Glow data filler runs at 1.40 orders per thousand viewer-minutes against 11.57 for CTA; an unshifted join flattens that gap and can invert it for short CTAs.
• Short things suffer most. A twenty-minute demo loses a few seconds off its end and nobody notices. A forty-second call to action can lose most of its orders to the next row. The segments the leader track cares about most are precisely the short, sharp ones.
• The forecast learns the wrong lead. Session 7's nowcast reads the last few minutes of content to predict the next five of orders. Trained on a misaligned table it learns that filler predicts orders, and cues the host accordingly.

What you did tonight is a special case of something every streaming system has to handle: the time an event happened and the time it was observed are different, and the difference is not constant across sources. Stream processors solve it with event-time windows and watermarks, a running estimate of how late an event can still arrive before the window closes. The streaming data course builds that machinery properly, with late arrivals, out-of-order events and the accuracy-versus-latency trade the watermark encodes.

Here the problem is simpler for one reason: the delay is stable within a stream. Every ledger row in a two-hour recording is late by roughly the same d, so one measured shift solves what a watermark would otherwise track continuously. The moment that stops being true, for instance if you ingest a running stream in session 7 where the delay can drift as the room grows, you are back in the general case and the watermark is the right tool. Knowing which case you are in is most of the skill.

One more thing to write down: which clock each column is on, in the table's documentation. Frames and words are on the recording clock by construction. The ledger and the comment feed are on the platform clock and were shifted by d. A column added later by someone who did not attend tonight will be on whatever clock its source used, and the note is what stops it being joined raw.