Where the artifact stands
The order stream is live and deduplicated: 89.1% accurate, fresh to the second. The remaining error is all about TIME - late orders are being counted in the hour they arrived, not the hour they happened. Tonight fixes the bucketing. It does not yet fix the lateness.
★ What you walk out with: the three window shapes and when each is the right question, the event-time versus processing-time distinction that decides whether your hourly chart tells the truth, and first-hand experience of the strict-window trap - the reason watermarks exist.
Three ways to cut an endless stream 7 min live
The Flink documentation names the standard shapes, and every serious streaming system offers the same three. The shape you pick IS the question you are asking.
Live The shape is the question 5 min ▶
The definitions, straight from the Flink docs. Tumbling windows have a fixed size and do not overlap - each event belongs to exactly one. That is "revenue per hour", tonight's dashboard question. Hopping windows (Flink calls them sliding) have a fixed size plus a slide interval, so they overlap - a one-hour window recomputed every 15 minutes is a smooth trend line, and each event lands in several windows. Session windows have no fixed size at all: one closes when no elements arrive for a set gap. That is "one customer's visit to the shop".
- Hourly revenue report - tumbling 1h. One answer per hour, no double counting.
- "Are sales accelerating right now?" - hopping 1h/15min. Smooth, overlapping, deliberately redundant.
- "How long does a lunchtime customer browse?" - session with a 30 min gap.
Uptime pages compute error rates over hopping windows so an incident shows as a rising curve rather than a cliff at the hour boundary. Finance closes are tumbling by construction: a transaction belongs to exactly one month.
Self-study The fourth shape, and why it is a trap for beginners 3 min ▶
Flink also offers a global window: every event with the same key lands in one single window that never closes on its own. The docs are blunt that it is only useful with a custom trigger, because a window that never closes never emits an answer. It exists for building exotic logic by hand, not for reporting.
If you catch yourself reaching for a global window to compute "total revenue ever", the stream is telling you that question belongs to the warehouse, not the stream. Totals-over-all-time are batch questions wearing a streaming costume.
Event time versus processing time 8 min live
Windows need a clock, and there are two candidates. The Flink docs define them precisely: processing time is the system time of the machine executing the operation. Event time is the time each event occurred on its producing device. Everything session 5 fixes starts here.
Live Why anyone still uses processing time 4 min ▶
Because it is free. No timestamps to extract, no out-of-order events to reason about, no waiting: the machine's own clock decides everything, and results are emitted the instant a window's wall-clock boundary passes. The Flink docs name the price with unusual candor - processing time offers no determinism, because it depends on the speed at which records happen to arrive.
Event time buys back the truth: results are correct and consistent even when events arrive out of order. The cost is that the stream now has to ANSWER a question processing time never faces: how long do you wait for stragglers before declaring an hour finished? That question has a name - the watermark - and it is the whole of session 5.
Mobile analytics is the extreme case: a phone can go through a tunnel at 12:55 and sync at 6pm. Products that bucket by processing time report a phantom evening engagement spike that is really the lunchtime tunnel traffic arriving late, in a pile.
Self-study Determinism is what makes replay possible 3 min ▶
Event-time windows have a property that quietly powers session 5's replay lever and session 6's whole architecture: run the same log through them twice and you get byte-identical answers. Processing-time results can never be reproduced - the second run's arrival times are different by definition.
That means an event-time pipeline can be rebuilt from the log after a bug fix and converge on the truth, which is exactly Nathan Marz's recovery story for immutable data: fix the code, recompute from the master dataset. A processing-time pipeline cannot be recomputed, only apologized for.
Switch the buckets, watch the dip ★ 22 min · everyone builds
Same simulator, one new lever. Tonight's window is tumbling 1h on event time, and it closes STRICTLY - the moment the hour ends, its answer is frozen. Watch what strictness costs.
Rebuild last session's state: Stream it ON, Idempotent consumer ON. Confirm the headline reads 89.1% - fresh, deduplicated, and still lying about WHEN money happened, because buckets follow arrival time.
Turn on Event-time windows and read both numbers honestly. The chart's shape snaps to the truth - solid bars now line up under the dashed outlines, the lunch rush is back in the lunch hour. And the headline accuracy went DOWN, 89.1% to 88.0%.
Find out why in the counters: 6 late events dropped. A strict window froze each hour at the top of the hour; stragglers that used to be counted somewhere (the wrong somewhere) are now counted nowhere. The dashboard shows $3,275 of a true $3,723.
Say the trade out loud, because it is real: arrival-time bucketing smears late money across wrong hours but keeps the total closer. Strict event-time bucketing makes every bar honest and silently discards the stragglers. Neither is acceptable - which is precisely the argument for a window that stays open a little longer. That lever is session 5.
The dip is the lesson. A metric that only ever goes up when you do the "right" thing is a metric you should not trust. Doing windows correctly EXPOSED a real loss that sloppy bucketing was hiding. The Flink docs state the default plainly: late elements are dropped once the watermark passes the end of the window. Session 5 is about choosing that boundary on purpose instead of accepting it by accident.
Before session 5 15 min
- Take the dashboard you chose in session 1 and name its window shape and clock: tumbling, hopping or session, on event time or processing time. Most BI tools quietly use processing time - check yours.
- Find one chart in your organisation with a suspicious end-of-day or early-morning spike. Write two sentences on whether it could be late data arriving in a pile.
- For your hourly metric, write down how late an event can be before your business would rather drop it than restate the hour. Keep the number - it becomes your watermark choice next session.
Three questions
1. You want a smooth "sales in the last hour" trend, refreshed every 15 minutes. Which window?
A one-hour window restarted every 15 minutes overlaps deliberately - each event counts in four windows, which is what makes the trend smooth. Tumbling 15 min would answer a different, choppier question.
2. Why did switching to event-time windows make the accuracy number worse?
Arrival-time bucketing at least counted stragglers, in the wrong hour. A window that closes at +0 drops them entirely: 6 events, $448 of real revenue, gone from the dashboard. That loss is why watermarks exist.
3. What does processing time cost you, per the Flink docs?
Processing time is the lowest-latency option precisely because it never waits. What it surrenders is determinism: results depend on arrival speed, so the same log replayed tomorrow lands in different buckets.
Answer all three to see your score.