The stale dashboard

The nightly dashboard scores perfectly on one axis and terribly on the other. Everything it shows is correct: totals reconciled, duplicates cleaned, late records folded in. But its freshness is whatever the clock says minus the last batch run - at 2pm, that is 14 hours.

Streaming systems make the opposite trade by default. They can show you an order seconds after it happens, and in doing so they inherit every mess the batch job used to clean up overnight: retries that arrive twice, orders that arrive late, records that land in the wrong hour. Fresh but sloppy is the starting point, not the goal.

• Batch: correct, then eventually fresh (tomorrow).
• Naive streaming: fresh, then eventually correct (if you do the work).
• The craft of this course: fresh AND correct, on purpose.

The nightly batch is not a legacy embarrassment. It is a genuinely good design for its constraints: it sees the WHOLE day before it aggregates, so late arrivals, retries and corrections have all landed by the time it runs. Completeness is cheap when you are willing to wait.

That is the standing tension. A batch job answers with everything known at run time. A stream answers NOW, with only what has arrived so far. Every technique in sessions 2 through 5 - idempotency, event-time windows, watermarks, replay - is a way of buying back, at 2pm, the completeness the batch job gets for free at 2am.

Because a fact never changes, you can hand the same event to five different consumers and they can each build their own view of the world - a revenue total, a fraud score, an inventory count - without coordinating with each other. And because the facts are kept in order, any of those views can be rebuilt from scratch by replaying them.

Martin Kleppmann's framing is the cleanest: the database you read from is just a cached view of the event log. When something goes wrong, you replay the events in the same order and reconstruct exactly what happened. That idea returns in session 5 as your correction mechanism, and in session 6 as an entire architecture.

Every event in tonight's simulator carries two timestamps. WHEN it happened (the customer tapped pay at 23:41) and WHEN the system heard about it (maybe 23:41, maybe 2:50am if the store's tablet was offline). For on-time events the two are the same and nobody cares. For late ones they diverge, and which timestamp you aggregate by decides whether your hourly chart tells the truth.

Hold the distinction; do not solve it yet. It becomes the whole of session 4 (windows) and session 5 (watermarks).

Jordan Tigani, a founding engineer of BigQuery, published the uncomfortable measurements: among customers actually paying for the service, 90 percent of queries processed less than 100MB of data, and heavy users' median total storage was under 100GB. He also measured the recency bias - data more than a week old is roughly 20 times less likely to be queried. Most "big data real-time" needs are neither big nor real-time.

And streaming's price is not only infrastructure. Gunnar Morling, who builds these systems for a living, puts the cost where it belongs: the price is complexity - reasoning about streaming joins, late data, and reprocessing is genuinely harder than reasoning about a nightly job.

• The steelman is real too. Fraud detection, anomaly detection, and rule-based alerting are the canonical cases where seconds change the outcome - a fraud block after the money leaves is worthless.
• And once teams taste it, they keep it. Morling again: experience a data freshness of a second or two in your warehouse and you never want to go back.