Late data

Read the definition again slowly: a watermark is not a measurement, it is a declaration. The system announces "event time has reached t - nothing with a timestamp at or before t is still coming", and then acts on its own announcement: windows ending at or before t close and emit. A late event is what it sounds like when that promise turns out to be false. And the default consequence is unsentimental - late elements are dropped when the watermark is past the end of the window. That is precisely the mechanism behind session 4's six lost stragglers.

Which makes the watermark setting a business decision wearing an engineering costume. "How long does the lunch hour stay open for corrections?" is not a tuning knob - it is a question about how your data actually arrives, and about who would rather wait than restate a number. Somebody has to own the answer, and it is rarely the person writing the pipeline config.

Flink splits the decision into two levers. Allowed lateness keeps a window reopenable for a bounded time after the watermark passes its end: a straggler inside that grace period triggers the window to fire again, and those late firings are "updated results of a previous computation". Downstream consumers have to be ready for a number that was already published to change - which is exactly what a self-correcting hourly bar means.

And for events later than even THAT, there is sideOutputLateData: instead of vanishing, too-late events are captured to a side channel where they can be counted, monitored, and investigated. The professional standard is simple: late data may be excluded from a view, but it should never be invisible. A dropped-events counter you can see is a data-quality metric. A silent drop is a mystery you will meet again during an audit.

Nathan Marz compressed it to an equation: "Query = Function(All Data)". If the data is immutable - facts appended, never edited - then a bug in your query logic is a recoverable event: fix the function, recompute from the master dataset, and history itself is repaired. Mutable systems cannot do this, because the bug already overwrote the evidence.

Martin Kleppmann then flipped the mental model of where truth lives: "The database you read from is just a cached view of the event log." Replay the events in the same order and you reconstruct exactly what happened. And here is where session 4 pays off: event-time windows are deterministic - the same log through the same windows gives the same answer, every run. Determinism is what turns "replay" from a hopeful word into a guarantee that the rebuilt view CONVERGES.

Two named patterns take the log-as-truth idea further, and both live in chapter 11 of Kleppmann's Designing Data-Intensive Applications. Change data capture turns a database's own changes into a stream: every insert, update and delete becomes an event other systems can consume, so an existing application database becomes a log producer without being rewritten. Event sourcing goes further and stores the events AS the system of record - the application's state is always a view computed from them.

Same shape at two levels of commitment: CDC bolts the log onto a mutable database; event sourcing builds the whole system on the log from day one. The book is the canonical treatment, and it is the right next read once tonight's replay lever has made the idea physical.