Streaming and CDC

Batch processes a bounded chunk on a schedule - simple, cheap, and right for most data. Streaming processes records one at a time as they arrive on an unbounded log - lower latency, more moving parts. The consumer tracks an offset so it knows where it left off and can resume after a restart.

• The log is the core idea: events are appended to an ordered log; many consumers can read the same log independently, each at its own offset. The log does not care who reads it or when.
• CDC turns a database into a stream: Change Data Capture reads the source database's own transaction log and emits each insert, update, and delete as an event - so downstream systems see changes seconds after they happen, without re-querying the whole table.
• CDC is how "real-time replication" is really done: instead of re-extracting a table every hour (b3's incremental batch), CDC streams just the changed rows off the DB's log.

Streaming systems make a delivery promise, and it is worth knowing the two you will hear. At-least-once guarantees no event is lost, but an event may be delivered twice after a failure - so your processing must be idempotent (b5) or you double-count. Exactly-once guarantees each event is processed once and only once - stronger, but costs more coordination and throughput. Most systems default to at-least-once and lean on idempotent transforms to make duplicates harmless. It is the same idempotency discipline from b5, now protecting a stream instead of a batch.

Pretend events is a live log. A micro-batch processes just the slice inside the current window - here, the first week of March - and aggregates by event type. In a real streaming job this same query would fire every few minutes over the newest window. Press ▶ Run.

-- one micro-batch window over the "stream" of events
SELECT event_type,
       count(*) AS events_in_window
FROM events
WHERE CAST(event_date AS DATE)
      BETWEEN DATE '2026-03-01' AND DATE '2026-03-07'
GROUP BY event_type
ORDER BY events_in_window DESC;

A micro-batch inherits the same delivery question. If a window is retried after a crash, at-least-once means some events in it may be processed twice. The safe pattern: make each window's output idempotent - key the result by (window, event_type) and rebuild it with CREATE OR REPLACE or a merge, so re-processing a window overwrites rather than appends. This is why b5's idempotency lesson matters even more once data is arriving continuously: retries are not rare edge cases in streaming, they are routine.

A Kafka consumer reading events off a topic - the log side. This needs a running Kafka cluster, so read it, do not run it:

And Kinesis, AWS's managed log, reading one shard's records - same shape, managed service:

No SQL - use the one test that cuts through the hype. Ask: does lower latency change a decision or an action? If a fraud check must block a transaction in the next 200 milliseconds, or an alert must fire the instant a sensor spikes, you need true streaming - the latency is the product. If freshness of a few minutes is plenty (a live dashboard, near-real-time metrics), use micro-batch - almost all the benefit, a fraction of the operational cost. If nobody acts on the data until tomorrow anyway (a daily report, a monthly finance close), stay on batch - streaming would be complexity nobody uses. The honest default: batch unless someone can name a decision that a few minutes of delay would ruin, then micro-batch, and reserve true streaming for when milliseconds genuinely matter.