Where the artifact stands
Five sessions of levers: stream, dedupe, event-time windows, watermark, replay. The ladder ends at 100.0% and $6,115 of full-day truth. Tonight nothing gets more accurate - and that is the point. One lever remains, and it is a warning label: it improves nothing the dashboard measures and multiplies what the dashboard never shows you.
★ What you walk out with: consumer lag as the one operational number that matters, the always-on billing shape that makes streaming expensive at rest, the lambda-versus-kappa decision with the receipts, and a strict five-row ship-or-not scorecard you can run against any stream - including the one you built in this course.
Operating a stream is watching lag 7 min live
Session 2 defined it: LAG is log-end-offset minus current-offset - how far the consumer's bookmark trails the end of the log. In production that arithmetic becomes the whole job. Kafka's own operations guidance is blunt: "max lag needs to be less than a threshold and min fetch rate needs to be larger than 0", and the metric to watch is records-lag-max.
Live Alert on lag, not on "is the process running" 4 min ▶
A consumer that has crashed is the easy failure - everything pages. The dangerous failure is a consumer that is alive, healthy on CPU, green on every uptime check, and processing slower than producers are writing. Nothing is down. The log simply grows faster than the bookmark advances, and every minute of that gap is a minute of staleness on the dashboard downstream.
That is why the ops guidance is phrased as two thresholds rather than a liveness check: max lag under a threshold, min fetch rate above zero. Lag is the difference between what has happened and what has been seen - which is exactly the number this whole course has been about. The stale dashboard of session 1 was 14 hours of lag with a nightly excuse. A streaming pipeline with unwatched lag is the same dashboard without the excuse.
The 2pm dashboard, some months after shipping: a slow deserializer update lands at 11am, the consumer keeps "running", and lag grows quietly for three hours. At 2pm the dashboard confidently shows 11am's world. Nobody set a lag alert, so the first detector is a human squinting at a suspiciously flat lunch rush.
Self-study Backpressure in one breath 3 min ▶
When a consumer slows down, where does the pressure go? Into the log. That is what retention is FOR: the broker keeps events for its configured window whether or not anyone has read them, so a slow consumer does not lose data - it falls behind in a buffer measured in days, not seconds. The failure mode of a struggling stream is not silent loss; it is growing lag.
Which is quietly good news: growing lag is measurable, alertable, and recoverable - catch it inside the retention window and the consumer just reads faster until the bookmark catches the head. The unrecoverable version is the one nobody was watching until retention expired under the unread events. The metric was there the whole time.
What it costs, and the two architectures 8 min live
Batch bills you while it runs. Streaming bills you for existing: AWS charges a provisioned Kinesis shard per hour whether or not data flows - $0.015 per shard-hour in us-east-1 - so the meter runs 24/7 even at 3am when the coffee shop is closed. Multiply by every always-on pipeline, and architecture becomes a money question.
Live Lambda, kappa, and who pays the complexity 5 min ▶
Nathan Marz's lambda architecture answered the fresh-AND-correct problem structurally: a batch layer recomputes complete truth from the master dataset, a realtime layer covers the recent gap, and queries merge the two. It works. Its price is the part Jay Kreps named from experience: "maintaining code that needs to produce the same result in two complex distributed systems is exactly as painful as it seems like it would be". Every metric definition exists twice, in two frameworks, and must agree forever.
Kreps's kappa alternative drops the second code path: retain the full log, run one stream-processing job, and when the logic changes, spin up a second job that reprocesses from the start of the log, switch the readers over when it catches up, and retire the old one. "You only do reprocessing when your processing code changes" - it is a repair procedure, not a permanent parallel system. Gunnar Morling adds the deflating footnote: the batch-versus-streaming dichotomy is largely false anyway - streaming systems batch internally for efficiency. Notice what you built this course: live windowed view for NOW, end-of-day replay for EXACT, one log under both. You have been holding the kappa shape since session 5.
The honest architecture decision names three things: the decision being served, the freshness it truly needs, and who pays the complexity. Fraud blocking, anomaly detection and rule-based alerting - Flink's own use-case list - genuinely need seconds. The monthly finance close needs to be right to the cent and does not care about 2pm; making it streaming adds cost and reasoning burden for a freshness nobody asked for. Batch remains the correct answer more often than streaming vendors say.
Self-study The small-data check, and what this course did NOT cover 4 min ▶
Before architecting anything always-on, rerun Jordan Tigani's measurements against your own workload: 90% of real queries processed under 100MB, heavy users' median total storage under 100GB, and week-old data roughly 20x less likely to be queried. Most workloads are small and recent-biased, and a single modern machine handles most of them. If yours fits that profile, the honest architecture might be a nightly job and this course was a vaccination, not a blueprint.
And named honestly: deep operations was out of scope here. Broker tuning, the control plane, tiered storage, geo-replication - running the LOG itself, rather than consuming it - is its own discipline. The next step is Confluent's Kafka Internal Architecture course by Jun Rao, which covers exactly that territory. This course taught the semantics; that one teaches the machine.
Pull the anti-lever, then run the scorecard ★ 22 min · everyone builds
One last lever, and it is a trap with a warning sign on it. Everything the ladder earned is on screen. Now watch what streaming MORE buys you: nothing the dashboard measures, everything the bill measures.
Rebuild the full ladder: Stream it, Idempotent consumer, Event-time windows, Watermark, then the replay (passport at 4 stamps, or the honest bypass). Confirm the summit: accuracy 100.0%, $6,115, lag in seconds.
Flip "⚠ Stream all 13 tables" and stare at the two headline numbers. Accuracy: 100.0%, unmoved. Lag: seconds, unmoved. Eleven new always-on pipelines changed NOTHING the dashboard measures, because the orders table was already streaming and the other twelve never mattered to it.
Now read what DID move: the cost meter, $340 to $1,980 per month, pipelines 2 to 13. Note the badge on those figures - "modelled", not "measured" - unlike every accuracy number, which is computed live from the event log. The model quotes the real billing shape as its evidence: provisioned shards bill per hour whether or not data flows, so a pipeline for a table that changes once a month pays for 720 shard-hours a month to carry almost nothing.
Finish with the final scorecard below - strict, ship-or-not. Score the simulator's stream, then score a real one you know. Every ✗ is a reason the stream is not done, whatever the demo looked like.
| Ship check | ✗ looks like | ✓ looks like |
|---|---|---|
| Lag alerting exists with a threshold | "The consumer is running" is the only health signal | records-lag-max alarmed at a number someone chose |
| Freshness requirement written down and owned | "Real-time" appears in the ticket, undefined | One sentence: metric, allowed staleness, owner's name |
| Cost fits the decision it serves | Always-on pipelines feeding a weekly meeting | The shard-hour math is attached to the decision it buys |
| Replay path tested, not just believed | "We could rebuild from the log if we had to" | A rebuild has actually run and reconciled to truth |
| Tables streamed only where seconds matter | All 13 tables streaming because the platform could | Each streamed table names the decision seconds improve |
The verdict rule: this scorecard has no partial credit and no weighting, on purpose. A stream with four ✓ and one ✗ is a stream with a named piece of unfinished work - say which row and who owns it. "It demos well" appears nowhere in the table, because the demo was never the hard part. Sessions 1 through 5 were.
After the course 15 min
- Run the five-row scorecard against one real pipeline you rely on. Write the verdict and the single ✗ you would fix first - then send it to the person who owns that row.
- Take the table list from session 5's homework and mark which rows genuinely earn seconds-level freshness. For everything else, write the batch cadence that would serve the same decision.
- Read Kreps's "Questioning the Lambda Architecture" in full - it is short - and, when you are ready for the machine underneath, take Confluent's Kafka Internal Architecture course as the next step.
Three questions
1. Which signal tells you a live, healthy-looking consumer is quietly making the dashboard stale?
A consumer can be running, on healthy hardware, and still fall behind the log. Lag measures exactly the gap between what happened and what has been seen - which is why the ops guidance alarms on max lag and fetch rate, not on liveness.
2. What did streaming all 13 tables actually change in the simulator?
Accuracy and lag were already served by streaming the one table that mattered. The other twelve bought eleven always-on pipelines - billed by the shard-hour whether or not data flows - for tables whose decisions never needed seconds.
3. What distinguishes kappa from lambda?
Lambda runs the same logic twice - batch layer plus realtime layer, merged at query - which Kreps called exactly as painful as it sounds. Kappa retains the full log, runs one job, and spins up a replay job only when the processing code changes, switching over when it catches up.
Answer all three to see your score.