Where the artifact stands
The 2pm dashboard still shows midnight: $0 of today's $3,723 on screen, 14 hours behind. Tonight the coffee shop's orders start flowing through a log instead of waiting for the 2am batch. Freshness arrives in one lever flip - and so does the first honest look at what freshness costs.
★ What you walk out with: the four words that unlock every Kafka conversation (topic, partition, offset, consumer group), the exact shape of the ordering guarantee - and its sharp edge, the two numbers ops people watch (order and lag), and a live stream feeding the 2pm dashboard for the first time.
Kafka is an append-only log 7 min live
Strip away the ecosystem and Kafka is a durable, ordered, append-only log of events. The Kafka documentation's own metaphor: a topic is similar to a folder in a filesystem, and the events are the files. Nothing in that folder is ever edited - only appended, and eventually aged out.
Live Four words that carry every Kafka conversation 5 min ▶
Everything you will ever say about Kafka is built from four nouns:
- Topic - a named category of events. The docs' metaphor: similar to a folder in a filesystem, and the events are the files. Ours is
orders. - Partition - topics are split into "buckets" spread over the brokers of the cluster. Partitions are how one topic scales past one machine and one reader.
- Offset - each event's position in its partition. The design docs make a point of the smallness: a consumer's position is just a single integer.
- Consumer group - a set of readers acting as one logical consumer, tracking its own offsets. Different groups do not share a position, or even know about each other.
And the property gluing them together: producers and consumers are fully decoupled. The producer appends and moves on. It does not know who reads, when, or how many times.
The coffee shop's order stream feeds the revenue dashboard AND a fraud checker. They are two consumer groups on the same topic: the dashboard skims along near the log's head, the fraud model batches up a few minutes at a time. Neither waits for the other, neither steals from the other, and adding a third reader next month touches nobody's code.
Self-study Retention: the log keeps events whether or not anyone read them 4 min ▶
This is where Kafka quietly parts ways with a classic message queue. A queue deletes a message once a consumer takes it - consumption IS destruction, so one reader's read is another reader's missing data. Kafka never deletes on read. Events age out on a per-topic retention policy, and until then they sit there for anyone, read once, five times, or never.
The dials: log.retention.hours defaults to 168 - seven days - and log.retention.bytes defaults to -1, no size cap. Seven days of history on tap is what makes the ideas later in this course possible at all: a new consumer group can start from a week back, and a broken one can rewind and re-read, because reading was never destructive in the first place.
Hold that thought for session 5, where re-reading the log stops being a recovery trick and becomes the correction mechanism.
The two numbers ops people live by 7 min live
Order and lag. The first is a guarantee with a precisely drawn boundary; the second is the single most-watched health metric in any streaming system. Learn where the boundary sits and how the metric is computed, and most production incidents become legible.
Live Ordered per partition, keyed by order id - exactly enough 4 min ▶
The guarantee, verbatim from the Kafka introduction: "any consumer of a given topic-partition will always read that partition's events in exactly the same order as they were written." Read the boundary as carefully as the promise: it is per partition. Across partitions there is NO ordering at all - partition 1's events interleave with partition 2's however the readers happen to run.
The lever that makes this useful: events with the same key go to the same partition. Key the order stream by order id, and everything about order #8412 - placed, paid, refunded - lands in one partition, in write order, forever. That is exactly enough for the coffee shop:
- What we need ordered: the life of one order. A refund processed before its payment would corrupt every downstream view.
- What we do not need ordered: order #8412 versus order #8413. They are different customers; no correct answer depends on who appears first.
Demanding global order would mean one partition, one reader, no parallelism. The craft is noticing how little ordering your problem truly needs, then paying for only that.
This is the standard playbook everywhere entities have lifecycles: user id for account events, device id for telemetry, ticket id for support flows. Per-entity order, cross-entity parallelism.
Self-study Replication in one breath 3 min ▶
Each partition has a leader broker and follower replicas; followers that are caught up form the ISR, the in-sync replica set. The design docs state the durability contract plainly: "a committed message will not be lost, as long as there is at least one in sync replica alive."
Producers choose how hard to lean on that: acks=0 (fire and forget), acks=1 (leader wrote it), acks=all (the ISR has it), with min.insync.replicas setting the floor. That one sentence and three settings are all this course needs - broker tuning, the control plane and the rest of the deep-ops iceberg are Confluent's Kafka Internal Architecture course territory, and we will point at it honestly in session 6 rather than pretend to cover it here.
Flip the stream on ★ 22 min · everyone builds
Same simulator, same deterministic day of coffee-shop orders. Tonight it grows its first lever. You are about to fix the only thing session 1 proved was broken - and meet everything the 2am batch had been quietly cleaning up for you.
Read the baseline one last time. Batch mode, freshness lag 14h, "showing yesterday": $0 of today's $3,723 on screen. This is the number the whole course exists to fix - say goodbye to it.
Flip Stream it ON. Two things happen at once, and only one of them is good. Freshness lag collapses from 14 hours to seconds - and accuracy lands at 78.9%. The dashboard now shows $4,073 against a true $3,723, with 15 retries double-counted. It is not just wrong, it is wrong UPWARD.
Read the chart, not just the headline. Solid bars finally exist - today is on screen for the first time - but they sit wrong against the dashed truth outlines: some hours too tall (retries counted twice), some money in the wrong hour (late arrivals bucketed by when they showed up).
Say session 1's trade out loud, because you just watched it happen: freshness arrived, honesty left. The batch job's overnight cleanup did not come along for the ride. Sessions 3 through 5 buy honesty back, one lever at a time - dedupe, honest buckets, patience for stragglers.
These errors are the stream's defaults, not bugs. Nothing in the simulator is sabotaging you. Retried sends and late arrivals are simply what at-least-once delivery looks like when nobody has done the correctness work yet - the producer resends because it cannot know whether the first attempt landed. Why that resend is the rational choice, and what to do about the duplicates, is the whole of session 3.
Before session 3 15 min
- Find one event stream in your organisation - a Kafka topic, a webhook feed, anything append-shaped. Write down its four nouns: what is the topic, how is it partitioned, who are the consumer groups, and roughly where do their offsets sit.
- Name its key. If order id is the coffee shop's answer, what is yours - and does everything that must stay ordered actually share that key?
- Find out its retention. If a consumer went down on Friday evening, would the events still be there on Monday morning? Write the yes or no down; session 5 will make you glad you know.
Three questions
1. Kafka guarantees ordering... where, exactly?
The guarantee is per topic-partition: any consumer reads that partition's events in exactly the write order. Across partitions there is no order - which is why the coffee shop keys by order id, putting each order's whole life in one partition.
2. The dashboard group's lag on partition 2 is 4. What does that mean?
LAG = LOG-END-OFFSET minus CURRENT-OFFSET, per partition. It counts events, not seconds, and nothing is lost - the events are sitting in the log waiting. Lag is the freshness gauge, which is why ops people alert on it.
3. The fraud group reads event 5 from the orders topic. What happens to event 5?
Reading never deletes. Events expire on the per-topic retention policy (log.retention.hours defaults to 168 - seven days) whether they were read once, many times, or never. That is the difference between a log and a queue, and it is what makes replay possible.
Answer all three to see your score.