Where the artifact stands
Fresh to the second, 78.9% accurate, wrong upward. The over-count is not noise - it is the signature of at-least-once delivery, and it has a signature fix. By the end of tonight the dashboard is de-duplicated, the accuracy reads 89.1%, and the remaining error will have flipped direction in a way worth staring at.
★ What you walk out with: the three delivery promises and the failure each one accepts, the precise (narrow) sense in which exactly-once exists, the idempotency habit that protects any consumer regardless of broker guarantees - and a halfway scorecard that says, honestly, what is fixed and what is not.
Three promises, three failure modes 8 min live
Every delivery system chooses what to do when an acknowledgment goes missing. There are only three coherent choices, and each accepts a different failure. Kafka's design docs state the house position without hedging: Kafka guarantees at-least-once delivery by default.
Live The retry is rational 5 min ▶
Put yourself in the producer's shoes. You sent order #8412 and no acknowledgment came back. Two worlds are now indistinguishable from where you stand: the message died on the way in, or it landed fine and the ACK died on the way back. The Kafka design docs describe the pre-0.11 reality plainly: a producer that missed its ack "had little choice but to resend the message." That resend is not sloppiness - it is the only move that never loses an order.
- At-most-once: disable retries (and commit before processing on the consumer side). No duplicates, ever - because you have agreed to lose data silently instead. You traded duplicates for silent loss.
- At-least-once: retry until acknowledged. Nothing is lost - and the price is the 15 double-counted retries currently inflating the dashboard.
- Exactly-once: the promise everyone wants, available under conditions we will draw precisely in Part 2.
Notice which failure each policy chooses. There is no fourth option where the network stops eating acks.
Payment webhooks are delivered at-least-once for exactly this reason: the sender cannot tell whether your server processed the charge notification or just failed to say so, and a payments product that silently drops one is worse than one that repeats itself. Which is why every serious webhook consumer dedupes by event id before acting - the receiving side assumes duplicates as a fact of life.
Self-study The consumer makes the same choice, with a commit 3 min ▶
Delivery semantics are not only a producer story. A consumer's promise is set by one ordering decision: when does it commit its offset relative to doing the work?
- Commit BEFORE processing = at-most-once. If the consumer crashes between the commit and the work, the offset has already moved past events that were never processed. They are skipped forever.
- Commit AFTER processing = at-least-once. If the crash lands between the work and the commit, restart re-reads and re-processes those events. Nothing skipped - some things done twice.
Same lost-ack logic, one layer up: the consumer cannot tell "I did this and failed to record it" from "I never did this" - so it must pick which mistake to make. The dashboard consumer commits after processing, which is the right choice, and is precisely why it needs the dedupe you are building tonight.
Exactly-once, narrowly 7 min live
Exactly-once is real, it is engineering rather than marketing, and it holds inside a boundary you must be able to draw. Drawing it wrong is how teams ship over-counting dashboards while their architecture diagram says "exactly-once" in a confident font.
Live The machinery, and the boundary 4 min ▶
Since Kafka 0.11, two pieces of machinery exist. The idempotent producer: the broker assigns each producer an id and each message a sequence number, and dedupes resends as they arrive - the retry still happens, but only one copy lands in the log. And transactions: a producer can write to several partitions atomically, all visible or none, with consumers reading read_committed to see only completed transactions.
Now the boundary, in the docs' own words: "the simplest way to get exactly-once semantics from Kafka is to use Kafka Streams" - that is, a consume-process-produce pipeline whose input offsets and output records commit in one transaction. Exactly-once is a property of that closed Kafka-to-Kafka loop, not of your whole system. The moment your consumer writes to an external dashboard, database or API, the broker's guarantee has ended and your sink's discipline begins. DDIA's framing is the one to memorise: "Idempotent operations can be an effective way of achieving exactly-once semantics with only a small overhead." That external-sink idempotency is literally the lever you flip in tonight's build-along.
Dedupe by a stable key is the universal fix precisely because it makes no assumptions upstream: keep the set of order ids already counted, and counting #8412 a second time becomes a no-op. It works whether duplicates come from producer retries, consumer restarts, or a replay you triggered on purpose - which session 5 will.
Self-study Schema discipline in one breath 3 min ▶
Deduping by order id quietly assumed something: that every producer and every consumer agrees on what an order event looks like and where the id field lives. In production that agreement is not a wiki page - it is enforced by a schema registry (Confluent's Schema Registry is the standard tool in the Kafka world), which producers and consumers check event shapes against so a renamed field breaks loudly at publish time instead of silently at 2am.
Confluent runs a whole Schema Registry 101 course on the practice; this course names the discipline and moves on rather than teaching the tool. The takeaway that matters here: your idempotency key is part of your event's contract. Treat it like one.
Dedupe the stream ★ 22 min · everyone builds
Same simulator, second lever. The consumer is about to start remembering which order ids it has already counted - and the error left behind will teach you something the fix itself cannot.
Rebuild last session's state: Stream it ON. Confirm the headline reads 78.9% with 15 retries double-counted, dashboard at $4,073. This is at-least-once delivery doing exactly what it promised - nothing lost, some things twice.
Flip Idempotent consumer ON. The retries counter drops to 0, accuracy climbs to 89.1%, and the dashboard reads $3,485. Every duplicate order id is now counted once, no matter how many times it arrived.
Now stare at the direction of the remaining error, because it flipped. Before: $4,073, OVER the $3,723 truth. After: $3,485, UNDER it. The double-counts had been partly masking the late orders that have not arrived or were bucketed wrong - two wrongs almost cancelling into a total that looked closer than either error alone. That is not correctness; it is luck wearing its costume. Deduping removed one wrong and exposed the other honestly.
Close on the halfway scorecard below. Two boxes ticked, three open - and the three open ones are all about TIME, which is exactly where sessions 4 and 5 are aimed.
| Halfway check (end of session 3) | Status |
|---|---|
| Stream is on | ✓ fresh to the second |
| Retries deduplicated | ✓ counted once, by order id |
| Hourly buckets honest | ✗ still bucketed by arrival time |
| Stragglers handled | ✗ late orders still missing or misplaced |
| Day converges to truth | ✗ no path to 100% yet |
| Verdict | Fresh and de-duplicated. Halfway is a real place; do not ship it as done. |
This is the forgiving halfway check, not the exit exam. It asks whether the stream is clean; it does not yet ask whether you can operate it. The strict final scorecard waits in session 6 with harder rows of its own - alerting on lag, fitting the cost, and proving the replay path.
Before session 4 15 min
- Pick one consumer in your organisation - a webhook handler, a stream job, anything that reacts to events. Answer in writing: if the same event arrived twice, would the result be wrong, and would anyone notice?
- Name its idempotency key. If you cannot name one, that is the finding - write down what stable id COULD serve, the way order id serves the coffee shop.
- Look at the simulator's remaining 10.9% error and write one sentence on where that money is. You will check your answer against session 4's chart.
Three questions
1. Why is at-least-once the default, rather than a bug to be fixed?
When the ack goes missing, "it never arrived" and "it arrived but the ack died" look identical to the producer. Not resending risks silent loss; resending risks a duplicate. At-least-once picks the recoverable failure - duplicates can be deduped, lost orders are just gone.
2. How does the idempotent producer (Kafka 0.11+) stop resends from duplicating in the log?
The dedupe is broker-side: producer id plus sequence number lets the broker recognise a resend and keep exactly one copy. The retry still happens - it just stops leaving a mark. Consumer-side filtering is a different (also necessary) layer: it is tonight's lever.
3. Your Kafka Streams job runs with exactly-once semantics, then a consumer writes hourly totals to a Postgres dashboard. What is true?
The docs' own boundary: the simplest exactly-once is Kafka Streams - a closed consume-process-produce loop committing offsets and output together. An external sink sits outside that transaction, so the consumer must make its writes idempotent, which is DDIA's small-overhead route to the same guarantee.
Answer all three to see your score.