Where we are in the track
You have the lifecycle (a1) and you know your sources are systems you do not own (a2). Now comes the first big spending decision that lands on a leader's desk: how fresh does the data need to be? For Daybreak, the honest answer for most reporting is "yesterday's numbers are fine by breakfast." But every vendor in the market is selling "instant." This session hands you the frame to tell the two apart - to know when a few hours old is plenty, when seconds actually change a decision, and how to spot the expensive answer to a cheap question before you have signed for it.
Two clocks 8 min live
There are really only two ways to move data, and they run on different clocks. Batch gathers data into scheduled chunks - every night, every hour - and processes each chunk as a unit. Streaming processes each event the moment it happens, in a continuous flow that never stops. Batch is simple and cheap and is the default for a reason. Streaming is powerful and materially more complex and expensive. Almost every architecture argument you will hear is really an argument about which clock, so it is worth seeing them side by side.
LiveBatch, the sensible default3 min▶
Batch means: wait, gather, process together. Daybreak pulls the day's orders at midnight, reshapes them, and by breakfast the finance and growth teams have yesterday's clean numbers. It is the default for good reasons:
- Simple to build and reason about. A chunk either processed or it did not. When something goes wrong, you re-run last night's batch. There is a clear before and after.
- Cheap to run. The compute wakes up, does its job, and sleeps. You are not paying to keep a system alive every second waiting for the next event.
- Enough for most decisions. Finance, planning, weekly growth reviews, board reporting - none of these change because a number is one hour fresher versus one day. The decision cadence is slower than the data cadence either way.
If someone cannot articulate why daily or hourly batch is insufficient, the answer is batch. That is not being cheap; that is matching the tool to the decision.
Self-studyStreaming, and why it costs more3 min read▶
Streaming means the system is always on, reacting to each event as it arrives. That capability is real and sometimes essential - but the "always on" is exactly what makes it expensive. You now run infrastructure that never sleeps, and you have to answer hard questions that batch simply sidesteps: what happens when events arrive out of order, or arrive twice, or arrive late after you already reported the total? Each of those is a genuine engineering problem, and solving them well takes more specialized people and more always-running infrastructure. Industry rule of thumb: expect roughly 5 to 10 times the engineering and infrastructure cost of the equivalent batch pipeline. That premium can be entirely worth it - but only for the decisions that actually need it.
When real-time earns its cost 6 min live
The test is simple and it is the only one that matters: does a fresher number change a decision inside the window streaming buys you? If a human or a system will act differently because the data is seconds old instead of hours old, streaming may earn its premium. If nothing happens any sooner, you are paying 5-10x for a number that just sits there looking impressive.
| Use case | Batch or streaming | Why |
|---|---|---|
| Payment fraud detection | Streaming | a blocked transaction is worthless one hour later; the decision is in the moment |
| Live operations dashboard | Streaming | an ops team is actively steering something right now and needs current state |
| In-session personalization | Streaming | recommending the next coffee only helps while the customer is still on the page |
| Daily / weekly reporting | Batch | the decision cadence is days; fresher-by-the-hour changes nothing |
| Finance + month-end close | Batch | accuracy and auditability matter far more than latency |
| Planning + forecasting | Batch | models run on history; seconds of freshness are irrelevant |
LiveSkepticism about "real-time" and "zero-ETL"3 min▶
Two phrases deserve a raised eyebrow in any vendor pitch:
- "Real-time." Ask: real-time to do what? A dashboard that updates every second, watched by nobody at 3am, is not value - it is cost with a light show. Real-time is only real value when something acts on it inside the window.
- "Zero-ETL." The work of moving and reshaping data does not vanish because a slide says "zero." It moves - into the vendor's platform, into your bill, or into a lock-in you feel later. The transformation still happens; someone still pays for it. Treat "zero" as "hidden," and ask where it went.
Six months of streaming for a report read at noon. A team was convinced a key report needed to be real-time and spent six months building a streaming pipeline to deliver it. After launch, the analytics on the report itself showed the truth: almost every viewer opened it once, mid-morning, over coffee. Nobody looked before 11am, and nobody looked twice. A nightly batch would have served every single reader perfectly, at a fraction of the cost and complexity. The six months bought a capability no decision ever used. The question that would have saved it: "who acts on this, and when?"
Self-studyThe honest split, in one line3 min read▶
Notice the pattern in the table. Streaming wins where a decision is made in the moment and being late makes it worthless - fraud, live ops, in-session personalization. Batch wins everywhere the decision cadence is slower than the data cadence, which is most of reporting, all of finance, and all of planning. The mistake is almost never "we used streaming where batch would do" being caught early - it is caught six months and a large bill later. So the default posture is batch, and the burden of proof sits on whoever wants to spend the 5-10x.
The leader's decision frame 6 min live
Put it all together into a posture you can hold in any planning meeting: start batch, and earn your way to streaming only where latency changes a decision. And know that it is not a binary - there is a sensible middle road that gets you most of the freshness for a fraction of the streaming cost.
LiveMicro-batch, the middle road3 min▶
Most leaders think the choice is "yesterday's data or real-time," but there is a whole ladder in between. Micro-batch runs the same simple, cheap batch machinery, just far more often - every few minutes instead of every night. You get data that is minutes old for a tiny fraction of true streaming's cost and complexity, and none of streaming's hard out-of-order and duplicate-event problems. For a huge share of "we need it fresher" requests, micro-batch is the honest answer: it satisfies the real need without buying the whole streaming stack.
Take it back to your team - this week ◐ 20-30 min total
- List every "real-time" or "live" data request currently in flight in your org. For each, write who acts on it and how quickly - and be honest where the answer is "nobody, really."
- Pick your most important report and ask what would break if it were one hour old, or one day old. Notice how often the honest answer is "nothing."
- Find one place a vendor has pitched "zero-ETL" and ask your team where the transformation work actually went - into the platform, the bill, or a lock-in.
- For one genuinely fresh-data need, check whether micro-batch (every few minutes) would satisfy it before anyone commits to true streaming.
- Adopt the one question - "who acts on this, how fast, what breaks if it's an hour old?" - as a standing gate on freshness requests. Bring a real example to session a4.
Official sources covered
This session teaches the leadership-level thinking from the DeepLearning.AI Data Engineering Professional Certificate (Joe Reis) and Reis & Housley's Fundamentals of Data Engineering, kept out of the tooling and focused on the spending decision a leader actually owns. Certificates, AWS labs, and code stay on the official platforms. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Roughly how much more do streaming pipelines cost to build and run versus equivalent batch?
Streaming means always-on infrastructure and harder problems (out-of-order, duplicate, late events), so the industry rule of thumb is roughly 5-10x the cost of the equivalent batch pipeline. Worth it sometimes - but only when a decision needs it.
2 · What is the right test for whether real-time earns its premium?
If a human or system acts differently because the data is seconds old instead of hours old, streaming may be worth it. If nothing happens any sooner, you are paying 5-10x for a number that just sits there.
3 · A team wants data "fresher than nightly" for a report checked once each morning. Best first move?
The choice is not binary. Micro-batch runs the same cheap batch machinery far more often, delivering minutes-old data without streaming's cost or its out-of-order and duplicate-event problems. For a report read once a morning, it is almost certainly plenty.