Where data is born: sources and the overnight pickup

Every source at Daybreak - and very likely at your company - is one of these:

• The app database is where the business officially happens. When order #8412 was placed, the checkout app wrote a row: who, what, $42, 11:41pm. Orders, customers, subscriptions all live here. This is the most trusted family - the app itself would break if these records were wrong.
• Event streams are the behavioral confetti around the business: every click, tap, page view, and abandoned cart. Enormous volume, low ceremony - a single visitor can generate hundreds of events before buying once. Great for "how do people browse?", never the place you count revenue.
• Files are the unglamorous workhorses: the roaster emails a CSV of bean costs, a logistics partner drops a spreadsheet of delivery times. If it arrives as an attachment, it is a file source.
• Third-party APIs are other companies' systems that hold your data: Stripe knows the payments, the ad platforms know the campaign spend, the support tool knows the complaints. You do not own these databases - you ask politely through an API, which is just a service window for software.

There is a fifth thing you will find in every inventory, and it deserves its own warning label: the spreadsheet a human fills in by hand. Regional targets someone updates monthly, a price list marketing "keeps current", a tab of store openings. Technically it is a file source. Practically it is the least reliable source in the building: no system witnessed the facts, a typo looks identical to a truth, columns get renamed on a whim, and the one person who understands it goes on holiday.

The point is not to mock it - sometimes a hand-kept sheet is genuinely the only home a fact has (targets and budgets are decisions, not events, so no app records them). The point is to flag it: when a dashboard number looks odd, the hand-maintained source is the first suspect, because it is the only farm with no machinery guaranteeing what got planted. Today's exercise asks you to mark yours with a warning flag.

Choice one: how much to haul. A full copy re-hauls the entire order history every night - simple, bulletproof, and increasingly slow as history grows. An incremental copy takes only new and changed rows since the last pickup - the design choice we spotted in session 1's 2am stop. Nearly every mature pipeline goes incremental, which quietly creates a new job: deciding what counts as "changed". That decision is invisible until the night it is wrong.

Choice two: who builds the truck. Copying data out of Stripe correctly - payments, refunds, disputes, currency quirks - is fiddly work, and thousands of companies need the exact same truck. So companies exist that sell pre-built ones:

• Fivetran - decoded: a company selling ready-made pickup trucks for hundreds of common sources. You point it at Stripe and your warehouse; the copying, scheduling, and retrying is their problem.
• Airbyte - decoded: the same job, with an open-source flavor teams can run and modify themselves.

One quiet danger to file away: the truck's route assumes the farm stays where it was. If Daybreak's app team renames a column, drops a field, or changes what "order status" can say, the 2am pickup may keep running and happily deliver something subtly different - or nothing at all - without anyone shouting. This is called a schema change, it starts at the source, and it is the single most common way pipelines break. Session 5 is where we watch it happen and learn how teams catch it. For now, just remember: the farm can move, and the truck will not always notice.

Batch means scheduled runs - the truck departs at set times, hauls everything that accumulated, and rests. In practice that schedule is often nightly or hourly, tuned to what the business actually needs. It is simpler to build, cheaper to run, and easier to fix, because there are quiet hours between runs when nothing is moving.

Streaming means continuous - each event rides the belt within seconds of happening. That immediacy costs real money and real engineering: the belt never stops, so neither can the people responsible for it. The honest test is a single question: would a decision change if this number were seconds fresh instead of hours fresh?

• Yes, stream it: fraud detection (a stolen card must be blocked mid-purchase, not tomorrow), live operations like delivery tracking or inventory that sells out in minutes.
• No, batch is honestly fine: the Monday revenue dashboard. Daybreak's executives read it at 9am with coffee; whether order #8412 arrived at 2:15am or 2:15pm Sunday changes nothing they decide. Daybreak runs batch, and should.

A script for the meeting where this comes up: "It could be - streaming is a real option. It costs meaningfully more to build and to keep healthy, so the question is which decision we would make differently with second-fresh numbers. If we can name one, let's take it to the data team. If we can't, the overnight batch is giving us the same decisions for a fraction of the cost." This reframes the conversation from speed as status to freshness as a purchase - and data teams will love you for it. Session 4 picks this thread up when we talk about what freshness each consumer of the pipeline actually needs.