Where we are in the track
Last session gave you the map: the five-stage lifecycle and where data engineering ends and its siblings begin. This session zooms into the very first stage - generation - because it is where most pipeline pain is actually born. For Daybreak, the source is the operational database behind the app plus a handful of outside services. You will leave understanding why "just pull it from the source" is never as simple as it sounds, and knowing the three questions that separate a team that sleeps at night from one that gets paged when a system they do not own quietly changes.
Where data is born 8 min live
Before data is anything useful, it is an event in some operating system - a subscription started, a card charged, a page clicked, an error logged. Those systems come in a handful of recognizable shapes, and each has a different cadence, structure, and reliability. A leader does not need to operate them, but recognizing the zoo tells you why one source is easy and another is a constant fire.
LiveThe shapes, and why they differ3 min▶
Each source shape behaves differently, and the difference is what makes ingestion hard:
- App databases (OLTP). The transactional store behind the product - Daybreak's orders, customers, subscriptions. Tuned for fast writes, not for the heavy reads analytics wants. This is the primary source and the star of session a3's tension.
- REST APIs. How you reach systems you truly cannot see inside - a payment processor, an email tool. You get what they choose to expose, at the rate they allow, in the format they decide.
- Event and click streams. A firehose of small events as they happen. High volume, no natural stopping point - the source that pushes teams toward the streaming question in a3.
- Logs and files. The unglamorous majority. Server logs, nightly CSV exports, a spreadsheet a partner emails. Messy, but often the fastest data to get.
ACID is why the app trusts its database. Daybreak's operational database guarantees that a subscription charge and the order it creates either both happen or neither does (the "A" and "C" in ACID). That guarantee is precious for running the business - and it is exactly why you should not run heavy analytics queries against it, because those queries fight the writes the business depends on. A leader does not need the acronym. They need to know: the system that runs the business is not the system you report from.
Self-studyCadence, structure, reliability3 min read▶
Three dimensions tell you how hard a source will be to live with. Cadence: does it arrive in tidy nightly batches or as a never-ending stream? Structure: neat rows and columns, or free-form JSON and text that changes shape? Reliability: is it a managed service with an uptime guarantee, or a script on someone's laptop? A source that is streaming, semi-structured, and owned by an outside vendor is three kinds of hard at once. One that is batch, tabular, and internal is the easy case. Knowing which you are dealing with sets realistic expectations - and realistic timelines.
The systems you don't own 6 min live
Here is the hard truth that surprises every leader new to this: your data team almost never owns the source. It belongs to the product team, or a vendor, or an ops group with entirely different priorities. They will change a column name, deprecate an API, or throttle your requests - and they have no obligation to tell your data team first. Everything downstream depends on systems your data team cannot control. That is not a bug in how you are organized. It is the permanent condition of the work.
LiveSchema drift, rate limits, downtime3 min▶
Three ways an un-owned source turns on you:
- Schema drift. The product team renames
signup_datetocreated_atin a routine release. Their app keeps working. Every downstream report that expected the old name silently breaks or, worse, quietly returns nothing where it used to return data. - Rate limits. A vendor API allows so many calls per minute. Your pipeline hits the ceiling during a busy period and starts dropping data - not because anything is broken, but because you do not control the tap.
- Downtime. The source goes down for maintenance on its own schedule. Your pipeline was mid-pull. Now you have half a day's data and a decision to make about the other half.
The rename that broke every report. A product team shipped a normal release that renamed one column in the orders table. No announcement to the data team - why would there be, the app was fine. Overnight, the ingestion job kept running without error but pulled an empty column. Every revenue dashboard showed a plausible-looking but wrong number for two days before anyone noticed. The cost was not the fix, which took an hour. It was the two days of decisions made on bad numbers, and the trust that took months to rebuild. The lesson: a source you do not own can break you without anyone doing anything wrong.
Self-studyWhy "just query the prod database" hurts both sides3 min read▶
The tempting shortcut is to point the analytics tools straight at the production app database. It seems free - the data is right there. It is not free. Heavy analytics queries compete with the reads and writes the live app needs, so a big report can slow down or even stall the product for real customers. And the app database is shaped for transactions, not analysis, so the queries are painful to write and slow to run. This is the operational-versus-analytical split that the sibling learn-data-warehouse course is built around: the system that runs the business (OLTP) and the system you analyze in (OLAP) should be two different places, and the pipeline you fund is what safely moves data from one to the other.
What a leader should ask 6 min live
You will not write the ingestion code, and you should not try. But three questions, asked in a planning meeting, will tell you more about how exposed your data operation is than any architecture diagram. If the answers are vague, you have found your next investment before it finds you as an outage.
LiveThe three questions, unpacked3 min▶
- Do we have documented contracts with source owners? Not a formal legal document - a shared, written understanding of which fields matter and how change is communicated. If the answer is "we just know," you are one release away from a silent outage.
- Who gets paged when a source changes? If nobody, then you find out about breakage the way the retail team did - two days later, at quarter-end, from an angry stakeholder. (The paging itself lives in the learn-dataops course; the question of whether it exists is a leadership question.)
- Are we reading replicas, not prod? A read replica is a safe copy that analytics can hammer without slowing the live app. If your team is querying the production database directly, you are risking the customer experience every time a report runs.
Take it back to your team - this week ◐ 20-30 min total
- List the top five source systems your reporting depends on. For each, note its shape (database, API, stream, file) and whether your team owns it or someone else does.
- Ask the three leader questions about your single most important source - documented contract, who gets paged, replica not prod. Write down where the answer is a shrug.
- Find out, honestly, whether any analytics query in your org points straight at a production app database. If yes, that is a risk to name out loud, not to hide.
- Identify one source owned by an outside vendor and ask what happens to your reports if that vendor changes their API next month. Notice whether anyone has thought about it.
- Recall the last time a report "suddenly broke." Trace it back - was it really a source change nobody was warned about? Bring the story to session a3.
Official sources covered
This session teaches the leadership-level thinking from the DeepLearning.AI Data Engineering Professional Certificate (Joe Reis) and Reis & Housley's Fundamentals of Data Engineering, kept out of the console and focused on the decisions a leader owns. Certificates, AWS labs, and code stay on the official platforms. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · What is the defining reality of source systems for a data team?
Source systems belong to product teams, vendors, or ops groups with their own priorities. They change, rate-limit, and go down on their own schedule - and that permanent condition is the root of most pipeline pain.
2 · Why is pointing analytics queries straight at the production app database a bad idea?
The app database (OLTP) is tuned for fast transactions the product depends on. Heavy analytics queries fight those, risking the customer experience - which is why the pipeline moves data to a separate analytical home (the learn-data-warehouse topic).
3 · A report that ran fine for a year suddenly returns wrong numbers, with no error. Most likely cause?
A silent wrong-number failure is the classic signature of schema drift: the source owner shipped a routine change (a renamed column), the app kept working, and the pipeline kept running while quietly pulling the wrong thing.