How this track works
Six sessions, built for people who fund and steer data work, not write it. We use one running example throughout: Daybreak, a direct-to-consumer coffee-subscription brand whose operational database powers the app and whose warehouse powers the reporting. Across the leader track you will meet source systems (a2), the batch-versus-streaming cost question (a3), build-versus-buy (a4), storage economics (a5), and how to shape the org and roadmap (a6). Today is the map of the whole territory. You will leave able to tell data engineering apart from the crafts it is constantly confused with - and to explain, in one sentence, why it deserves budget before the AI does.
The lifecycle, in plain language 8 min live
Joe Reis and Matt Housley, in Fundamentals of Data Engineering, framed the whole discipline as a lifecycle rather than a pile of tools - and that framing stuck because it outlives every tool that comes and goes. Five stages carry data from where it is born to where it is used. Beneath them run a set of undercurrents - concerns present at every stage. As a leader you do not need the tools. You need this shape, because it is what you are actually funding.
LiveThe five stages, in one breath3 min▶
Data is generated in source systems the data team usually does not control - Daybreak's app database, a payment provider's API, a stream of website clicks. Engineers ingest it, copying it out on some schedule. They store it in a shape built for what comes next. They transform it into something trustworthy and useful. And they serve it to the consumer - the warehouse, a dashboard, a model. That is the entire job, repeated at every scale from a laptop to a petabyte.
- The stages are stable, the tools are not. The vendor logos on the roadmap (Spark, Airflow, Kafka, dbt, Fivetran) churn every few years. The five stages have held for decades. Fund the stages, not the logo.
- Your team rarely owns generation. Source systems belong to other teams or outside vendors. A huge share of the work - and the pain - is coping with sources nobody can change (session a2).
- Serving is the whole point. Nobody thanks a team for a clean pipeline nothing consumes. For Daybreak, serving means feeding the warehouse the finance and growth teams live in.
Why the lifecycle beats a tool list. A team hired around "Spark and Airflow experience" rebuilt everything two years later when the stack shifted - but the lifecycle they were really running never moved an inch. Leaders who describe the work in stages hire people whose skills survive the next tool change. Leaders who hire for tools keep re-hiring.
Self-studyThe undercurrents, and which are ours3 min read▶
Beneath the five stages run six undercurrents: security, data management, data architecture, DataOps, orchestration, and software engineering. They are not stages - they are concerns present at every stage, which is exactly why they are easy to under-fund. Two of them have their own home elsewhere, and knowing the line saves you from paying twice:
- DataOps + orchestration → learn-dataops. Scheduling, testing, monitoring, alerting, infrastructure. Building pipelines and operating them reliably are different budgets and often different people.
- Modeling + warehouse design → learn-data-warehouse. Star schemas, slowly-changing dimensions, marts. Data engineering delivers clean data to the warehouse door; the warehouse craft models what happens inside.
- What stays here. Getting data out of sources, moving it, choosing formats and storage, and the compute that transforms it. The engineering of the pipe itself.
What data engineering owns 6 min live
Data engineering, data warehousing, and DataOps blur together in job ads and headcount plans, but they are three distinct crafts. Confusing them is the single most common - and most expensive - leadership mistake in this space. Get the lines right and you hire the right people, fund the right work, and know exactly where one team hands off to the next.
| Craft | Owns | In Daybreak's world |
|---|---|---|
| Data engineering (here) | ingest, move, store, transform-compute | getting Daybreak's source data reliably to the warehouse door |
| Data warehousing | modeling: star schema, SCD, marts | shaping that data into the facts finance and growth query |
| DataOps | orchestrate, test, monitor, deploy | keeping the pipeline running, and paging someone when it breaks |
LiveThe one-sentence test3 min▶
When a request lands on your desk, ask which verb it really is. "Get this data here" is data engineering. "Shape it into facts and dimensions" is warehousing. "Make sure it runs at 2am and pages someone if it fails" is DataOps. Most real roles blend all three, but the verb tells you which skill you are hiring and which team owns the outcome. Say the sentence out loud in your next planning meeting and watch how fast a muddled request sorts itself out.
Self-studyThe confusion, and what it costs3 min read▶
The most common failure is treating "we need data engineers" and "we need dashboards" as the same request. They are not. Dashboards are a serving-stage artifact that sit on top of a warehouse, which sits on top of pipelines. Asking a data engineer for a dashboard, or asking a BI analyst to build ingestion, produces a frustrated hire and a half-built system either way.
The mis-hire that cost a year. A retail exec, told "we need data engineers," hired two - then handed them a backlog of dashboard requests. The engineers, hired to build pipelines, spent months on BI work they were not suited to; the pipelines the company actually needed never got built. When the reporting finally broke at quarter-end, there was no foundation underneath to fix it. One conflated word, one wasted year. The fix was not more hiring - it was naming which craft each request belonged to before writing the job description.
Why it is a strategic investment 6 min live
Here is the uncomfortable truth: a garbage pipeline does not fail loudly. It quietly makes everything downstream confidently wrong. The dashboard still renders. The model still trains. The board deck still has a number in it. It is just the wrong number, and nobody knows. That is why the data foundation is not a cost centre to squeeze - it is the thing every analytics and AI ambition is standing on.
LiveThe cost of not investing3 min▶
When the foundation is under-funded, the work does not stop - it goes underground. That is worse. You get:
- Shadow pipelines. An analyst wires up a fragile export script on their laptop because the real pipeline never got built. It works until they leave, then nobody can fix it.
- Spreadsheet exports everywhere. The same number gets pulled five different ways by five teams, and now there are five versions of "revenue" in the building.
- Nobody trusts the numbers. The most expensive failure state of all. Once leadership stops believing the dashboard, every decision reverts to gut feel and the entire data investment is dead weight.
Monica Rogati's AI Hierarchy of Needs makes the point crisply: reliable data plumbing sits at the base, analytics in the middle, and AI at the very top. Companies want to buy the top. You cannot. You have to build up.
Self-studyConfidently wrong, downstream3 min read▶
The reason a weak pipeline is so dangerous is that its failures are silent and they compound. A missed row in ingestion becomes a slightly-off daily total, which becomes a wrong month, which becomes a model trained on skewed history, which becomes a personalization engine quietly recommending the wrong coffee to the wrong customer. At no point does anything throw an error. The chain just gets more confidently wrong the further it travels.
Take it back to your team - this week ◐ 20-30 min total
- Name the five lifecycle stages from memory, then pick one data product your team relies on and map it onto them. Which stage is missing, manual, or fragile?
- Find one recent data request that stalled or disappointed. Say the one-sentence test over it out loud - was it engineering, warehousing, or DataOps? Did you staff it as the right craft?
- Observe where the shadow pipelines live in your org. Whose laptop, whose spreadsheet, whose "I'll just export it" is quietly holding a number together? List three.
- Ask your team, honestly: would you bet a board decision on the top three numbers we report? Note where the answer is a hesitation, not a yes.
- Before you approve the next AI or analytics initiative, locate it on the hierarchy of needs and ask what base layer it assumes exists. Bring the gap to session a2.
Official sources covered
This leader track teaches the executive-level thinking from the DeepLearning.AI Data Engineering Professional Certificate (Joe Reis) and Reis & Housley's Fundamentals of Data Engineering, translated out of the tooling and into the decisions a leader actually makes. Certificates, AWS labs, and code stay on the official platforms. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · What are the five stages of the data engineering lifecycle, in order?
Data is generated in source systems, ingested (copied out), stored, transformed, then served to a consumer. The stages outlast every tool that implements them - which is why leaders fund the stages, not the logos.
2 · A request says "shape our source data into the revenue facts finance queries." Which craft owns it?
"Shape it into facts and dimensions" is the warehousing verb. Data engineering gets the data to the warehouse door; the warehouse craft models what happens inside. The one-sentence test tells you who to staff.
3 · Why frame data engineering as a lifecycle rather than a set of tools?
Tools turn over every few years; the generation-to-serving lifecycle has held for decades. Hire and fund against stages and your investment ports across every tool change instead of being rebuilt with it.