learn-data-warehouse-with-phoebe / Leader session 6 of 6
Learn Data Warehouse with Phoebe · Leader track · Session 6 of 6

The warehouse in the AI era

Every AI ambition in your company - the churn model, the copilot, the "agent that answers business questions" - lands on the same foundation: the warehouse. In this final session we look at who the warehouse serves beyond dashboards, when real-time is worth its price tag, and the crawl-walk-run roadmap that turns everything from a1-a5 into a plan. Daybreak wants an AI assistant for its support team; whether that works is decided in the warehouse.

🔵 Leader track Leaders: CxO · VPs · managers No code - thinking mode 45 min
0-3 · Welcome 3-20 · Serving beyond dashboards 20-42 · Real-time + roadmap 42-45 · Q&A
Part 0

How this session works

Five sessions ago the warehouse was a copy of the app database for dashboards. Today we finish the picture: the same governed tables now feed machine learning, AI agents, and the operational tools your teams live in. The stakes changed with the customers - a wrong number on a dashboard misleads one meeting; a wrong number inside an AI system misleads every conversation it has, confidently. Then we close the track with a roadmap you can put on one slide.

Live - discussed in session Self-study - read after For decision-makers
★ What you walk out with today A map of the warehouse's four modern customers, an honest calibration on real-time (most decisions are fine at daily; streaming costs 5-10x), the crawl-walk-run roadmap for your own company, and the quarterly questions that keep it moving.
Part 1 · covers DLAI Data Engineering C4 M4 - serving data for analytics AND machine learning

The warehouse's new customers 8 min live

Joe Reis's data engineering curriculum makes a point most AI strategy decks skip: serving is the last mile of data engineering, and analytics is only one of its destinations. The same warehouse now serves four customers - and the newest ones are the least forgiving of bad data, because they never pause to ask "does this look right?"

The warehouse governed · fresh · defined one set of numbers, tested before serving 1 · Dashboards & BI the classic customer - humans reading charts 2 · ML features churn scores, forecasts - models eat warehouse tables 3 · AI agents & RAG assistants answering from your governed numbers 4 · Reverse ETL to tools scores pushed back into CRM, ads, support desks Customers 2-4 never ask "does this look right?" - a garbage warehouse makes AI confidently wrong.
🔍 Click to zoom - one governed warehouse, four consumers, rising stakes down the list
LiveWhy AI raises the stakes on data quality4 min

A dashboard has a built-in safety net: a human looks at it. When the churn number looks weird, someone says so before the board sees it. AI systems remove that net. A model trained on mislabeled subscriptions learns the mistake. An agent answering from a warehouse where "revenue" still has two definitions (the a3 problem) picks one at random and answers fluently. The failure mode of AI on bad data is not silence - it is confident wrongness at scale.

  • ML features: models consume warehouse tables as ingredients - "days since last order", "orders per month". Wrong ingredients, wrong predictions, discovered months later in the metrics.
  • AI agents and RAG: assistants ground their answers in your data. The grounding is only as good as the definitions and freshness underneath - a1 to a3 were secretly your AI-readiness sessions.
  • Reverse ETL: warehouse scores flow back into CRM and support tools, where frontline staff act on them directly. The warehouse stops describing the business and starts steering it.
Real world

The copilot that failed its demo. A retail exec team piloted an AI assistant over their data. First question: "what was revenue last quarter?" The agent answered with the finance definition; the CFO had the product-dashboard number in hand; the two differed by the familiar refund gap. The pilot was paused - not because the AI was weak, but because the warehouse had never settled one definition of revenue. Six months of definition work later, the same assistant passed. The AI project was a warehouse project wearing a costume.

Self-studyViews and materialized views: how serving actually happens2 min read

One mechanism from the DLAI curriculum worth knowing by name. A view is a saved question - "active subscribers" defined once, referenced everywhere, so every consumer inherits the same definition. A materialized view is that same question with the answer pre-computed and stored (a5's lever 3). Together they are how one definition serves all four customers consistently: the dashboard, the model, and the agent all read "active subscribers" from the same governed shelf. Builder session b7 constructs these for Daybreak.

Self-studyDaybreak's four customers, concretely3 min read

The abstract map, made specific with our running case:

  • Dashboard: the founder's Monday view - revenue by month, churn by cohort. The classic customer from a1, still the workhorse.
  • ML feature: the churn model reads "days since last order" and "orders per month" from warehouse tables to score every subscriber weekly. If order dates are wrong upstream, the model quietly learns nonsense.
  • AI agent: the support assistant answering "what plan is this customer on, and what did they last order?" from governed tables. Wrong data here reaches a customer's ears in seconds.
  • Reverse ETL: the churn scores flow back into the support desk, so agents see "high churn risk" next to the ticket. The warehouse now steers frontline behavior, not just meetings.

One warehouse, four exits. Every exit inherits whatever quality - or whatever mess - lives in the tables.

Part 2 · covers 365DS real-time data warehousing

Real-time, priced honestly 8 min live

Every vendor demo now ends with "and it's real-time". Here is the executive calibration: real-time is a genuine capability with a genuine price, and most business decisions are perfectly served by yesterday's data. The question is never "can we be real-time?" - it is "which decision, exactly, changes if this number is 5 minutes old instead of 24 hours old?"

LiveThe freshness ladder: match latency to the decision4 min

Think in three rungs, each roughly an order of magnitude more expensive to operate than the one below:

  • Daily batch (the workhorse): board reports, finance, marketing reviews, churn models, almost every dashboard. The warehouse loads overnight; the business runs on it. This rung covers the large majority of decisions in a typical company.
  • Hourly / near-real-time: operational monitoring - today's orders, campaign spend pacing, inventory alerts. Worth it where someone acts within the hour.
  • Streaming (seconds): fraud detection, live personalization, logistics tracking. Real engineering, real on-call, real money - typically 5-10x the cost of batch for the same data. Buy it per use case, never as a platform-wide default.
The one-question filter For any real-time request: "who acts on this within minutes, and what do they do differently?" A named person with a named action justifies the rung. "It would be nice to see" does not - that is a daily dashboard wearing a streaming price tag.
Live"Zero-ETL" and other claims to hear skeptically4 min

Two phrases from the 2026 vendor floor deserve a leader's raised eyebrow:

  • "Zero-ETL": usually means the vendor moves raw data automatically between their own products. Genuinely convenient - but the modeling, definitions, and quality work of a2-a3 does not disappear; it just stops being visible on the diagram. Nobody has automated deciding what "active customer" means.
  • "Real-time by default": ask what the default costs at your volume, and which of your decisions consumed the freshness. Paying streaming prices for daily decisions is the a5 amber band in a new outfit.
  • The lakehouse convergence (the real trend): warehouses and lakehouses are genuinely growing together - warehouses read open lake formats, lakehouses add governance. Directionally true and good for buyers: it keeps exit doors open. The label on the box matters less every year; the discipline inside it does not change.
Real world

The streaming bill for a daily decision. A subscription company built a beautiful streaming pipeline so the exec dashboard updated live. Cost: roughly 8x the batch equivalent. A year in, the team audited usage: the dashboard was opened each morning, once, with coffee. The pipeline was moved back to hourly; nobody noticed except finance. The lesson stuck as a one-liner: pay for the freshness the decision uses, not the freshness the demo showed.

Part 3 · the plan on one slide

Crawl, walk, run - and what to ask quarterly 8 min live

Everything in this track compresses to one strip. Companies that skip crawl to buy run-stage tools get the a4 lakehouse story; companies that stay in crawl forever starve their AI ambitions. The skill is knowing which stage you are in and funding the next one, not the flashiest one.

The warehouse roadmap on one strip CRAWL daily batch loads one star schema one mart, one team served definitions written down WALK tested definitions more marts, more teams monthly cost review (a5) quality checks before serving RUN near-real-time slices where money justifies it features served to ML + AI reverse ETL into the tools Each stage earns the next: run-stage AI on crawl-stage definitions produces confidently wrong answers. Most companies belong in walk. Run is bought per use case, not as a platform badge. Locate your company on this strip today - that answer sets next year's data budget.
🔍 Click to zoom - crawl, walk, run: each stage earns the next
LiveFour questions to ask your team quarterly4 min
  1. "Which stage are we in, and what moves us to the next?" Forces an honest self-placement and a concrete gap list - the answer should name deliverables, not tools.
  2. "Which definitions did we test this quarter?" The walk-stage heartbeat. If the answer is none, your AI ambitions are drifting toward the failed-copilot demo.
  3. "What is the freshest data any decision actually consumed?" Keeps real-time honest: if nothing used sub-daily freshness, nothing should be paying for it.
  4. "If we pointed an AI agent at the warehouse tomorrow, what would embarrass us?" The best audit question of the AI era - it surfaces the two-definitions problem, stale tables, and ungoverned corners in one sentence.
Self-studyWhere to go from here3 min read

This closes the leader track: a1-a3 gave you the why (workloads, star schemas, one truth), a4-a5 the buying and running (engines, contracts, cost), and a6 the future (AI serving, real-time judgment, the roadmap). Three good next doors:

  • The builder track of this course (b1-b10): the same Daybreak story hands-on - a real columnar engine in your browser, from raw source to served answer. Skim b1 even as a non-coder; you will recognize everything.
  • learn-data-engineering: the pipelines that feed the warehouse - where the data comes from and how it arrives on schedule.
  • learn-ai-governance: the oversight layer for the AI systems this session put on top of your warehouse - risk tiers, obligations, and the questions regulators ask.
Between sessions

Take this back to your desk ◐ 20 min

Source material

Official sources covered

The leader track distills the strategy layer of the major curricula - serving, freshness, and roadmap judgment, without the pipeline labs. This session covers:

DeepLearning.AI Data Engineering C4 (Joe Reis) M4 - serving data for analytics & ML, views + materialized viewsPart 1 · the four customers, exec altitude; b7 builds the views
365DS Intro to Data Warehousing - real-time data warehousing (S7)Part 2 · the freshness ladder and its honest price tags
IBM Data Warehouse Fundamentals - architecture & governance threadsPart 3 · roadmap framing; module depth lives in a1-a4 and the builder track
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · Why does putting an AI agent on top of the warehouse RAISE the stakes on data quality?

A human notices when a chart looks weird. An agent grounded in stale or double-defined data just answers. The a1-a3 definition work was secretly your AI-readiness program.

2 · A team requests streaming freshness for the executive revenue dashboard. The leader's filter question is...

Capability and price come second. Most decisions are fine at daily; streaming costs 5-10x batch. A named person with a named minute-level action justifies the rung - "nice to see" does not.

3 · Your company has one daily-loaded star schema, one mart, and definitions written down but not tested. The right next investment is...

That description is the end of crawl. Each stage earns the next: run-stage AI on untested definitions produces the failed copilot demo. Fund the next stage, not the flashiest one.

Leader session 6 cheat sheet · pin this

Four customersDashboards, ML features, AI agents/RAG, reverse ETL into tools. One governed warehouse serves all four.
The AI stakesDashboards have human safety nets; AI does not. Garbage warehouse = confidently wrong AI, at scale.
ViewsA definition saved once, inherited everywhere - how "active subscriber" stays one number across all consumers.
Freshness ladderDaily batch serves most decisions. Hourly for operations. Streaming (5-10x cost) only where someone acts in minutes.
"Zero-ETL"Convenient data movement - but modeling, definitions, and quality work do not disappear. Nobody automated deciding what "active" means.
Convergence trendWarehouses and lakehouses are merging around open formats. Good for buyers - exit doors stay open. Discipline stays the same.
Crawl-walk-runCrawl: daily star schema + one mart. Walk: tested definitions, more marts, cost review. Run: real-time slices + feature serving, per use case.
The audit question"If we pointed an AI agent at the warehouse tomorrow, what would embarrass us?" Ask it quarterly.