Why leaders need the map
You do not need to build data infrastructure - but you approve its budget, you wait on its timelines, and you trust (or distrust) what comes out of it. Companies store data in a handful of very different places, each built for a different job, and most "data frustration" at the executive level is really a mismatch: asking a place to do a job it was never built for. Today we draw the city map - application databases, warehouses, marts, lakes, lakehouses - and follow one order across it.
OLTP vs OLAP 5 min live
Every company runs two fundamentally different data workloads. One RUNS the business: take the order, charge the card, update the stock - thousands of tiny operations, each in milliseconds. The other ANALYZES the business: scan five years of history and tell me the trend. Same data, opposite engineering. Confusing them is how websites go down.
LiveWhy the CFO's report must never touch the checkout database3 min▶
An application database (OLTP: online transaction processing) is tuned like a race car for one move: find one record, change it, confirm it, in a few milliseconds, ten thousand times a minute. An analytical system (OLAP: online analytical processing) is tuned like a freight train: haul enormous loads of history in one slow, powerful pass.
- Run a freight job on the race car and every customer at checkout waits behind the CFO's five-year revenue scan. Pages time out, carts abandon, and on peak days the site simply falls over.
- That is why data gets copied. Companies duplicate data out of the application DB into analytical stores - not bureaucracy, physics. Two workloads, two engines.
- The exec tell: if a "simple report" request gets pushback from engineering, this split is usually why. The data exists; the safe place to query it may not yet.
Amazon learned this the expensive way: during the 2004 holiday season, outages traced back to overloaded transactional databases pushed Amazon to redesign how it stores order data - work that produced its famous Dynamo system, built so checkout stays fast no matter what else is happening. The lesson stuck industry-wide: protect the transaction path, and analyze somewhere else.
Self-studyRow vs column storage, the 60-second intuition2 min read▶
Imagine customer records as a giant paper ledger.
- Row storage keeps each customer's full record on one line: name, email, city, total spend. To answer "show me customer 4812", you read one line - instant. That is OLTP's layout.
- Column storage keeps each column as its own long list: all names together, all spends together. To answer "sum total spend across 50 million customers", you read just the spend list and skip everything else - hundreds of times faster than reading 50 million full lines. That is OLAP's layout.
Same information, different filing. Neither is "better" - each is unbeatable at its own job and painful at the other's. When someone name-drops "columnar" (Snowflake, BigQuery, Redshift), they mean "built for the freight-train scans".
Warehouse, mart, lake, lakehouse 6 min live
Think of your company's data estate as a small city. Applications are the shops where business happens. Everything else exists to move what the shops record into places where people can learn from it - a library, its branches, and a harbor district full of raw cargo.
LiveWhat each place is for (and what happens when you skip governance)4 min▶
- Data warehouse - the city library. Data arrives cleaned, deduplicated, joined and catalogued so everyone reads the same truth. Slow to build, wonderful to use. The two founding schools of shelf-design are Kimball and Inmon - recognize the names, let your architects fight about them.
- Data mart - the branch library. A department-sized slice of the warehouse: finance gets its tables, marketing gets its campaigns. Faster, closer, less waiting on the central team.
- Data lake - the harbor warehouse district. Everything unloaded raw: clickstreams, logs, images, exports. Extremely cheap per terabyte, no upfront structure ("schema-later"). Brilliant for data science exploration; useless for a board pack until someone refines it.
- Lakehouse - the modern merge. The last few years' big idea (the Databricks and Snowflake era): keep the lake's cheap open storage but add the warehouse's tables, governance and reliability on top of one copy of the data - instead of paying to maintain both places.
Netflix runs its analytics on a lake-turned-lakehouse: petabytes of viewing and operational data on cheap cloud storage, made trustworthy by a table layer Netflix built and open-sourced (Apache Iceberg) - lake economics, warehouse manners. The cautionary twin: industry post-mortems have long estimated that a large share of early data lakes - figures around 80% get cited - decayed into "data swamps" because companies bought the cheap storage but skipped the cataloguing and governance. The harbor fills up either way; only governed harbors pay rent.
ETL, ELT and why dashboards go stale 4 min live
Between the shops and the library run the delivery trucks: pipelines. Extract data from the source, Transform it into clean shape, Load it into the warehouse - ETL. The modern default flips the last two letters: ELT loads raw data first and transforms it inside the warehouse, where compute is cheap and every step is repeatable.
Live"The dashboard is wrong" - the leader's diagnostic2 min▶
- Most "wrong" dashboards are late dashboards. A pipeline failed overnight, numbers froze at yesterday, and nobody was told. Before doubting the analysis, ask when the data last refreshed.
- The second most common cause: two trucks, two routes. Marketing's number comes through one pipeline, finance's through another, with slightly different cleaning rules. Both "correct", never equal. Session 4 fixes this with a shared metric layer.
- ETL vs ELT in one sentence for meetings: ETL cleans on the way in; ELT dumps raw and cleans inside the warehouse - cheaper, more flexible, and every transformation is written down where it can be audited and rerun.
Airbnb had so many delivery trucks it built a traffic-control tower: by 2014 its pipeline count outgrew human tracking, so engineers built Airflow, a scheduler that watches every job and flags failures - later open-sourced and now standard across the industry. The takeaway for the boardroom: pipeline monitoring is not gold-plating, it is why your Monday numbers can be trusted on Monday.
Self-studyThree questions to ask your data engineer2 min read▶
| Question | What it surfaces | Good answer sounds like |
|---|---|---|
| "How fresh is this number?" | pipeline schedule + last successful run | "Synced nightly at 02:00; last run succeeded." |
| "Where does this break, and who finds out?" | monitoring and alerting maturity | "Failures page the on-call and stamp the dashboard." |
| "What does keeping this flowing cost?" | cloud compute + maintenance load | "About X per month; the big join is the expensive bit." |
Cloud economics footnote: storage is famously cheap, compute is what bills you. A lake full of cold data costs coffee money; scanning it carelessly every hour is what makes the invoice interesting. That is one reason ELT won - and one more reason the catalogue matters.
Map YOUR company's city ★ 6 min · everyone maps
Two minutes solo: list the systems in your business that hold data - ERP, CRM, web analytics, the e-commerce platform, HR system, support desk, and yes, the spreadsheets.
Tag each one OLTP-ish (runs the business, live records) or OLAP-ish (analyzes the business, history and summaries). Some will be both - flag those, they are usually the pain points.
Now find the "spreadsheet shadow warehouse": the workbook someone exports into every Monday, joins by hand, and half the company quietly treats as the source of truth. Every company has at least one.
Compare with a neighbor: whose shadow warehouse is scarier? Keep your map - Session 4 builds the metric layer that retires it.
Right workload, right home ★ 8 min · everyone sorts
Six jobs from one company. Send each to the place built for it. Two are sneaky.
The freshness audit ★ 7 min · everyone audits
Pick 3 numbers you personally saw this week: a dashboard tile, a number in a deck, a figure quoted in a meeting.
For each, answer (or honestly guess): how fresh is it - real-time, last night, last week, or "when someone last exported it"? And which home did it come from - app DB, warehouse, mart, lake, or a shadow spreadsheet?
Mark the scariest one: the number you weight heavily in decisions but whose freshness and origin you just guessed at. Most rooms discover at least one board-level number nobody can trace.
Write the single question you will actually send your data engineer this week - steal from the self-study table above. That question is today's deliverable.
This week ◐ 25 min total
- Send your freshness question from Apply-along 3 to a real data engineer or analyst, and note how long the honest answer takes to arrive. The lag itself is data.
- Finish your city map from Apply-along 1: every system tagged OLTP-ish or OLAP-ish, shadow warehouses circled in red.
- Read the self-study cards above: row vs column storage, and the three engineer questions.
- Optional deep end: the Kimball Group's "dimensional modeling" primer, first pages only - to see what your architects mean by facts and dimensions.
Three questions before you go 🎯 ◐ 90 seconds
1 · Why should the CFO's five-year revenue report never run directly on the checkout database?
OLTP engines are race cars for single records; a five-year scan is a freight job. Run it there and on a peak day the site slows or falls over - which is why data is copied to OLAP stores at all.
2 · Your team debates "lake vs warehouse" for a new initiative. What is the actual tradeoff?
It is a tradeoff between cost-and-flexibility and trust-and-readiness. Lakes shine for exploration and raw scale; warehouses shine for the numbers you put in front of a board. Skip governance on a lake and you get a swamp.
3 · A vendor pitches a "lakehouse". What is the promise you should test?
The lakehouse pitch is "stop paying to run a lake AND a warehouse". The test question stays the same as ever: who owns the catalogue and the governance? The architecture enables trust; people still have to supply it.
Concepts covered & their origins
The working 80% of each idea, credited to its canon - the deep end stays open for anyone who wants to swim.