learn-data-warehouse-with-phoebe / Leader session 5 of 6
Learn Data Warehouse with Phoebe · Leader track · Session 5 of 6

Cost and performance

The warehouse bill is the most negotiable line in your data budget - and the least understood. Most of it is compute, most of the compute is scans, and a surprising share of the scans serve nobody. This session teaches you to read the bill, name the four levers that shrink it, and run the ten-minute monthly conversation that keeps it shrunk. Daybreak's founder just opened her first real invoice; let's read it with her.

🔵 Leader track Leaders: CxO · VPs · managers No code - thinking mode 45 min
0-3 · Welcome 3-20 · Why bills explode 20-42 · The four levers + monthly questions 42-45 · Q&A
Part 0

How this session works

In a4 you learned the pricing shapes: per TB scanned, per compute-second, per credit. Today we follow the money after the contract is signed. No tuning syntax - the builder track's session b8 is the hands-on twin of this page. Your job is different: understand the physics well enough that "the warehouse bill doubled" becomes a conversation you can lead, with questions that have right answers.

Live - discussed in session Self-study - read after For decision-makers
★ What you walk out with today The anatomy of a warehouse bill (storage is the sliver, compute is the bulk, waste hides inside the compute), the four levers every engine offers in some form, and five monthly questions for your data lead - with a feel for what good answers sound like.
Part 1 · covers 365DS partitioning/indexing/query performance (exec level) · b8 is the hands-on twin

Why warehouse bills explode 8 min live

Cloud warehouses charge for work: per TB scanned (BigQuery's shape) or per second of compute (Snowflake's shape). That is fair - and it means every wasteful query is a small invoice. Bills do not explode because storage got expensive. They explode because compute quietly serves habits instead of decisions: whole-table scans for three columns, dashboards refreshing for empty rooms, one runaway join.

Anatomy of one month's warehouse bill storage compute doing useful work decisions, reports, models compute doing waste scans nobody needed storage: pennies per GB - almost never the problem compute: the bulk - and the amber band is the target What lives in the amber band SELECT * on wide tables 60 columns read, 3 used - billed for all 60 Dashboards on auto-refresh every 5 min, all night, for a room with nobody in it The intern's cross join one missing join condition: billions of rows by accident The bill is a behavior report. Shrink the amber band and performance improves for free - the same waste that costs money is what makes dashboards slow.
🔍 Click to zoom - storage is the sliver, compute is the bulk, waste hides in amber
LiveThe meter runs on scans, not on answers4 min

The mental model that unlocks everything: a warehouse charges you for the data it touches, not the rows it returns. A query that returns one number can scan two terabytes to compute it. That is why identical-looking dashboards can differ 100x in cost - the difference is invisible on the screen and enormous on the meter.

  • Per-scan engines (BigQuery shape): the invoice literally lists bytes scanned. SELECT * on a 60-column table bills all 60 columns even if the chart uses 3.
  • Per-compute engines (Snowflake shape): you pay for the seconds a warehouse runs. Wasteful scans keep it running longer - same waste, different meter.
  • Why nobody notices: each incident is small. Fifty small habits, compounding daily across every dashboard, are how a bill doubles in two quarters with no single culprit.
Real world

The dashboard that cost more than the analyst. A CFO asked why the analytics bill had tripled. The audit found one executive dashboard set to refresh every five minutes, around the clock, scanning the full order history each time. It was viewed roughly twice a day. The refresh interval was changed to hourly during business hours - a one-line setting - and the bill dropped by a third. Nobody had done anything wrong; nobody had ever looked.

Self-studyWhy "add an index" is not the answer here2 min read

If you learned databases in the app world, your instinct says "slow query, add an index". Warehouses mostly do not work that way. Indexes accelerate finding a few rows; analytics scans millions. Columnar engines got rid of classic indexes and replaced them with a different toolkit - reading fewer columns, skipping irrelevant slices of data, remembering answers. That toolkit is Part 2, and it is the vocabulary your data lead already uses: pruning, partitioning, clustering, materialized views, caching.

Self-studyA leader's glossary for the invoice3 min read

Five words that appear on real warehouse invoices, translated once so the bill stops being a foreign document:

  • Bytes scanned / TB processed: how much data your queries touched. The core meter on scan-priced engines - this is the number the four levers shrink.
  • Compute hours / credits / DBUs: how long engines ran, in the vendor's local currency. Same physics, different unit: waste keeps the clock running.
  • Slots / warehouse size: how much horsepower was reserved. Reserved and unused still bills - that is lever 4's territory.
  • Storage (active vs long-term): what sits on disk. Cheap per GB, but old copies and abandoned tables add up quietly - and clutter has a governance cost beyond the invoice.
  • Egress / data transfer: data leaving the platform or crossing regions. Small on most months, decisive when you migrate - which is why a4 put it in the contract questions.
Part 2 · covers 365DS partitioning & performance, IBM M2 materialized-view ideas · b7/b8 build them

The four levers, in exec terms 9 min live

Every warehouse engine - Snowflake, BigQuery, Redshift, Databricks, even DuckDB - offers the same four levers under different brand names. You do not pull them yourself. You fund them, ask about them, and recognize when one is being ignored.

Four levers, one bill 1 · Columnar + column pruning name 3 columns, read 3 columns - never SELECT * ask for less 2 · Partitioning + clustering slice tables by date so "last 30 days" skips the other years skip most of the data 3 · Materialized views + caching pre-compute the daily rollup once, serve it 500 times compute once, reuse 4 · Right-size compute auto-suspend when idle, size to the job, not to the peak warehouses that sleep Levers 1-2 shrink what each query reads. Lever 3 stops re-answering the same question. Lever 4 stops paying for silence. Brand names differ - micropartitions, slots, clustering keys - but every engine sells these same four moves. Order matters: teams reach for lever 4 (bigger or smaller machines) first. The cheap wins are usually levers 1-3 - they need attention, not budget.
🔍 Click to zoom - four levers with their plain-words payoffs
LiveWhat each lever sounds like in a status update4 min

You will never pull these levers, but you will hear them. Translation table:

  • "We cleaned up the worst queries" = lever 1. Analysts named columns instead of SELECT *, trimmed wide joins. Cost: a few days of attention. Often the single biggest win.
  • "We partitioned the big tables by date" = lever 2. The 30-day dashboard now reads 30 days, not five years. One-time engineering work that pays rent forever.
  • "We materialized the daily rollups" = lever 3. The revenue dashboard reads a small pre-computed table instead of re-scanning raw orders 500 times a day. Builder session b7 builds exactly this for Daybreak.
  • "We set auto-suspend to 60 seconds" = lever 4. Compute now sleeps between queries instead of idling on the clock. In credit-based engines this is often the fastest fix on the list.
The question that reveals lever health Ask: "if our data doubled tomorrow, would the bill double?" A tuned warehouse answers no - partitioning and materialized views break the link between data size and query cost. A raw one answers yes, and that is your roadmap conversation.
Self-studyPerformance and cost are the same lever2 min read

Nothing in Part 2 was a cost trick. Reading fewer columns, skipping partitions, serving cached rollups - each one also makes the dashboard load faster. In a warehouse, waste is measured twice: once in seconds (the analyst waits) and once in dollars (the meter runs). This is why "make it cheaper" and "make it faster" are one project, not a trade-off - a genuinely unusual property. Spend it in the boardroom: the cost-review project is also the user-experience project.

Self-studyThe same levers, in each vendor's accent2 min read

When your team names a feature, map it back to the lever - the brand names change, the four moves never do:

LeverSnowflake accentBigQuery accent
1 · Ask for lessColumn pruning over micropartitionsFewer columns = fewer bytes billed
2 · Skip dataClustering keys, partition pruningPartitioned + clustered tables
3 · Compute onceMaterialized views, result cacheMaterialized views, BI Engine cache
4 · Right-sizeAuto-suspend, warehouse sizingSlot commitments sized to need

Redshift, Databricks, and even DuckDB tell the same story in their own vocabulary. If a proposed fix does not map to one of the four rows, ask which lever it is pulling.

Part 3 · the ten-minute monthly review

Five questions for your data lead 7 min live

You do not manage warehouse cost by reading queries. You manage it by asking five questions once a month and listening for whether the answers exist. Teams that can answer them have the amber band under control; teams that cannot are about to surprise you.

LiveThe five questions, and what good answers sound like5 min
  1. "What are our ten costliest queries this month?" Good: a list exists, with owners, and the top one is being fixed. Bad: "we'd have to look into that" - every engine ships this report; not reading it is a choice.
  2. "Is scan-per-dashboard trending up or down?" Good: down or flat while usage grows - the levers are working. Bad: nobody tracks per-dashboard cost, so growth and waste are indistinguishable.
  3. "What is our storage vs compute split?" Good: storage is a sliver and they know the ratio. Bad: a big storage bill usually means old copies and abandoned tables nobody dares delete - cheap to fix, embarrassing to admit.
  4. "What share of dashboard reads hit a cache or materialized view?" Good: the top dashboards read pre-computed rollups, raw scans are the exception. Bad: every glance at the revenue chart re-scans all of history.
  5. "Who reviews new dashboards before they ship?" Good: a named person checks scan cost and refresh schedule - a five-minute gate. Bad: anyone can point anything at the biggest table on a five-minute refresh, and eventually someone does.
Real world

The boardroom moment. A CEO opened a quarterly review with one slide: warehouse spend up 80%, revenue up 12%. The data lead had the answer ready: spend per active dashboard was down 30% - the growth was 40 new dashboards shipped for two new business lines, each reviewed and each cached. The conversation took ninety seconds and ended in a bigger budget. That is what the five questions buy: not a smaller bill, a defensible one.

Self-studyWhen the bill SHOULD go up2 min read

A word of balance before you take the five questions home. The cheapest warehouse is an unused one. If the bill is rising because more teams make more decisions from data - new marts, new markets, a model in production - that is the system working. The target is not a small number; it is no amber band: every scanned terabyte serving someone. Judge the bill the way you judge a sales-travel budget: not by size, but by what came back.

Between sessions

Take this back to your desk ◐ 20 min

Source material

Official sources covered

The leader track distills the cost-and-performance material of the major curricula to executive altitude - the physics and the questions, without the tuning labs. This session covers:

365DS Intro to Data Warehousing - partitioning, indexing & query performance (S3)Parts 1-2 · exec level; the hands-on version is builder b8
IBM Data Warehouse Fundamentals M2 - cubes & materialized viewsPart 2 lever 3 · full depth lands in builder b7/b8
LinkedIn Learning Advanced Snowflake - micropartitions & clusteringPart 2 lever 2 · topic level only; brand-name mechanics stay in b8
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · Your warehouse bill doubled in six months. Statistically, the most likely culprit is...

Storage is the sliver; compute is the bulk, and waste hides inside it. Bills explode from fifty small habits compounding daily, almost never from one dramatic cause.

2 · "We materialized the daily revenue rollup" means...

Lever 3: compute once, read many. The revenue chart now reads a small pre-built table - faster for users and cheaper on the meter, at the same time.

3 · The best sign that warehouse cost is under control is...

A falling bill might just mean falling usage. Control means visibility plus a gate: the amber band is measured, owned, and shrinking - even while the total grows with the business.

Leader session 5 cheat sheet · pin this

The meterWarehouses charge for data touched (scans, compute-seconds), not answers returned. One number can cost two terabytes.
Bill anatomyStorage = sliver. Compute = bulk. Waste hides inside compute: SELECT *, always-on refreshes, runaway joins.
Lever 1 · pruningAsk for less: name columns, never SELECT *. Attention, not budget. Often the biggest single win.
Lever 2 · partitioningSkip most of the data: slice by date so "last 30 days" never reads five years.
Lever 3 · materializeCompute once, read many: pre-built rollups + caching serve the top dashboards.
Lever 4 · right-sizeWarehouses that sleep: auto-suspend when idle, sized to the job, not the peak.
Five monthly questionsCostliest queries? Scan trend per dashboard? Storage/compute split? Cache hit rate? Who reviews new dashboards?
The real targetNot a small bill - no amber band. Cost and speed are the same lever; a rising bill serving real decisions is health.