learn-analytics-engineering-with-phoebe / Builder session 4 of 10
Learn Analytics Engineering with Phoebe · Builder track · Session 4 of 10

ref() and the DAG

Two functions do most of dbt's work. source() names a raw table once, and ref() names another model - and because you never write a table name, dbt can derive the entire dependency graph from the SQL itself. That graph decides build order, decides what runs in parallel, decides what gets skipped when a test fails, and tells you exactly what breaks if you rename a column. This session explores the real graph from this project's manifest: 32 nodes, 47 edges, nobody drew it.

🟡 Builder track Analytics engineers · analysts · DE Live lineage from manifest.json 45 min
0-3 · Welcome 3-18 · ref, source, and compilation 18-42 · Explore the graph, read the impact sets 42-45 · Q&A
Part 0

The trade you are making

Writing {{ ref('stg_bazaar__orders') }} instead of main_staging.stg_bazaar__orders looks like ceremony. It buys four things you cannot get any other way: build order you never maintain, parallelism that is safe by construction, a stop signal when something upstream fails, and an answer to "what breaks if I change this" that is derived rather than remembered.

Live - presented in session Self-study - read after class ▶ Live lineage & SQL Sources covered
★ What you walk out with today A working understanding of compilation (what you write versus what the warehouse runs), the four things the DAG buys you, fluency with dbt's selector syntax, and the habit of checking an impact set before you rename anything.
Part 1 · covers dbt documentation on ref, source and compilation

Compilation, in one picture 6 min live

dbt is a templating engine plus a graph. Your model is SQL with a little Jinja; compilation resolves the Jinja into a plain query the warehouse understands, and records what it referenced along the way. That recording is the graph.

1 · What you write select * from {{ ref('stg_bazaar__orders') }} a model name, never a table name 2 · What dbt compiles select * from "main_staging"."stg_bazaar__orders" plain SQL, schema-qualified 3 · What dbt records edge: stg_bazaar__orders -> fct_order_items one edge of 47 in this project What step 3 pays for build order - derived, so nobody maintains a runbook of what to run first safe parallelism - independent models run at once because the graph proves they are independent a stop signal - a failed test SKIPs everything downstream instead of rebuilding on bad data impact sets - "what breaks if I rename this" answered by walking edges, not by memory
🔍 Click to zoom - one ref() call, three outputs
Livesource() versus ref(), and why the difference matters3 min▶
  • source('bazaar', 'orders') points at a table dbt did not build. It appears in exactly eight places in this project - the eight staging models - because _bazaar__sources.yml is the single place a raw table name is written. When the loader renames something, you edit one file.
  • ref('stg_bazaar__orders') points at a model dbt did build. It resolves differently per environment: in your dev target it points at your personal schema, in production at the production schema. The same code, no branching.

That second point is the one people underestimate. Hard-coded table names are why hand-written warehouse SQL cannot be run in a sandbox without editing it - and why nobody tests a change before shipping it. With ref(), dev and prod differ by a target, not by a diff.

The rule If dbt built it, use ref(). If something else built it, declare it as a source and use source(). A raw table name written directly in a model is always a bug, even when it works.
Self-studySelector syntax, the part you will use daily3 min read▶

Once there is a graph, you can address it. These are the selectors worth memorising:

CommandWhat it runs
dbt buildeverything - 23 models, 100 tests, 1 snapshot
dbt build --select fct_order_itemsthat model and its tests, nothing else
dbt build --select fct_order_items+that model and everything downstream of it
dbt build --select +fct_order_itemsthat model and everything upstream - the rebuild-from-source path
dbt build --select staginga whole folder, by path
dbt build --select state:modified+what changed on this branch plus its downstream - the CI selector

That last one is how a pull request tests only what it touched, which is what makes CI on a 500-model project take minutes rather than hours. Session b9 wires it up.

Live lineage · 14 min · everyone explores

The real graph, and what it says ★ read from manifest.json

Below is this project's actual dependency graph, read from target/manifest.json after a real build. Click any node: you get its upstream (highlighted pale), its downstream (highlighted red), the impact set spelled out, and both versions of its SQL - what you write, and what dbt compiled.

Start where it opens, on fct_order_items. Two marts depend on it; fifteen nodes feed it. That asymmetry is what a well-layered project looks like.

The graph around fct_order_items Feeds it (upstream) 15 nodes Depends on it (marts) 2 marts A well-layered project looks like this: many nodes feed a model, few depend on it directly.
🔍 Click to zoom - many nodes feed it, few nodes depend on it directly

Now click stg_bazaar__orders. Seven nodes downstream. This is the model you cannot casually rename, and the graph told you so without anybody keeping a list.

Click a source. Everything in the project is downstream of some source, which is why a source schema change is the most expensive change there is.

Compare the two SQL blocks on any model. The Jinja on top, the compiled query underneath. That is the whole of what dbt did.

Real world

The graph is the reason a rename becomes a conversation. In a hand-written estate, renaming a column is a two-minute edit followed by three weeks of breakage discovered one dashboard at a time. Here, dbt build --select stg_bazaar__orders+ tells you in advance that seven nodes and their tests are in the blast radius - and CI runs exactly those on the pull request. Same rename, same risk, completely different amount of surprise.

✗ Hand-written rename a two-minute edit three weeks of surprise breakage found one dashboard at a time ✓ dbt graph rename select stg_bazaar__orders+ 7 nodes in the blast radius, known CI tests exactly those nodes Same rename, same underlying risk - completely different amount of surprise for the team.
🔍 Click to zoom - same rename, same risk, no surprise
Demo 2 of 2

Your turn: query the compiled reality ★ 8 min · build your own

The graph is metadata. Underneath it there are real objects with real schema names, which is what the compiled SQL points at.

LiveQ1 · See what ref() resolved into4 min▶
SELECT table_schema, table_name, table_type
FROM information_schema.tables
WHERE table_schema IN ('main_staging', 'main_marts')
ORDER BY table_schema, table_name;

Twenty objects for twenty-three models: the three intermediate models are ephemeral, so they exist only inside the SQL of whatever referenced them. Nothing to query, nothing to keep fresh, nothing for someone to find and misuse.

LiveQ2 · Prove the ephemeral models really are inlined4 min▶
SELECT count(*) AS objects_named_int_anything
FROM information_schema.tables
WHERE table_name LIKE 'int_%';

Look at fct_order_items in the explorer above and read its compiled SQL: the intermediate model appears as a CTE inside it. That is what ephemeral means, and it is why intermediate models cost nothing to have.

Self-studyQ3 · Where the DAG stops helping3 min▶

No SQL. The graph knows about dependencies you expressed through ref() and source(). It knows nothing about:

  • Consumers outside dbt. A BI dashboard, a reverse-ETL sync, somebody's scheduled export - all downstream of your marts, none of them in the manifest. Renaming a mart column is safe according to dbt and catastrophic in the BI tool. This is precisely the gap model contracts close, in session b9.
  • Dependencies you smuggled in. A hard-coded table name, or a macro that builds a table reference from a string, is invisible to the graph - so dbt may build in the wrong order and be right to.
  • Semantic dependencies. Two models can be logically coupled - the same business rule implemented twice - with no edge between them. The graph cannot see a copy-paste.
Homework

Try it yourself - this week ◐ 20-30 min total

Source material

Sources covered

This page covers:

✓dbt documentation - ref, source, compilation, the manifest, node selection syntaxParts 1-2 + the explorer · graph read from this project's own manifest
✓Ephemeral materializationDemo 2 · proved by absence in information_schema
◐State-based selection (state:modified+) for CINamed here, wired up in session b9
◐Exposures - declaring the consumers dbt cannot seeSelf-study Q3 · the gap; contracts and exposures are b9
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · Why write {{ ref('stg_bazaar__orders') }} instead of the table name?

One call does two things: it names the dependency (which becomes an edge) and it defers the actual schema until run time (which is why the same code runs in dev and prod without a diff).

2 · You want to rename a column in a staging model. Which command tells you the blast radius?

Trailing plus is downstream (what depends on me), leading plus is upstream (what I depend on). Downstream is the blast radius; upstream is the rebuild path when you need fresh inputs.

3 · Which dependency is invisible to the DAG?

dbt only knows what you expressed through ref() and source(). Consumers outside dbt, hard-coded table names, and duplicated business logic are all real dependencies with no edge - which is why b9's contracts and exposures exist.

Builder session 4 cheat sheet · pin this

ref()Names a model dbt built. Records an edge, resolves per environment.
source()Names a table dbt did not build. Declared once in a sources yml.
The four payoffsBuild order · safe parallelism · skip-on-failure · impact sets.
Selectorsmodel+ downstream · +model upstream · folder by path · state:modified+ for CI.
This graph32 nodes, 47 edges, derived. fct_order_items: 15 upstream, 2 downstream.
EphemeralInlined as a CTE, never materialized. 3 here, 20 objects for 23 models.
Compiled SQLRead target/compiled/ to debug Jinja. It is the fastest debugging move there is.
The blind spotBI tools, exports, copy-pasted logic. No edge, real dependency. Next: b5.