The signal that changes the first ten minutes
Sessions 2 to 6 built a monitor set on analytics.orders_daily at Bellwether and moved part of it into a framework. Every one of those monitors ends the same way: a notification saying a number is outside its band. What happens next is triage, and triage is a graph problem. Tonight is the fifth signal, the maintenance cost that makes it the last one to buy, and the query that turns a page into a list of names. Session 8 then scores the whole set against 30 nights of history.
★ What you walk out with tonight. An impact query that takes a table and a column and returns the assets that read it, ranked by whether a human or a customer sees the output. The honest trade between the three ways a lineage graph gets built, so you can pick one on Monday rather than evaluating four vendors for a quarter. And the export nobody knew about, which almost every team finds on the first run of this exercise.
LiveWhy lineage is the only signal that answers why3 min▶
The four metadata signals are properties of one table measured over time. They are cheap, they are precise about what they see, and they are structurally incapable of saying anything about cause, because cause lives in a different table. A volume drop on orders_daily is a fact about orders_daily. That one of three source shards stopped arriving is a fact about the edge between two nodes.
The same asymmetry runs downstream. A freshness alert tells you the table is late. Whether that matters depends entirely on who reads it in the next hour, and no amount of metadata about the table itself contains that. This is why teams with excellent detection still have bad incidents: they knew at 06:32 and told the wrong people at 09:15.
Night 23 at Bellwether: ship_region dropped and promo_code arrived. The schema monitor fired correctly. The on-call read it, saw that no dashboard was obviously broken, and logged it as an additive change to look at later. Three days of the regional margin report were quietly wrong, and the CRM sync that segments customers by region had been writing nulls into a live campaign the whole time. The detection was fine. The triage had no graph.
Self-studyWhat lineage is not for3 min▶
- It is not a detector. A graph does not fire. It answers questions when something else fires, which is why buying lineage first and monitors later gets the order backwards and is a common way to spend a budget without reducing detection time.
- It is not documentation. A generated graph records what the code does, including the things nobody meant. That is its value and also why it disagrees with the architecture diagram, and when they disagree the graph is right.
- It is not a data model. How the warehouse should be shaped is a design discipline with its own course. Lineage is a description of what currently exists, however ugly.
- It is not free. Every route to a graph has a maintenance cost proportional to how much of the estate it covers, and that cost is the reason table tiering from leader session 5 decides where lineage goes.
Lineage is the only signal that answers why
9 min liveA lineage graph is nodes for datasets and jobs, edges for reads and writes, and optionally edges at the level of individual columns. That is the whole data model. Everything difficult about lineage is how the edges get there and what happens when they go stale.
LiveThe three ways a graph gets built4 min▶
Three routes, and most estates end up with two of them stitched together. Pick deliberately rather than by whatever the first tool you installed happened to do.
| Route | How it works | The honest trade |
|---|---|---|
| Parsed from SQL | A parser reads your transformation code and maps output columns back to input columns | Cheap, needs no cooperation from anybody, and wrong at the edges. Dynamic SQL, string-built queries, stored procedures, macros and select * all degrade it silently |
| Emitted at runtime | Jobs report what they read and wrote as they run, using a standard event format such as OpenLineage | Accurate, including things no parser could infer, and it requires every job to be instrumented. The uninstrumented job is invisible rather than wrong, which is worse |
| Collected from a catalog | A catalog or warehouse platform exposes lineage it derived from query history or its own metadata | Fastest to switch on and bounded by what that platform can see. Anything outside it, which is usually the BI tool and every export, simply is not in the graph |
The parsing route fails loudest and is therefore the safest to start with: you can read the parse errors. The runtime route fails quietly, which is why an emission-based graph needs a coverage number of its own, tracking what fraction of your jobs actually emit.
OpenLineage, in one paragraph. An open standard for lineage metadata: jobs emit events about a run, naming the datasets they read and the datasets they wrote, and each event can carry extensible facets with extra detail, including a column-level lineage facet. The value of a standard here is that a dbt run, a Spark job and a Python script can all report into one graph without anybody writing three integrations. The cost is that the events only exist for the jobs you instrumented.
LiveTable level versus column level3 min▶
Table-level lineage says the region mix panel reads orders_daily. Column-level lineage says it reads ship_region from orders_daily. The difference sounds academic until you are holding a schema alert at 06:40.
- Table level answers a yes or no. Given a broken table, which assets are potentially affected. On a wide table with forty columns and twenty readers, the answer is usually "all twenty", which is not a triage decision.
- Column level answers a which. Night 23 dropped
ship_region. Of Bellwether's downstream assets, exactly two read that column. Eighteen readers are unaffected and you can say so, with evidence, in the first message. - That narrowing is the whole value. An alert that names two affected assets gets acted on. An alert that names twenty gets a promise to investigate, which is how three days of a wrong margin report happen.
- Column level costs more to maintain. Every route degrades faster at column granularity: a parser that handles
select *badly is fine at table level and useless at column level. Budget for that rather than being surprised by it.
The practical shape most teams land on: table-level lineage across whatever the catalog gives them for free, and column-level lineage maintained deliberately for tier 1 tables only.
Self-studyThe maintenance cost, and why lineage is bought last4 min▶
The four metadata signals have a maintenance cost close to zero: a query that computes count(*) keeps working when the pipeline is rewritten, because it does not know anything about the pipeline. A lineage graph is the opposite. It encodes the structure of the estate, so every change to the estate is a change to the graph, and a graph that is 80 percent correct is more dangerous than no graph because people trust it.
- Staleness has no alarm. A missing edge looks exactly like an asset that genuinely does not read the table. There is no signal that distinguishes them, which is why a stale graph produces confident wrong answers during triage.
- Coverage is the number to track. For an emission-based graph, what percentage of scheduled jobs emitted an event in the last seven days. For a parsed graph, what percentage of your transformation files parsed without error. Put that number next to the graph, in the tool, where anybody reading it can see how much to trust it.
- Tier 1 only, deliberately. Leader session 5 makes the case for tiering. Lineage is where it pays hardest: a maintained graph over the twenty tables that matter beats an unmaintained graph over nine hundred, and the second option is what happens by default when a tool is switched on estate-wide.
- It is the last of the five to buy. Not because it is least valuable but because it is the only one whose value depends on ongoing effort. Freshness and volume monitors keep working if the team stops caring for a quarter. A lineage graph does not.
The failure pattern is consistent: a lineage tool is bought, switched on across the whole warehouse, produces an impressive graph in week one, and by month four half the edges predate a pipeline migration nobody re-ran the collector after. The graph is then quoted in an incident review and turns out to have been wrong about the affected assets. The team concludes lineage does not work. What did not work was estate-wide coverage with no owner and no coverage number on the screen.
Blast radius, and the asset nobody documented
8 min liveA page has fired. There are four questions, in this order, and the order matters because the first two decide whether anybody needs to be woken up. What reads this table. Which of those are customer-facing or feed a payment. What has already run since the data went wrong. And who to tell, before they ask.
LiveThe reverse-ETL and export problem4 min▶
The single most common surprise on a first lineage exercise is not a dashboard. It is a quiet job, written eighteen months ago by somebody who has since changed team, that reads a warehouse table and pushes rows somewhere outside the warehouse. A CRM sync. A scheduled CSV to a partner SFTP. A customer-facing export in the product. A spreadsheet on a refresh schedule that a finance analyst built and never mentioned.
- They are invisible on a dashboard-only graph. A BI tool's lineage shows BI assets. A catalog's lineage shows what the warehouse can see. A reverse-ETL job that authenticates as a service account and issues a plain
selectlooks like any other query, and its destination is not a node anybody registered. - They are usually the highest-stakes reader. An internal dashboard being wrong for a morning is embarrassing. A CRM segment being wrong means a live campaign went to the wrong people, and a customer-facing export means the wrong numbers left the building with your name on them.
- Query history is where you find them. Your warehouse logs who queried what. Group the last 30 days of queries against a tier 1 table by service account and role, and every row you cannot immediately name is a candidate. This is a 20-minute exercise and it is the highest-yield 20 minutes in the whole track.
- Then register them as nodes. An export is an asset. Give it a name, an owner, an audience and an entry in the graph, so the impact query returns it next time instead of you remembering it.
Bellwether's query history for orders_daily showed nine distinct readers. Six were accounted for. Of the remaining three, one was an analyst's ad hoc exploration, one was a decommissioned job still authenticating successfully, and one was the CRM sync that segments customers by ship_region. On night 23 that column stopped existing. Nobody found out from the graph, because the graph did not contain it.
LiveFreshness propagation, and why nothing is wrong3 min▶
A late upstream table makes every downstream table late. Each of those downstream tables is internally consistent, correct, and built from exactly the data it was given. Every freshness monitor on every one of them fires. Every one of those alerts is technically true and none of them is the incident.
- One cause, N alerts. Six downstream tables plus their own downstreams means a dozen pages for a single late shard, arriving over 40 minutes as each job starts and finds nothing new. The graph is what lets you collapse them into one incident with a named root.
- Suppress by ancestry, not by time. The naive fix is to hold alerts for 15 minutes and dedupe. The right fix is: if an ancestor of this table is currently alerting on freshness, this alert is a symptom, so attach it to that incident rather than raising a new one. That requires a graph, which is one of the concrete arguments for building one.
- Late is not wrong, and the message must say which. A downstream table that is late because its source was late needs the message "waiting on orders_daily", not "data quality issue". Those prompt completely different behaviour from whoever reads it.
- The root is not always the loudest. The first alert to fire is the table with the tightest deadline, not the table that caused it. Sorting alerts by time is how people spend twenty minutes debugging a healthy mart.
Self-studyHold the dashboard, or serve stale with a banner4 min▶
Once the graph has told you which assets are affected, there is a decision per asset, and it is not a technical one. You can block the asset so nobody sees wrong numbers, or let it serve the last known good data with a visible notice. Both are defensible and the graph supplies the input for choosing.
| Situation | Choose | Because |
|---|---|---|
| A number will be quoted in a meeting or an invoice | Hold it | A wrong number that has been read aloud cannot be recalled, and the correction costs more than the delay |
| The reader wants a trend or a direction | Serve stale, banner it | Yesterday's shape is usually good enough, and blocking the panel produces a support ticket and a workaround spreadsheet |
| A customer sees the output | Hold it, and say so | Reputational cost is asymmetric, and a maintenance notice is a much smaller event than a wrong statement |
| The asset feeds a payment or a payout | Hold it, always | Money moved on bad data is recovered by a person and a process, not by a backfill |
| An internal panel that ten people glance at | Serve stale, banner it | The banner is the whole intervention. Holding it just moves the question into a chat channel |
Two implementation notes that make the difference between a policy and a slide. The banner text needs the age of the data and the reason, not the word "stale", because a reader has to decide whether it is usable. And the decision per asset should be recorded in the graph as a property of the node, agreed in advance with the owner, so that at 06:40 the on-call reads a decision rather than making one.
The impact query
★ 20 min · everyone buildsFour steps, and step one is deliberately low-tech because most people have no lineage store yet and can still finish this. The target is a query that takes a table and a column and returns the assets that read it, ranked by who sees the output. Then night 23 goes through it and produces a list of names.
1. Enumerate downstream assets by whatever means you have. Three sources, in the order they cost you time. Your warehouse query history, which is authoritative about who actually read the table. Your transformation project's own graph, which covers everything it builds. And your BI tool's dependency list. Union the three, by hand if necessary, into a list of assets.
Every row here that you cannot name is a candidate for the asset nobody documented. Service accounts are the interesting ones: a human running ad hoc queries is not a downstream asset, and a service account issuing the same query every morning at 07:15 certainly is.
2. Give the lineage store a shape. Two tables are enough, and starting with two tables you maintain by hand beats waiting for a platform. Edges carry the reads, assets carry the audience, and the audience column is what makes ranking possible.
The on_incident column is the decision from part 2, agreed in advance with the owner. Recording it here is what turns triage from a judgement call at 06:40 into a lookup.
3. Write the impact query. Given a table and a column, return every asset that reads it, transitively, ranked by audience and then by hops. The recursive term is what catches the asset two levels down that nobody thinks of.
Two details worth keeping. The source_column in ('ship_region', '*') clause is what makes one query serve both table-level and column-level questions: pass a column and you get the narrow answer, pass '*' alone and you get everything. And the depth guard is not paranoia; a hand-maintained graph acquires a cycle the first time somebody registers a round trip through a reverse-ETL job.
4. Walk night 23 through it, and write the list of names. Night 23 dropped ship_region and added promo_code. Run the impact query for that column and read the output as an incident, not as a table.
- Two assets read
ship_region. The region mix panel, audience internal, owner in growth, decision banner. And the CRM customer sync, audience customer, decision hold. Eighteen other readers of orders_daily are unaffected, and saying so in the first message is what stops the whole analytics team investigating. - The customer-facing one comes first. Sorting by audience puts the CRM sync at the top, which is the correct order to work in and the opposite of the order the alerts arrived in.
- Then step three: what already ran. The sync runs at 07:15 daily. Night 23's load landed before that, so it has already pushed a segmentation built on a missing column. That is a specific fact with a specific remediation, and no metadata monitor could have produced it.
- Then the names. The growth owner of the panel, so they know the last three days are wrong. The owner of the CRM sync, and whoever runs campaigns off those segments. The person who dropped the column, not to blame but because they are the only one who knows whether the change was meant to be additive. Four names, all from the graph plus the assets table.
- Now find the one you missed. Diff the query-history list from step 1 against the assets you registered in step 2. Whatever is in the first list and not the second is your undocumented asset. Register it, give it an owner and a decision, and note the date, because the interesting number is how long it had been reading a tier 1 table unrecorded.
The output to aim for after 20 minutes. One query, two small tables, and a five-line incident note for night 23 naming the two affected assets, the job that already ran on bad data, and four people. Compare that note against what a schema alert on its own gave you in session 4. Same detection, same minute, a different next hour.
Before session 8
35 min- Run the query-history sweep against your own tier 1 table and list every reader you cannot name. Most people find between one and three. Whatever you find, write down how long it had been there.
- Register your assets with an owner who is a person and an audience that distinguishes customer-facing from internal. A team alias in the owner column means nobody gets told.
- Agree the hold-or-banner decision for your three highest-audience assets with their actual owners, and record it on the node. Doing this in advance is the entire point.
- Put a coverage number next to your graph: what fraction of jobs emit, or what fraction of transformation files parse. A graph without that number gets trusted more than it deserves.
- Bring your monitor set from sessions 2 to 6 to session 8, including the thresholds. The bench scores what you actually built rather than an example.
Three questions
Night 23 dropped ship_region from a table with twenty downstream readers. What does column-level lineage give you that table-level lineage does not?
Table level answers a yes or no and returns all twenty. Column level answers a which, and an alert naming two affected assets gets acted on where an alert naming twenty gets a promise to investigate.
A lineage graph is built by runtime emission from instrumented jobs. What is the characteristic failure mode?
Runtime emission is the accurate route and it fails quietly. That is why an emission-based graph needs its own coverage number, tracking what fraction of scheduled jobs actually emitted in the last seven days.
A late source shard produces twelve freshness alerts across downstream tables over 40 minutes. What is the right handling?
Every one of the twelve is technically true and none is the incident. Suppressing by ancestry collapses them into one incident with a named root, and it needs a graph, which is one of the concrete arguments for building one.