Where tonight sits on the ladder
Four paths remain to Ironwood's customers table after MFA, key rotation and the private bucket. Tonight closes one: the path that starts at a package registry, lands on the CI runner, reads the ETL service key out of the runner's environment and authenticates to the warehouse as the pipeline. Pinning and verifying dependencies takes the count from four to three. The three that remain are the two insider paths and the Friday admin grant, none of which is a door in the engineering sense. Tonight is the last door.
★ What you walk out with tonight. A lockfile for the pipeline's dependencies, committed, with hash verification on every install so a build cannot silently pull a different artifact. A list of every secret the CI runner's environment holds and the ones the build step does not need removed. Build and deploy split into two steps, with only deploy holding a credential, and that one short-lived. And the inventory file's "closed by" column filled for the CI runner edge.
LiveWhy this door looks like normal engineering3 min▶
A phished login leaves a login. A public bucket leaves a URL in an index. A supply-chain compromise leaves a green build. The runner did exactly what it does every night: resolved the dependency list, downloaded what the registry offered, installed it, ran the pipeline. Every step is the legitimate process. The only thing that changed is that one of the packages was not the one the team reviewed, and nothing in a standard build checks that.
That is why the control is not a scanner and not a firewall. It is a written record of exactly which artifact, by hash, the team agreed to run, and an install step that refuses anything else. The build turns red on the night the artifact changes, which is the night it needed to.
Ironwood's pipeline declared 31 direct dependencies. The resolved install pulled 214 packages. Nobody on the team could name more than a dozen, and the requirements file said >= on every line, so the 214 could be a different 214 tomorrow. The runner that installed them held the ETL key, the orchestrator token and a cloud deploy role, because it had always been simpler to give the runner everything.
Self-studyThe vocabulary for the supply-chain lane3 min▶
- Dependency - code your pipeline runs that your team did not write. Direct if you asked for it, transitive if something you asked for asked for it. Transitive dependencies outnumber direct ones by five to ten to one on a typical data pipeline.
- Lockfile - the fully resolved list of every package and its exact version, committed next to the code. A requirements file says what you want; a lockfile says what you got.
- Hash verification - the install step checks each downloaded artifact against a digest recorded in the lockfile and refuses a mismatch. Same name, same version, different bytes: refused.
- CI runner - the machine that installs dependencies and runs the build. Whatever is in its environment is available to every package it installs. That is the whole problem in one sentence.
- Build step - produce and test an artifact. Deploy step - put the artifact where it runs and give it what it needs to run. Only one of these needs a warehouse credential, and it is not the one that installs code from the internet.
- Short-lived credential - one that is issued for the run and expires in minutes. A stolen short-lived credential is a door that closes itself.
Your pipeline is mostly other people's code
9 min liveThe supply-chain surface of a data stack has five lanes, and they share one property: each is code that arrives from outside and runs on a machine that holds a credential. The path is the same in every lane. The registry is the door, the runner is what it yields, the key is what the runner holds.
LiveThe five lanes of the supply-chain surface3 min▶
| Lane | What arrives | Where it runs | What it can reach there |
|---|---|---|---|
| Python packages | The pipeline's libraries and their transitive tree | CI runner, orchestrator workers | Everything in the process environment |
| dbt packages | Macros and models pulled by the packages file | Wherever dbt runs, with the warehouse profile | The warehouse, as the transformation role |
| Connectors | Source and destination plugins, often binaries | Ingestion workers | Source credentials and the landing zone |
| Orchestrator plugins | Providers, operators, hooks | The scheduler and every worker | Every connection the orchestrator stores |
| Container base images | The operating system under all of the above | Everywhere | Everything, before your code starts |
The fourth column is the one that matters. In every lane the answer is "whatever that machine holds", which is why the control has two halves: make sure the code that arrives is the code you reviewed, and make sure the machine it arrives on holds as little as possible in case the first half fails.
LiveThe Ironwood path, edge by edge3 min▶
Internet to CI runner: a package the build pulls arrives changed. Not a new package the team added; a version of one they already trusted, republished with something extra. The build installs it because the requirements file says any version at or above a floor. CI runner to ETL service key: the runner's environment holds the key, so anything running on the runner can read it. This edge has no control that closes it, because the runner holds the key legitimately; the pipeline needs it. Key to warehouse: the key authenticates. Warehouse to customers: a SELECT as the ETL role, which reads every table because it loads every table.
Session 2 rotated the key, and the rotation did nothing to this path: the runner got the new key, because the runner is a legitimate holder. The graph shows the distinction. Rotation closed the copy in git history and left the copy in the runner's environment, which is the copy this path uses. Pinning closes the first edge, and the count moves from four to three.
Ironwood's nightly build ran at 02:00. The pipeline's query history shows the ETL role reading customers every night at 02:14, as it should, to build the marketing extract. A malicious read at 02:15 by the same role from the same runner would be one more line in the same log, indistinguishable by identity. Session 5 is about what would distinguish it: the volume and the destination. Tonight is about the package never landing.
Self-studyWhy the runner's environment is the real finding3 min▶
Pinning takes the path count from four to three, and the honest observation is that it does so by closing one door out of a class that keeps growing. Registries get new packages every day; a team adds a dependency every sprint; the lockfile is regenerated every time somebody upgrades. Each regeneration is a moment when a changed artifact can enter, reviewed or not. The control holds as long as the review holds.
So the second half of the design is not optional. A runner that holds the ETL key, an orchestrator admin token and a cloud deploy role is a runner where a single missed review costs the estate. A runner that holds a registry read token and nothing else is a runner where the same miss costs a build. The environment is what decides which of those you are running, and it is a list you can print tonight.
- Every secret on the runner is reachable by every package. Not by a clever package; by any package. Environment variables are read with one line.
- A runner that needs a warehouse key to run tests is a design smell. Tests run against a fixture. If they cannot, the pipeline is not testable, and that is the finding.
- The credential belongs to the step that loads. And that step should be issued the credential for the run, not hold it between runs.
Pin, verify, minimise, and separate
8 min liveFour moves, in the order that pays off soonest. Pin what runs. Verify that what arrives is what was pinned. Minimise what the machine that installs it holds. Separate the step that installs from the step that needs the credential, and make the credential short-lived.
LivePin with a lockfile, verify against recorded hashes3 min▶
A requirements file with version floors is a wish. The lockfile is the fully resolved tree, every package at an exact version, with the digest of the artifact that was downloaded when the lock was made. It is committed next to the code, so the version of the pipeline in git names the exact bytes it runs.
Verification is the half that makes the lockfile a control rather than documentation. With hash checking on, the installer compares each downloaded artifact to the digest in the lockfile and refuses a mismatch. Same package name, same version number, different bytes: the build fails. That failure is the whole point. A republished artifact under an existing version, a registry that served something else, a mirror that was tampered with: all of them are the same red build.
Ironwood's first locked build failed on three packages. Two were pinned to versions that no longer existed on the registry, which meant the previous nightly builds had been silently resolving to something newer for months. The third had a hash mismatch against a version the team had never upgraded, and the investigation of that one is the story the platform team tells new hires.
LiveMinimise what the runner holds, and separate build from deploy3 min▶
Print the runner's environment. Every secret on the list is reachable by every one of the 214 packages. Then ask of each one: does the step that installs code need this to do its job? A registry read token, yes. A warehouse key, no; the build does not query the warehouse, it builds an artifact and tests it against a fixture. An orchestrator admin token, no; nothing in a build needs to administer the scheduler. A cloud deploy role, no; that is the deploy step's job.
The separation follows from the list. The build step installs, tests, produces an artifact and its digest, and holds a read token. The deploy step fetches the artifact by digest, requests a short-lived credential from an issuer that trusts the pipeline's identity, loads to the warehouse, and lets the credential expire. It installs nothing from the internet, so a changed package never runs where a credential is. The credential is issued for minutes, so even a copy of it is a door that closes itself.
The cost is a second job in the pipeline and an issuer to configure. Ironwood's split took a day and a half, most of it discovering which tests had quietly been hitting the warehouse.
Self-studyLockfile diffs are code review, and the lane teams forget3 min▶
A lockfile changes when somebody upgrades, and an upgrade is the moment a changed artifact can enter with a reviewer's approval. So the lockfile diff is part of the pull request, read like code. For each version change: is this a version the maintainer published, does the changelog account for it, did the transitive tree grow, and by what. A diff that adds forty packages to upgrade one is a question, not a merge.
The lane teams forget is the one that is not "code" in the repository's sense. dbt packages are declared in a packages file and pulled at run time by the dbt process, which holds the warehouse profile. Connectors are configured in a UI and updated by the platform. Orchestrator providers arrive with the orchestrator image. None of these appear in the Python lockfile, and each runs with a credential. Every one of them needs the same treatment: an exact version written down, a check that the version running is the version written, and a diff when it changes.
- dbt packages: pin to an exact version or a commit in the packages file, and diff it in review like the lockfile.
- Connectors: record the version running against each source; the platform's own changelog is the diff.
- Base images: pin by digest, not by tag. A tag is a name that can move; a digest is the bytes.
- Orchestrator plugins: they arrive with the image, so pinning the image by digest pins them.
Self-studyThe honest limit of pinning2 min▶
Pinning and verification defend against a changed artifact: a package that was one thing when the team reviewed it and another thing when the build pulled it. They do not defend against a maintainer who was malicious from the first version, or a package that was fine for three years and whose maintainer account changed hands last month with the code changing in the release the team then reviewed and pinned. Review catches some of that. Nothing catches all of it.
Which is why the runner still holds as little as possible even after every dependency is pinned and verified. The first half of the control makes the door hard to open; the second half makes the room behind it nearly empty. On the bench, pinning is the 4 to 3 rung. Minimising the runner does not move the count at all, and it is the reason the count stays at three when a pin is wrong.
Pin, verify, minimise
★ 20 min · everyone buildsEvery command runs against your own pipeline's repository and your own CI configuration. The output is a committed lockfile with hash verification on, a runner environment shorter than it was, a build step that holds no warehouse credential, and a filled "closed by" cell for the CI runner edge.
Produce the lockfile for the pipeline's dependencies and commit it. Resolve the full tree from the requirements file with digests recorded for every artifact. The exact tool differs by ecosystem and the output does not: one file, every package at an exact version, a hash per artifact, committed next to the code it locks.
Ironwood's count was 214 from 31 direct entries. The commit message is the point: from this commit on, the pipeline's git history names the exact bytes each version ran.
Enable hash verification on install, in CI and nowhere else first. The install step reads the lockfile and requires every artifact to match its recorded digest. Run it in CI before changing developer machines, so the first mismatch is a red build rather than a confused colleague.
Expect the first run to fail. Ironwood's failed on three packages, and every one of the three was a finding about the previous year's builds, not about the lockfile.
List every secret the CI runner's environment holds, and remove the ones the build step does not need. Print the variable names, never the values. For each name ask: does installing and testing code need this? Then remove every "no" from the build job's configuration. The list of names goes into the inventory's credential lane; the values go nowhere.
A secret that was on the runner and is now removed was readable by every package for as long as it was there. Rotate it after removing it, by the five-step sequence from session 2, and write the rotation date next to it.
Split build from deploy so that only deploy holds a credential. Two jobs in the pipeline configuration. Build installs from the lock, runs tests against a fixture, publishes the artifact and its digest. Deploy fetches by digest, requests a short-lived credential from the issuer, loads to the warehouse, and exits. The deploy job installs nothing.
The tests are where the split gets hard. Ironwood found six tests that had been reading the warehouse for years because it was easier than building a fixture. Building the fixture was a day; those six tests were the day.
Diff last month's lockfile against today's and read every version change. The lockfile is now in git, so its history is readable. For each changed line: who changed it, in which pull request, and did the transitive tree grow. Make this a standing part of code review for any pull request that touches the lock.
A lock that changed and nobody can say why is the diff-review finding. A lock that has not changed in a year is a different finding: the pipeline is running twelve months of unpatched dependencies, and both are worth writing down.
Update the inventory file's "closed by" column for the CI runner edge, with the date. One edge closes tonight: internet to CI runner, closed by pinned and verified dependencies. The runner-to-key edge stays open, because the deploy step holds a credential legitimately; write the minimisation and the short-lived credential in its cell so the reader knows why the edge is small rather than gone.
The lockfile is a control only while somebody reads its diffs. Verification stops the artifact changing under a pinned version. It cannot stop a reviewer approving a pull request that pins a bad version. The diff review in step 5 is the human half, and a pipeline where the lock is regenerated by a bot and merged without a reader has the first half of the control and not the second.
Before session 5
30 min- Lock every pipeline that deploys to the warehouse, with hashes, verification on in CI. Count the first-run failures across all of them and write the count down; each one is a month somewhere in the past when a build ran something nobody reviewed.
- Print the environment variable names on every runner and worker that installs code. Remove every secret the install step does not need. Rotate each one you removed.
- Walk the four lanes that are not the Python lockfile: dbt packages, connectors, base images, orchestrator plugins. For each, write down the exact version running and whether anything checks it. A blank is the finding.
- Find every test that touches the warehouse. Count them. That count is the size of the build-deploy split, and it goes to whoever plans the next sprint as a number, not as a complaint.
Three questions
Session 2 rotated the ETL service key. Why did rotation not close the CI runner path, which also uses that key?
Rotation closes a leaked copy. The runner holds the current copy because the pipeline needs it. The path is closed at its first edge instead, by pinning and verifying what lands on the runner, and made small by the runner holding as little as possible.
Hash verification is on and every dependency is pinned. What does that not defend against, and what is the answer to it?
Pinning defends against a changed artifact. It cannot tell a trusted version from a malicious one the team reviewed and approved. That gap is why the second half of the design exists: the room behind the door is nearly empty.
A team's CI runner needs the warehouse key to run its test suite. What is that a sign of?
Every package the build installs can read every secret on the runner. A build that needs a warehouse key has put the credential in the one place code from outside runs. Ironwood's split found six such tests, and they were the whole cost of the split.