learn-model-risk-with-phoebe / Practitioner session 1 of 10
Learn Model Risk with Phoebe · Practitioner session 1 of 10

The inventory and the tiering call

Brackwater keeps a spreadsheet with one row per model. Each row has a name, an owner, a build date and a date somebody last looked at it, and the file is titled "model inventory". It is not one. The test is not how many rows it holds, it is whether the thing can answer a question about the portfolio: which models read the collections log, how many of the heavily used ones rest on the same bureau assumption, what else moves if that feed changes shape. None of those can be answered from names and dates. The columns are the whole design problem, and tonight they get designed.

🟠 Advanced Validators and model risk teams Data scientists who own a model 45 min
0-3 · Welcome 3-16 · What an inventory holds 16-28 · Scoring materiality 28-45 · Build-along and Q&A
Part 0

Where this sits in the practitioner track

Ten sessions, 45 minutes each, and a laptop from here on. Tonight is the one that comes before any model gets opened: the inventory that says what exists, and the tiering call that decides how much review each thing earns. Session 2 reads Brackwater's design before touching a number. Session 3 is the leak hunt. Sessions 4 to 7 are replication, challengers, outcomes analysis and out-of-time testing. Session 8 is monitoring, session 9 is writing findings, and session 10 runs the whole workflow against a model whose truth is known by construction. The worked example throughout is Brackwater's consumer scorecard, with 4,000 development applications at a bad rate of 0.292.

Live = presented in the 45 minutes Self-study = the full depth, after class 🔍 Every diagram zooms

★ What you walk out with tonight. A column list you can defend, derived from the one clause that does the work: the inventory has to support management of model risk at the individual level and in aggregate, and an aggregate question cannot be answered by a column you did not record. A way to score materiality that separates the measurable half from the judged half, so two people scoring the same model disagree about something specific. And the failure mode that no field catches on its own: Brackwater's scorecard was built to rank applicants and by year two was setting prices, which is a change of materiality that nothing in the register notices unless a person re-tiers it.

LiveThe clause that decides the columns4 min▶

It is common industry practice to maintain a comprehensive set of information for models under development and models in use, and the level of detail may reasonably vary with a model's complexity and usage. That much sounds like a filing exercise. The clause that turns it into a design problem is the next one: an effective inventory holds enough information to understand model risks and support the management of them both at the level of the individual model and in aggregate.

Read that as a requirement on the columns rather than on the rows. Individual-level management is satisfied by a row: who owns this, what is it for, when was it last validated, what is still open. Aggregate management is a question asked across rows, and you can only ask it about things you wrote down. Nobody can see that eleven models share one data feed if the feed was never a field. Nobody can see that a whole portfolio rests on one assumption about bureau comparability if assumptions live in PDFs. A register of names, owners and dates is a list of models. It is not an inventory, because the second half of the job is structurally impossible with it.

Real world

Brackwater's spreadsheet had a column called "data sources" and every cell in it said "warehouse". True, useless, and the reason nobody could answer the only question that mattered when the collections table was re-platformed: which models would change behaviour on the Monday. The answer turned out to be found by asking around for three weeks.

Self-studyThe words tonight uses precisely3 min▶
  • Inventory. The comprehensive record of models under development and in use, detailed enough to support management of model risk individually and in aggregate.
  • Exposure. The measurable half of materiality. How much rides on the output: portfolio size, business impact, the volume of decisions the thing touches.
  • Purpose. The judged half. What the output is for, with uses that meet a regulatory requirement or manage financial risk weighing more than uses that size a campaign.
  • Materiality. Purpose together with exposure. In the current guidance these two are what determine a model's materiality, and overall model risk is inherent risk read in the context of it.
  • Tier. An editorial device, not a regulatory object. The guidance asks that the nature and rigour of review align with the model's approach, use and materiality; a numbered tier is one house convention for writing that alignment down.
  • Aggregate risk. Interactions and dependencies among models, and reliance on common assumptions, data or methodologies. The reason the columns look the way they do.
  • Immaterial. A scored outcome, not an exemption. An immaterial model is still identified, and still monitored for the conditions that would make it material.
  • Re-tier. The act a person performs when purpose or exposure moves. No field does it automatically, which is the subject of Part 2.
Part 1 · covers what the inventory has to hold

A register says a model exists. An inventory answers a question.

12 min live

Every column in a good inventory earns its place by being the answer to a question somebody will ask under pressure. Drop the aggregate clause and you can justify four columns. Keep it and the list roughly doubles, because dependencies, shared assumptions and shared methodologies are all cross-row properties, and a cross-row property that nobody recorded does not exist as far as the portfolio is concerned.

The aggregate clause is what turns the left column into the right one. A register Model name Owner or team Date added Date somebody last looked Answers one question only: does this model exist An inventory Purpose, in one sentence, and the decision it feeds Exposure, as a measured quantity rather than a word Tier, set from purpose and exposure together Upstream feeds, every one of them, named Shared assumptions and shared methodologies Last validation, next review, open findings The question the left column cannot answer, at any row count Which models depend on the collections log, and what happens to all of them if that feed changes shape. Dependencies, common data and common methodologies are cross-row properties. An unrecorded column cannot be grouped by.
🔍 Click to zoom - the extra columns on the right are not bureaucracy, they are the only way an aggregate question can be asked at all.
LiveAggregate risk is a property of the portfolio, not of any model in it4 min▶

Aggregate model risk reflects interactions and dependencies among models, and reliance on common assumptions, data or methodologies. Every clause in that sentence describes something that is invisible inside a single model document, because a single model document has no reason to mention what the model next door is doing.

The practical consequence is that the inventory is the only artefact in the whole discipline that can see this. A validation report covers one model. A monitoring pack covers one model. The inventory is where twenty individually reasonable models turn out to share one vendor feed, one macro assumption and one preprocessing library, and where somebody can notice that a correction to any of the three moves the whole book at once. If your columns cannot express that, the aggregate view is not being done anywhere, no matter how many committees discuss it.

LiveWhat the documentation is actually for3 min▶

Documentation gets treated as a tax, which is why it gets written last and read never. It is worth being specific about the jobs it does, because each one implies a field. It supports continuity of operations, so the model survives the departure of the person who built it. It tracks recommendations, responses and exceptions, so an argument that was settled two years ago stays settled and the reasoning is recoverable. And it supports the management of remediation, so a finding has a state rather than a memory.

Read as column requirements, those three sentences produce most of the right-hand list. Continuity means owner, purpose, feeds and assumptions written where a stranger can find them. Tracking recommendations and exceptions means a findings register that the inventory points at, including the exceptions that were granted and by whom. Remediation management means each open finding carries an owner and a date, which is what separates a condition from a sentence.

Self-studyModels under development belong in it too3 min▶

The scope covers models under development as well as models in use, and teams routinely read this as pedantry. It is not. A model that is being built is a model that somebody is already planning around, and the gap between "we are building a pricing model" and "the pricing model exists" is precisely the window in which validation has to be scheduled, data access negotiated and a tier assigned. A model that appears in the inventory on the day it goes live has already skipped the part where the review could have changed anything.

There is a second reason, which is that the pipeline is where the aggregate view gets its warning. Three teams independently building on the same new feed is a concentration that has not happened yet, and the cheapest moment to notice it is while all three are still choosing.

Part 2 · covers materiality and the tiering call

Score the materiality, do not describe it

11 min live

Model purpose, together with model exposure, determines model materiality. Two halves, and they behave completely differently. Exposure is measurable, so two people who disagree about it are disagreeing about a quantity and can go and look. Purpose is judged, so two people who disagree about it are disagreeing about what the output is for, which is a conversation with the business rather than a query. Write both down separately and the tier becomes reproducible; blend them into a single adjective and the tier becomes whatever the loudest person in the meeting thinks.

Exposure is measured. Purpose is judged. The tier is where the two meet. Regulatory or financial-risk use Operational decisions Internal or marketing Tier 2 targeted review Tier 1 full validation before use Tier 1 full validation before use Tier 3 recorded and monitored Tier 2 targeted review Tier 1 full validation before use Tier 3 recorded and monitored Tier 3 recorded and monitored Tier 2 targeted review Small exposure Material exposure Large exposure A change of purpose is a change of materiality Brackwater's scorecard was built to rank applicants and by year two was setting prices. The dashed arrow is that move. Three tiers is a house convention. What the guidance asks is that rigour align with the model's approach, use and materiality.
🔍 Click to zoom - no field in the register moves on its own. Somebody has to notice the purpose changed and re-tier it.
LiveScoring without adjectives4 min▶

The failure pattern is a tiering sheet whose inputs are words: "high impact", "significant", "business critical". Every model owner believes their model is business critical, and they are not being dishonest, they are answering a question that has no shared unit. Replace the words with two recorded quantities and one recorded sentence and the argument gets much shorter.

For exposure, pick a measurable thing the model touches and record the number with its unit and its date: balances scored, decisions per year, provisions influenced, notional covered. Any of these is defensible; none of them is a synonym for the others, so the inventory has to say which one it holds. For purpose, record what the output decides, in a sentence naming the decision and who acts on it, and record whether that use meets a regulatory requirement or manages financial risk, because those uses weigh more. The tier is then a stated function of the two, applied the same way to every model, and if somebody wants a different tier they have to argue with the function rather than with you.

Real world

Brackwater's original tiering sheet had one column called "criticality" with the options low, medium and high, chosen by the model owner. Every model in the consumer book was high. The sheet had been in use for four years and had never once caused a review to be scheduled differently, which is the operational definition of a control that does not exist.

LiveImmaterial is a score, not an exemption3 min▶

Rigour is commensurate with risk, which means some models correctly get a paragraph rather than three weeks. The part that gets dropped is the second half: a model judged immaterial is still identified, and still monitored for the conditions under which it would stop being immaterial. Those conditions are the useful artefact, and they have to be written at tiering time, while the reasoning is fresh, not reconstructed afterwards by someone explaining a loss.

A condition is specific and checkable. "Re-tier if this model is extended beyond the one product line." "Re-tier if the balances it touches pass the threshold in the tiering function." "Re-tier if its output starts feeding a price rather than a ranking." Each of those is a sentence somebody can be given ownership of, and each of them is a column in the inventory rather than a paragraph in a PDF, because a paragraph in a PDF cannot be reported on.

Self-studyWhy purpose is the half that moves silently3 min▶

Exposure changes loudly. Balances grow, volumes appear in a monthly pack, and somebody somewhere is already reporting the number, so a threshold crossing can be automated. Purpose changes quietly, because a change of purpose usually looks like a good idea implemented by a competent person: the scorecard ranks well, pricing needs a risk input, the score is right there, and the integration takes an afternoon. No document is amended, because no document was wrong. The model still does what it always did.

What changed is what the output decides, which is one of the two inputs to materiality. This is also the seam with model use, which session 2 covers: using a model beyond its intended purpose introduces additional uncertainty and risk, and when a use is extended the sound practice is additional analysis of the new usage and its limitations, plus a review of the existing controls. For tonight the inventory consequence is narrow and practical. Put a "purpose last confirmed" date next to the purpose sentence, and make confirming it an act somebody performs, because an undated purpose field ages into fiction without ever being edited.

Part 3 · build-along

Design the register

17 min build-along

Build the schema first, then prove it earns its keep by asking it a question no register could answer. The code below is deliberately plain: an inventory that needs a platform before it can exist will not exist. A dataframe with the right columns, kept honestly, beats a tool with the wrong ones.

Write the model identifier scheme before the first row. Stable, never reused, never encoding anything that can change. An id that embeds the owning team becomes wrong at the first reorganisation, and every downstream reference becomes wrong with it.

Give purpose a sentence and a date. What the output decides and who acts on it, plus the date somebody last confirmed that is still true. The sentence is what gets compared against reality; the date is what makes the comparison an event rather than a hope.

Record exposure as a quantity with a unit. One measure, stated, the same measure for every model of that kind. Two models whose exposure fields hold different units cannot be compared, and an inventory that cannot rank its own rows cannot allocate review.

Record every upstream feed as a list, not a sentence. This is the field that makes the aggregate view possible, and prose destroys it. A list can be exploded and grouped; "various warehouse tables" cannot.

Record shared assumptions the same way. Name the assumption in a form another team would recognise, so that two models holding the same assumption use the same string. This one needs a controlled vocabulary or it decays within a year into thirty phrasings of one idea.

Close the loop with validation dates and open findings. Last validation, next review implied by the tier, and the count of findings still open. These three make the inventory a work queue instead of an archive, which is the difference between a file that is maintained and a file that is recreated annually.

Then ask it an aggregate question immediately. Group by feed on day one, while the register has five rows. If the question is awkward to ask at five rows it will be impossible at two hundred, and the time to change the schema is now.

★ Try it now - the schema, chosen so the portfolio can be queried
import pandas as pd

# One row per model. Every column is here because some question needs it,
# and the cross-row questions are the ones that decide the design.
COLUMNS = [
    "model_id",            # stable, never reused, encodes nothing that can change
    "name",
    "owner",               # a named person, not a mailbox
    "status",              # "in development" or "in use" - both are in scope
    "purpose",             # what the output decides, in one sentence
    "purpose_confirmed",   # the date somebody last checked that sentence
    "regulatory_or_risk",  # True if the use meets a regulatory need or manages financial risk
    "exposure_measure",    # the unit, stated
    "exposure_value",      # the quantity, in that unit
    "tier",                # a stated function of purpose and exposure
    "retier_triggers",     # the conditions that would change the tier
    "upstream_feeds",      # a list, so it can be exploded and grouped
    "shared_assumptions",  # a list, from a controlled vocabulary
    "last_validation",
    "next_review",
    "open_findings",
]

rows = [
    {
        "model_id": "MR-0141",
        "name": "Consumer application scorecard",
        "owner": "A. Okafor",
        "status": "in use",
        "purpose": "Rank consumer applications at origination and set the approval cutoff",
        "purpose_confirmed": "2026-03-18",
        "regulatory_or_risk": True,
        "exposure_measure": "applications decided per year",
        "exposure_value": 180000,
        "tier": 1,
        "retier_triggers": ["output feeds price", "extended beyond consumer book"],
        "upstream_feeds": ["bureau_v3", "core_ledger", "collections_log"],
        "shared_assumptions": ["bureau score comparable across vintages"],
        "last_validation": "2026-03-18",
        "next_review": "2027-03-31",
        "open_findings": 4,
    },
    {
        "model_id": "MR-0207",
        "name": "Collections propensity",
        "owner": "R. Villanueva",
        "status": "in use",
        "purpose": "Order the outbound contact queue for delinquent accounts",
        "purpose_confirmed": "2025-11-04",
        "regulatory_or_risk": False,
        "exposure_measure": "accounts contacted per year",
        "exposure_value": 41000,
        "tier": 3,
        "retier_triggers": ["output feeds a provision or a price"],
        "upstream_feeds": ["core_ledger", "collections_log"],
        "shared_assumptions": ["contact outcome recorded within 30 days"],
        "last_validation": "2025-11-04",
        "next_review": "2027-11-30",
        "open_findings": 0,
    },
]

inventory = pd.DataFrame(rows, columns=COLUMNS)
print(inventory[["model_id", "tier", "exposure_value", "open_findings"]])
★ Try it now - ask the question a register cannot answer
# Explode the feed lists so one model becomes one row per feed it depends on.
pairs = inventory.explode("upstream_feeds")

concentration = (
    pairs
    .groupby("upstream_feeds")
    .agg(
        models=("model_id", "count"),
        tier1_models=("tier", lambda t: int((t == 1).sum())),
        exposure_behind_it=("exposure_value", "sum"),
    )
    .sort_values("exposure_behind_it", ascending=False)
)

print(concentration)
# collections_log appears under two models, one of them tier 1.
# Nothing in either model document says so, because neither document
# has any reason to mention the other model.

# The same shape answers the assumption question, which is the one
# that tends to be worse, because assumptions are shared silently.
shared = (
    inventory.explode("shared_assumptions")
    .groupby("shared_assumptions")["model_id"]
    .apply(list)
)
print(shared)

Run the concentration query before you add the sixth row. An inventory is a schema decision pretending to be an administrative task, and schema decisions get expensive per row. Nine models is an afternoon of backfilling. Ninety is a project nobody funds, and the register that results is the one that has a "data sources" column where every cell says "warehouse".

Check yourself

Three questions

Your register holds model name, owner, build date and last review date for every model in the bank. Which part of the inventory expectation is it structurally unable to meet?

An effective inventory holds enough to understand model risks and support managing them at the individual level and in aggregate. The four columns serve the individual level adequately. No amount of rows lets you group by a feed you never wrote down, which is why the aggregate half is impossible rather than merely difficult.

A model is scored immaterial on its purpose and exposure. What is the correct treatment?

Rigour is commensurate with risk, so a light touch is correct. Immaterial is a score rather than an exemption: the model stays identified and gets monitored for the conditions under which it would stop being immaterial. Those conditions belong in a field, written at tiering time, with an owner.

Brackwater's scorecard was built to rank applicants. By year two its output is setting prices. Nothing in the model changed and no document is wrong. What happened to its materiality, and what in the inventory catches it?

Purpose together with exposure determines materiality, so a change of purpose is a change of materiality. Exposure moves loudly and can be automated against a threshold. Purpose moves quietly, usually as a sensible integration nobody documents, which is why the purpose sentence needs a confirmation date and a named re-tier trigger.

Homework

Before session 2

40 min

Take whatever your organisation calls its model inventory and answer one question against it without asking a human: which models depend on the single most widely used data feed. Time yourself. If the answer takes more than a query, write down which column was missing, because that column is your finding. Then pick one model you own and fill in the two halves of materiality properly: the exposure measure with its unit and value, and the purpose sentence with the decision and the person who acts on it. Bring the purpose sentence to session 2, which reads a model's design before touching any number, and starts with exactly that sentence.

Sources covered

What tonight maps to

✓ The inventory expectation Common industry practice is a comprehensive set of information for models under development and in use, with the level of detail varying with complexity and usage. An effective inventory holds enough to understand model risks and support managing them at the individual level and in aggregate.
✓ Materiality as purpose plus exposure Under the revised interagency guidance of 17 April 2026, model purpose together with model exposure determines model materiality, and overall model risk is inherent risk read in the context of it. The nature and rigour of validation generally align with the model's approach, use and materiality. SR 11-7 is superseded.
✓ Aggregate risk, and the columns it implies Interactions and dependencies among models, and reliance on common assumptions, data or methodologies. Recorded as lists so the portfolio can be grouped rather than described.
✓ What the documentation is for Continuity of operations; tracking recommendations, responses and exceptions; and supporting the management of remediation. Each of the three implies a field.
✓ Immaterial models, and re-tiering Identified and monitored for the conditions that would make them material. Brackwater's scorecard was built to rank applicants and by year two was setting prices, which is a change of materiality no field catches on its own.
◐ Model use beyond its intended purpose Session 2. Tonight it appears as the re-tier trigger; the additional analysis and control review that an extended use calls for belong with conceptual soundness.
◐ Findings, severity and remediation tracking Session 9 owns the taxonomy. Tonight the inventory only has to carry the count and point at the register.
· What AUC and a base rate are Not covered by design. learn-model-evaluation-with-phoebe defines the metrics; here they appear only as evidence a validator weighs.
· Governance frameworks of every kind Not covered by design, in any session. The EU AI Act, the NIST AI RMF, ISO/IEC 42001, MAS and FEAT belong to learn-ai-governance-with-phoebe.

Session 1 cheat sheet

The clause that decides the columnsSupport managing model risk at the individual level and in aggregate. The second half is what the extra fields are for.
A register is not an inventoryNames, owners and dates answer whether a model exists. Nothing else.
Lists, not proseFeeds and shared assumptions have to be groupable. "Various warehouse tables" cannot be grouped by.
Materiality is two halvesPurpose together with exposure. Record them separately so a disagreement is about something specific.
Exposure gets a unitA quantity with its unit and its date. Two models measured in different units cannot be ranked.
Immaterial is a scoreStill identified, still monitored for the conditions that would make it material. Write those conditions at tiering time.
Purpose moves silentlyExposure grows in a monthly pack. Purpose changes in an afternoon integration nobody documents. Date the purpose field.
Development countsModels under development are in scope, because that is the only window in which review can still change something.