Effective challenge, and why it almost never happens

The current guidance defines effective challenge as the critical analysis conducted by objective experts who evaluate model risk and effect appropriate changes throughout the model lifecycle. Every clause in that sentence is load bearing, but the last one is the one that decides whether any of it happens: effect appropriate changes. Analysis that does not change anything is not challenge. It is commentary that has been filed.

The guidance then says what it takes, and names three things. Appropriate expertise: the reviewer has to understand the method well enough to find the flaw, not merely to notice the documentation is thin. Sufficient independence to maintain objectivity: the reviewer's judgement must survive knowing which answer the business is hoping for. And the organizational standing and influence to effect any change: the reviewer's conclusion has to be able to move something.

Say those three out loud about your own institution and the pattern usually appears immediately. Expertise is a recruitment problem and most places have solved it. Independence is a structural claim and most places can point at a line on a chart. Standing is a question about power, and almost nobody can answer it with a document.

The 2011 text rested effective challenge on incentives, competence, and influence. The 2026 text rests it on expertise, sufficient independence, and organizational standing and influence. Competence became expertise, influence survived and gained the word standing in front of it, independence was named explicitly for the first time in this list, and incentives dropped out of it.

That last move is worth a thought rather than a shrug. Incentives are the honest description of why challenge fails: a reviewer whose bonus, promotion or next project depends on the business unit being pleased will find fewer problems, and no amount of expertise fixes that. Naming independence instead is a tighter, more auditable requirement, and it is easier to supervise against. But independence is a statement about position and incentives are a statement about motive, and the two are not the same thing. A validator can be structurally independent and still be paid, rated and staffed by people who want the model approved.

The practical reading: the list tells you what a supervisor will ask about, and incentives are now something you have to raise yourself. The guidance has not gone quiet on the subject entirely, and the place it reappears is conflicts of interest, which Part 2 covers.

Standing is easiest to identify by its consequences rather than its definition. In an institution where validation has it, three things are observably true. A validator has said no to something in the last two years and the thing did not happen. Launch dates are set after validation has an opinion, not before. And when a finding is graded, the grade is argued about by the business, which only happens when the grade has teeth.

Where standing is absent, the same three observations invert, and the tell is usually the second one. If the launch date was fixed in a commercial plan before validation started, then validation's only available outputs are approve, or approve with words attached. Refusal has been designed out of the process before any analysis happens, and the review is a schedule risk rather than a decision.

Independence is not only about who signs the validator's appraisal. The guidance is explicit that conflicts of interest include the misalignment of incentives between different reporting lines, such as model development and validation groups. That is a precise thing to look for: not bad faith, but two groups whose definitions of a good quarter are incompatible.

Development is measured on models shipped and performance improved. Validation, if it is measured at all, is usually measured on reviews completed and backlog cleared. Notice that both of those metrics reward the same behaviour, which is finishing. Neither of them rewards finding something, and finding something makes both numbers worse. An institution that wants challenge has to be willing for validation's throughput to fall in a year when validation is working.

The leader's version of this is a single question at the point of approval: who, other than me, is worse off if this model is delayed, and are any of them in the review chain. If the answer includes the person who graded the findings, the grade is worth less than it looks.

Most institutions place effective challenge inside a committee, and most committees are structurally incapable of providing it. A committee sees a summary, not the work. It meets after the analysis is complete and before the decision is due, which is the one window in which nothing can be changed cheaply. Its members mostly lack the expertise requirement for any given model on the agenda, because no group of eight people is expert in credit scorecards, market risk models and an anti-money-laundering engine at once. And it operates by consensus, which is precisely the mechanism that converts one person's serious objection into a note in the minutes.

None of that makes committees useless. It makes them the wrong place for the critical analysis and the right place for the standing. A committee that reliably backs a validator who has said no is granting the third requirement, which is the thing only it can do. A committee that reads a summary and asks two questions is performing challenge, and the difference is visible in whether anything on its agenda has ever been sent back.

None of these require you to understand the model. All of them are visible in the paperwork or in the room.

• Findings downgraded to observations. Ask how many findings were raised and how many became conditions. A review that produced only observations either found nothing, which is rare in a first validation, or found things and graded them away.
• Validation arriving after the launch date is fixed. Compare the date the go-live appeared in a plan with the date validation started. If the plan came first, the review could only ever be a formality with a schedule attached.
• A validator who has never been allowed to say no. Ask for the last refusal, by name and date. If there has never been one, you do not have evidence that refusal is available, and neither does the validator.
• Challenge that arrives as a document rather than a conversation. A report emailed a week before a committee cannot be argued with, and a finding nobody has had to defend out loud has not been tested. Critical analysis is a dialogue; a PDF is its residue.
• The model owner choosing the validator. Selection is where independence is really decided. If the developer picks the reviewer, or picks the vendor who reviews, the reporting line on the chart is decoration.

This is confused almost everywhere, and it produces one of two failures: audit duplicating validation badly with less expertise, or audit assuming validation has it covered and looking at nothing. The guidance is unusually clear about the split.

Internal audit would generally not duplicate model risk management activities such as model development or validation. Instead, its role is generally to evaluate whether the model risk management practices are rigorous and effective and whether related policies are implemented accordingly.

Read that as a difference in the object being examined. Validation examines the model. Audit examines the validation. The question audit is answering is not whether the scorecard is sound, it is whether the process that declared the scorecard sound was rigorous, and whether what the policy says would happen actually happened. Those are different skills and different evidence, and an auditor does not need to be able to refit a logistic regression to answer the second one.

Applied to Brackwater, audit's finding would not have been that the collections field is a leak. It would have been that a finding was raised, graded as an observation with no recorded rationale, given no owner, and closed at launch, and that the policy required conditions to be tracked to closure. That is a process failure, it is fully evidenced by the paperwork, and it is arguably the more damaging of the two findings because it applies to every model the institution has approved and not just this one.

The expectation is that validation generally occurs prior to a model's first use. The guidance also accepts that an urgent business need may mean the model is used first, which is worth knowing because most leaders assume the choice is between waiting and quietly proceeding. There is a third option and it is the supervised one.

Where a model is used before validation is complete, sound practice is to give greater attention to the model's limitations, to inform the relevant stakeholders of those limitations, and to place controls on its use: limits on what it may decide, and closer monitoring while it runs. The pattern is simple to recognise. The urgency is named rather than implied, the limitations are written down rather than discovered, the people relying on the output know what they are relying on, and the use is fenced until the validation catches up.

Brackwater did the first half of this and none of the second. There was an urgent business need, the model went live, and then nothing else on the list happened: the limitation that had been identified was recorded as an observation rather than communicated as a constraint, no limits were placed on the decisions the score could make, and monitoring was the standard monthly pack. The difference between the legitimate version and what happened is not the decision to launch. It is everything that was supposed to accompany it.