Conceptual soundness

Validating conceptual soundness involves assessing and documenting model design, including key modelling choices, assumptions, qualitative judgments and data selection, together with construction and developmental testing. Sound practice subjects modelling aspects to critical analysis by evaluating both the quality and the extent of developmental evidence.

Three things are being assessed, and four things sit inside the first of them. That structure is worth holding because it is where findings get filed. Design covers the choices a person made: which modelling approach, which assumptions, which judgments where the data ran out, and which fields were selected in and out. Construction covers whether the thing that was built is the thing that was described. Developmental testing covers what was done to try to break it before anybody was asked to approve it. A finding that names its component travels; a finding that says "the model is not very good" gets argued with forever.

• Design. Key modelling choices, assumptions, qualitative judgments and data selection. All four are decisions somebody made, and all four are reviewable in prose.
• Construction. Whether what was built matches what was described. The gap between the document's model and the repository's model is a finding in its own right.
• Developmental testing. What the developer did to test the thing before submitting it. Ranges from out-of-sample and out-of-time testing to comparing alternative assumptions and methodologies, and to critically assessing data quality, relevance and inputs.
• Quality of evidence. Whether the evidence offered is any good: correctly computed, on the right sample, measuring the claim it is attached to.
• Extent of evidence. Whether there is enough of it. One chart is one test. A testing programme is a different object and looks different on the page.
• Purpose statement. The clear statement of what the model is for, which an effective development process begins with, so that development is aligned with intended use.
• Intended use, and beyond it. Effective use depends on a clear understanding of the model's limitations. Using a model beyond its intended purpose introduces additional uncertainty and risk.
• Interpretability and benchmarking. Two practical routes named inside conceptual soundness for models where assessing theoretical construction is not the useful move.

Development begins with a clear statement of purpose, so that the work is aligned with the intended use. That sentence is doing more than it appears to. It makes purpose a development artefact rather than a framing device, and it gives every subsequent choice something to be answerable to. A model that cannot produce its purpose statement has no standard against which its data selection, its sample window or its cutoff can be judged, so every review of it degenerates into taste.

The second half is just as useful to quote in a room: model development is not a purely technical exercise, and the developer's judgment shapes the conceptual design, the inputs, the assumptions and the methodology. This is not a criticism of developers, it is a description of the job. Judgment is unavoidable and often correct. What conceptual soundness asks is that the judgments be visible, so they can be examined by somebody who did not make them. An undocumented judgment is not a neutral omission; it is the part of the model nobody can review.

Model testing is a core component of development, and the range it covers is broader than the one number most documents carry. It runs from out-of-sample and out-of-time testing, through comparing alternative assumptions and alternative methodologies, to critically assessing data quality, relevance and the inputs themselves. Four different kinds of activity, and only the first is routinely present.

Read that list as a checklist against the document in front of you and the gaps become specific rather than vague. Was the model scored on a later vintage, or only on a holdout drawn from the same one. Was any alternative assumption tried, and what happened. Was any alternative methodology fitted, even a simple one, or is the chosen approach the only one that appears in the document. Was the data itself assessed, or only consumed. Each gap you find is a statement about extent, which is the second half of the pair, and each one is answerable by the developer without a fight because none of them accuses anybody of being wrong.

For some models, evaluating theoretical construction is the right move: there is an established theory, the model claims to implement it, and a reviewer can check the implementation against the claim. For others that route is unavailable or unhelpful, and interpretability measures or benchmarking to other models are more practical ways to get at the same question.

Two things follow. The first is that benchmarking is named inside conceptual soundness rather than alongside it. It is one practical technique for assessing whether a model's approach holds up, not a separate component of validation, and a page that presents it as a fourth pillar is making an editorial claim rather than reporting the guidance. Session 5 builds the Brackwater challenger and treats benchmarking properly as a technique. The second is that the choice between these routes is itself a judgment the validator should record, because "we benchmarked instead of assessing the theory" is a defensible decision and an indefensible silence.

Sound practice subjects the modelling aspects to critical analysis by evaluating both the quality and the extent of the developmental evidence. Both, named together, and they fail in different ways. Quality fails when the evidence is wrong: the wrong sample, a metric that does not speak to the claim, a chart whose axis hides the thing it is supposed to show. Extent fails when the evidence is fine and there is not enough of it to support the weight being placed on it.

The practical test is embarrassingly simple. Count the tests. Three charts on a slide are evidence of one test presented three ways, not of a testing programme, and a document that carries one performance number with three visualisations of it has produced one piece of evidence. Then ask what each test could have concluded if the model were bad. Evidence that had no way of coming out badly is not evidence of anything, however well it is drawn.

Extent is also the finding that is easiest to write without a fight, which matters more than it should. "This is wrong" is a confrontation. "This claim rests on one sample from one vintage, and the model is being approved for the following year's book" is a description, and the usual response to it is somebody going away and producing the missing evidence, which is the outcome you wanted.

Purpose statement first, then assumptions, then data selection, then method, and only then any number. The order is not fussiness. A validator who reads the results first is anchored for the rest of the review, and anchoring here has a specific shape: once 0.863 is in your head, every design choice you read afterwards gets evaluated as a possible explanation for a good number rather than as a decision that has to stand on its own. The feature list stops being a list of decisions and becomes the recipe that produced the result.

There is a reason to put the order in the report as well as in your practice. It tells the reader what the finding is not: a data selection finding reached before any number was computed cannot be a reaction to disappointing performance, and saying so removes the most common rebuttal before it is offered. It also sets an expectation for the next review, which is how a personal habit turns into a house standard.

Effective use depends on a clear understanding of the model's limitations, and using a model beyond its intended purpose introduces additional uncertainty and risk. When a use is extended beyond the original application, the sound practice is additional analysis of the new usage and its limitations, together with a review of the existing controls. Three obligations, all of them triggered by a change that leaves the model itself untouched.

This is why the purpose statement is the first rung rather than a preamble. It is the only artefact that makes "beyond intended purpose" a checkable claim instead of an opinion, and a model whose purpose statement says "improve credit decisioning" can never be used beyond its intended purpose, because the statement excludes nothing. A useful purpose statement names what is out of scope. Session 1 put the re-tier trigger in the inventory for the same reason, and the two artefacts are meant to be read together: the inventory notices the extension, the purpose statement says whether it is one.