When optimisation beats a model

A team asks for a better demand model. Pressed on what would change if the model improved, the answer is that the mix would be set differently. Pressed further, it turns out the mix has never been set by arithmetic at all, and that the ceilings it has to fit under have never been written down in one place. Improving the forecast in that situation moves a number that nothing downstream consumes.

The tell is simple and worth listening for: ask what decision the improved prediction would change, and how. If the answer is a quantity somebody sets, under limits, the work in front of you is allocation. The forecast may still need improving, but it is not the thing standing between the business and a better week.

• Prediction. Estimating a quantity you cannot yet observe. Output: an estimate, normally with uncertainty attached.
• Optimisation. Choosing among actions to make one number as large or as small as possible, subject to limits. Output: a plan.
• Allocation problem. A decision about how to split a scarce resource. Very often mistaken for a forecasting problem.
• Objective, again. The single number the optimiser maximises. Tonight it is the thing under examination rather than an assumption.
• Proxy objective. Something easier to write down standing in for what the business actually wants. Tonnage for margin, utilisation for value.
• Model risk, two shapes. A prediction that is wrong, which usually looks uncertain. An objective that is wrong, which looks optimal.
• Validation. Checking a model against outcomes it did not see. Straightforward for a forecast, and much harder for a plan the business has never run.
• Simplification. Deliberately leaving something out of the problem so that what remains can be trusted. The centre of tonight's unresolved argument.

Constraints are the hard part when the number of legal plans is enormous, when several limits interact, and when the answer is not obvious to an experienced person. Ravensworth qualifies on all three, which is why margin per tonne, a perfectly correct ranking, pointed at a product that turns out to be worth nothing to the week. No amount of forecast accuracy would have caught that, because the error was not about demand.

Uncertainty being mild is the other half, and it is the half people skip. Ravensworth's uncertainty is not mild, and session 4 measured exactly what that costs: the plan sells out only about a quarter of the time. Optimisation still earns its place here, but the honest version of the cue is that it earns a place alongside a distribution, not instead of one.

When the choice set is genuinely tiny, optimisation has nothing to do. Two options and no capacity interaction is not an allocation problem, it is a comparison, and the difficulty is entirely in estimating which side of the line you are on. Approve or decline, replace the part now or at the next service, treat this batch as suspect or not.

The signature is that the hard question is about the world rather than about you. If you would act correctly and immediately given a reliable number, you need the number. Building an optimiser around a two-way choice is machinery with nothing to grip.

A prediction can be checked against what happened. It arrives with an interval, it can be backtested against outcomes it never saw, and when it is unreliable there is usually machinery in place that says so. None of this makes forecasting easy, but it does mean the failure has a warning light on it.

A plan is harder. The business ran the plan, so the week that would have happened under a different plan is not observable, and the thing most likely to be wrong is the objective, which is not a statistical object and cannot be backtested at all. The plan arrives labelled optimal, with no interval and no error metric, and the label is accurate with respect to the objective it was given. That asymmetry is the subject of Part 2.

Nobody sets out to maximise the wrong thing. Proxies get written down because the real objective is hard to measure and the proxy is sitting in a system already. Tonnage is in the despatch report. Utilisation is on a dashboard. Protein consumption is in the stock file. Margin by product line, adjusted for what the relationship is worth over three years, is in nobody's system at all.

So the objective box gets filled with what is available, the optimiser does exactly as it is told, and the resulting plan is defended with the word optimal by people who are not wrong to use it. Step through the options in the box above and watch the plan move a long way while the report of it barely changes.

Not more computation, since the arithmetic was never in question. Three things help, and all three are organisational rather than technical. Write the objective in a sentence a person outside the analytics team can disagree with. Run the plan under two or three different objectives and look at how much the answer moves, because an answer that is stable under a small change of objective is a much safer answer than one that is not. And name, out loud, what the objective leaves out.

The second of those is worth pressing on. It is cheap, it takes one more solver run, and it converts a hidden assumption into a visible spread. If the plan barely moves, the objective was not doing much work. If it moves a long way, the objective is the decision, and it deserves the attention that was being spent on the model.

Everything above is agreed. What follows is not, and it is the real argument in this field rather than a staged one. Given that the objective is where the risk lives, do you optimise a simplified problem you can trust, or a richer problem you cannot validate?

The simplifiers hold that a small, honest model is the only kind whose answer means anything. Three products, three ceilings, one objective, every input measurable, and a plan whose logic a person can follow end to end. If something important is missing, and something always is, it is missing in the open and a human adds it back at the point of decision. Their charge against the other camp is that a model nobody can check is not a model, it is an opinion with arithmetic on it, and that the first time it produces a strange answer the organisation will either follow it off a cliff or stop believing it entirely.

The enrichers hold that the simplification is where the loss actually occurs. Leaving the Show relationship, the seasonality, the changeover time and the demand spread out of the problem does not make the plan safer, it makes the plan wrong about the business in ways the optimiser then maximises with great efficiency. A richer model is harder to validate, and being unvalidated is not the same as being wrong. Their charge against the simplifiers is that a model small enough to check is usually too small to be the thing the decision rests on, and that the humans supposedly adding the missing considerations back in are the very judgement the exercise was meant to improve.

Both camps have real failures on their record. Simplified models produce confident plans that ignore something everybody in the room knew. Rich models produce plans nobody can explain, which get overridden until the model is abandoned. There is no measurement that settles this, because the thing in dispute is how much you should trust what you cannot check. What can be said is that the question is worth asking explicitly at the start of a piece of work, in the room, rather than being answered by default by whoever builds the first version.