learn-prescriptive-analytics-with-phoebe / Practitioner session 6 of 10
Learn Prescriptive Analytics with Phoebe · Practitioner session 6 of 10

Modelling choices that bite

Sessions 3, 4 and 5 were about machinery, and the machinery is the reliable part. What actually goes wrong in practice is the writing down: a row whose terms are in two different units, a limit everyone knows about that nobody wrote, a real cost left out of the objective, and a hard requirement expressed as something the plan may trade away. None of those four raises an error. Each produces a plan that is internally perfect and externally wrong, and the wrongness has a different shape in each case. Tonight is how to recognise all four, plus the one mechanism that most often goes wrong on purpose.

🟠 Working depth Anyone who writes the model Anyone who reviews one 45 min
0-3 · Setup 3-20 · Four faults 20-32 · Big-M 32-45 · Build-along and Q&A
Part 0

The solver is the part that works

A linear solver does exactly one thing: it finds the best point in the region you described. If the region you described is not the region you meant, it will find the best point in the wrong region, report it as optimal, and be entirely correct in doing so. There is no diagnostic for this, because from the inside a mis-specified model is indistinguishable from a specified one. The failure mode is not a crash and it is not a wrong number. It is a right number to a question nobody asked.

Live = presented in the 45 minutes Self-study = the full depth, after class 🔍 Every diagram zooms

★ What you walk out with tonight. Four formulation faults you can name and detect, each with the specific way it shows up in the real world: a silently wrong answer, a confident plan reality rejects, an objective the solver exploits, and a model nobody can use. Big-M explained as a mechanism rather than a formula, including why a value chosen too large and a value chosen too small fail in completely different ways. And a habit of reading somebody else's model that starts with the units and ends with what the objective forgot to price.

LiveThe one-paragraph version4 min▶

Four questions, asked of every model, in this order. Are all the terms in a row measured in the same thing? Is every limit that physically exists actually written down? Does the objective contain every cost that changes when the plan changes? And is everything in the constraint block genuinely impossible to cross, rather than merely undesirable? Each question catches a different fault, each fault produces a different symptom, and none of the four is detectable by looking at the solution alone. You have to look at the model.

Real world

Ravensworth's milling row is 1.1, 1.5, 2.3 against a ceiling of 131. Every one of those four numbers is in hours: hours per tonne on the left, hours available on the right. That is not an accident of the write-up, it is the condition that makes the row mean anything, and it is the first thing to check on a model you did not write.

Self-studyThe words tonight adds4 min▶
  • Formulation. The translation from a real situation into variables, an objective and rows. The part with no error messages.
  • Dimensional consistency. Every term in a row measured in the same unit, and the right hand side measured in that unit too.
  • Missing constraint. A limit that exists in the world and not in the model. It never causes infeasibility; it causes a plan that cannot be run.
  • Unpriced cost. Something the plan consumes that the objective does not charge for. The solver will consume as much of it as it can.
  • Soft constraint. A requirement written as a penalty in the objective, so the plan may pay to break it.
  • Hard constraint. A row the plan may not cross at any price. Writing one as the other is a modelling decision, not a detail.
  • Indicator variable. A variable restricted to 0 or 1, standing for a yes-or-no decision such as whether a line is set up at all.
  • Big-M. The constant that links an indicator to a quantity, as in quantity <= M * indicator. Its value is a choice, and both directions of getting it wrong are bad.
Part 1 · covers four faults and their symptoms

Four ways to be wrong without being told

22 min live

These four are worth learning as a set because their symptoms differ, and the symptom is usually the only thing you get to observe. One gives an answer that is quietly wrong. One gives an answer that is confidently unrunnable. One gives an answer that looks too good and is being gamed. One gives an answer nobody can act on even though it is correct. Knowing which symptom belongs to which fault is most of the diagnosis.

Four faults, and the different shape each one leaves in the world THE FAULT WHAT THE SOLVER DOES WHAT THE WORLD DOES 1 · MIXED UNITS IN ONE ROW 1.1 hours + 90 minutes <= 131 Both terms are numbers. Only one of them is in the row's unit. Nothing. It is ordinary arithmetic and it returns an optimum, on time, with a clean status and real prices. The plan overruns or underuses a resource by a fixed factor, forever, and the number looked plausible. The answer is wrong, not infeasible. 2 · A CONSTRAINT NOBODY WROTE The limit exists on the floor and not in the matrix. Everyone who works there already knows it. Explores a region larger than the real one, so the answer comes back better than the truth, which is the most persuasive form of wrong. Rejected on day one by somebody who says "we can't do that", and the model loses its credibility for a reason that is entirely fair. 3 · A COST THE OBJECTIVE FORGOT Changeovers, overtime, waste, a haulage charge. Real money that the objective does not charge for. Consumes as much of the free thing as the rows permit. Not maliciously, not by accident: maximising is literally the instruction it has. A plan that looks unusually good on paper and is worse than today's in practice. The gap is exactly the thing you did not price. 4 · A HARD RULE WRITTEN AS SOFT A requirement that may not be broken, priced in the objective as though it could be bought out. Pays the penalty whenever breaking the rule earns more than the penalty costs, which is the correct answer to the question as written. A plan that breaks a rule the mill is not allowed to break. Unusable rather than inaccurate, and a second run will not fix it. None of the four produces an error. All four produce the status optimal, and three of the four produce a better number than the truth.
🔍 Click to zoom - read the third column first. In practice you meet the symptom long before anybody shows you the model.
Live The real solver, given a damaged model. Every scenario below returns the status optimal.

...

...
LiveUnits: wrong rather than infeasible4 min▶

A constraint row is a sum of terms compared against a ceiling, and the comparison only means something if every term and the ceiling are measured in the same thing. Write one product's milling requirement in minutes while the rest of the row is in hours and the arithmetic still works perfectly, because arithmetic does not know what an hour is. The row now says something false about the mill, and the solver optimises faithfully against the false statement.

What makes this the nastiest of the four is that it does not push the model towards infeasibility, where you would notice. It rescales one column of one row, which tilts the trade-offs between products by a constant factor. The plan shifts, the objective shifts, everything still solves, and the only evidence is that the mix looks slightly odd to somebody who knows the plant. Write the unit next to every coefficient when you build the matrix, and check the ceiling's unit against the row's, every time.

LiveA missing constraint never errors4 min▶

Adding a limit can only shrink the feasible region, so leaving one out can only enlarge it. A larger region contains everything the real one does and more, so the optimum over it is at least as good as the true optimum, and usually strictly better. That is the whole mechanism, and it explains the characteristic experience: the model's first run produces a suspiciously attractive plan, and the attractiveness is the symptom.

The limits that go missing are almost never the ones on the capacity sheet. They are the ones so obvious to the people on the floor that nobody thinks to say them. A minimum batch size. The fact that two products cannot run on the same day. A customer who must get something every week whatever the margin says. The reliable way to surface them is to show a specific optimal plan to somebody who runs the operation and ask what is wrong with it, because people recognise a violation far faster than they recall a rule.

LiveThe solver will take everything you left free4 min▶

If something the plan consumes is not in the objective and not in a row, then as far as the model is concerned it is free and unlimited. The optimiser will use as much of it as helps. This gets described as the solver gaming the model, which gives it too much personality: it is doing precisely what it was told, and the omission is the instruction.

The practical test is to list everything that changes when the plan changes, then check each item against the objective and the rows. Anything on the list that appears in neither is being consumed for free. Then decide deliberately: price it in the objective, or bound it with a row, and note that those two choices produce different plans. Pricing lets the plan buy more of it when it pays; bounding forbids that trade entirely.

Self-studyHard and soft is a decision, not a technicality5 min▶

Writing a requirement as a row means the plan may never break it, at any price. Writing it as a penalty in the objective means the plan will break it precisely when breaking it earns more than the penalty. Both are legitimate, and which one is right is a question about the world rather than about modelling: is there a price at which the organisation would genuinely accept the violation?

A hard rule written soft yields plans that break rules, which is a different kind of failure from the other three. The model is not inaccurate, it is unusable, and the people who have to reject its output learn to reject its output. The reverse error is quieter and also expensive: a preference written hard forbids plans that would have been worth their cost, and because those plans never appear, nobody sees what was lost. The second error has no symptom at all, which is why the question deserves to be asked explicitly for every row rather than settled by whichever is easier to write.

Part 2 · covers big-M

The constant that links a switch to a quantity

16 min live

Some decisions are not quantities. Whether to set a line up at all, whether to open a shift, whether to run a product this week: these are yes or no, and they are written as a variable allowed to take only the values 0 and 1. The difficulty is connecting that switch to the quantities, because the model needs to say "if the line is not set up then its output must be zero", and a linear row cannot contain the word if. Big-M is the standard way of saying it anyway, and it is the most common place a formulation goes wrong on purpose.

One row, doing the job of the word "if" quantity <= M * switch with switch restricted to 0 or 1, and M a constant you choose switch = 0 right hand side is 0, so the quantity is forced to zero. The line is off. switch = 1 right hand side is M, so the quantity is free up to M. The line is on. Choosing M: the same true maximum, three gates over it M FAR TOO LARGE the gate Every plan is allowed, so nothing valid is lost. But the relaxation can set the switch to a small fraction and still run a real quantity, so the bound is weak, pruning stops working, and the search takes far longer than it should. M JUST ABOVE THE TRUE MAXIMUM the gate No valid plan is excluded and the relaxation is as tight as this formulation allows. This is the target, and finding it means working out the largest quantity that could ever be genuine, from the rows rather than from a round number. M TOO SMALL the gate FORBIDDEN, AND NOBODY IS TOLD Valid plans are cut off. The model returns the best of a smaller set, with an optimal status and no warning at all.
🔍 Click to zoom - the pale bar is the range of quantities that are genuinely achievable. Only the middle gate sits where it should.
LiveWhy too large and too small fail differently4 min▶

Too small is a correctness failure. The row forbids quantities that were always achievable, so the true optimum may not even be in the model's feasible region any more. The output is optimal for a problem you did not mean, and there is no signal: not an error, not an unusual status, not a suspicious number. This is the one to be afraid of.

Too large is a performance failure, and it works through the relaxation. In the relaxed problem the switch is not forced to 0 or 1, it can take any value in between. With a very large M, a switch value of a tiny fraction still permits a substantial quantity, so the relaxation can enjoy the output while paying almost none of the setup cost. That makes the relaxed value far better than any real plan, the bound becomes weak, branches that should be pruned survive, and the search inflates. The answer stays correct and the time to get it does not.

Which is worse depends on what you can tolerate, and it is worth saying plainly that the two are not on the same scale. A slow model announces itself. A quietly infeasible-for-no-reason model does not.

LiveHow to choose M, and what to measure4 min▶

Derive it, do not pick it. For each switched quantity, work out the largest value it could take if everything else were maximally favourable, using the rows you already have: what the capacity ceilings permit on their own, what any demand limit permits, whichever is smaller. That number is the tightest honest M, and deriving it from the constraint data means it stays correct if the data change. A round number typed because it looks big enough is the version that goes wrong in both directions at once, on some future dataset.

Then measure rather than assume, because the trade-off is empirical. Solve at the derived M and record three things: the objective, the value of the relaxation at the root, and the node count. Multiply M by ten and record them again. The objective should not move at all, and if it does, the smaller M was cutting off valid plans and you have just found a real defect. The relaxation should get worse and the node count should rise, and how steeply they do that on your problem is the thing worth knowing. Then try a deliberately undersized M and watch the objective fall with no other indication that anything is wrong, which is the failure you are learning to recognise.

The mill's own numbers carry no big-M measurements to quote, so treat the paragraph above as the experiment to run on your model rather than a result to repeat. What is not in doubt is the shape: the objective is the correctness check, the node count is the cost, and they respond to M in opposite directions.

Deriving a bound rather than typing one
// The largest tonnage of product j that any ceiling could ever permit,
// taken one row at a time. The tightest of them is the honest M.
function deriveM(A, b, j) {
  var m = Infinity;
  for (var i = 0; i < A.length; i++) {
    if (A[i][j] > 1e-9) m = Math.min(m, b[i] / A[i][j]);
  }
  return m;   // and if a demand ceiling exists, take the smaller of the two
}

// The check that matters, run twice and compared:
//   solve at M, record objective and node count
//   solve at 10 * M, record both again
//   the objective must not move. If it moves, the smaller M was cutting
//   off valid plans, and that is a defect rather than a tuning question.
Self-studyWhere serious people disagree5 min▶

Big-M is taught everywhere and disliked by many of the people who teach it, and the argument is about whether it should be used at all in serious models.

One camp says it is fine and the objection is academic. It is one row, it is readable by anyone, it survives being handed to a colleague, and with a bound derived from the constraint data rather than typed in, the weakness of the relaxation is modest. Models are maintained by people, and a formulation that a maintainer understands at a glance has a practical value that does not appear in any benchmark.

The other camp says the construction is a trap with a friendly face. The relaxation it produces is weak by design, the weakness grows with the size of the constant, and the failure when the constant is too small is silent and severe. There are alternative formulations of the same logic that are tighter, and on larger models the difference is not a few percent but the difference between finishing and not. On this view the readability argument is how people end up with a model that runs overnight.

Both camps derive M from the data rather than guessing it, and both would run the objective-must-not-move check. The disagreement is about what to do when the derived M is still large: tolerate a weak relaxation, or reformulate. That is a real fork, and which side you take usually depends on whether you have personally waited on one of these.

Part 3 · build-along

Read a model the way you would read a contract

18 min hands on

Six steps, mostly on paper, on Ravensworth and then on something of your own. The aim is a reading order you can apply to any model in about ten minutes, and that finds the faults that do not announce themselves.

Five questions, in this order 1 · UNITS Is every term in this row measured in the same thing as its ceiling? catches: an answer that is silently wrong 2 · COMPLETENESS What limit exists on the floor and is not anywhere in the matrix? catches: a plan reality rejects on the first day 3 · THE OBJECTIVE What changes when the plan changes and is charged for nowhere at all? catches: a plan that games what you left out 4 · HARDNESS For each row, is there a price at which this could honestly be crossed? catches: a rule broken, or a good plan forbidden 5 · ANY BIG-M Was M derived from the rows, and does the objective stay put at ten times M? catches: valid plans cut off in silence The order is not arbitrary. A unit error changes every number downstream of it, so checking prices or bounds before checking units means checking them against a model that does not mean what it says.
🔍 Click to zoom - five questions, ten minutes, and four of them are answered by reading rather than by solving.

Annotate every coefficient with its unit

Write Ravensworth's three rows out and put the unit beside all twelve numbers, ceilings included. Say out loud why 1.1 and 131 belong in the same row and 0.1 and 29 belong in a different one. It takes two minutes and it is the check that is almost never done.

Break a unit on purpose and watch nothing happen

Change one milling coefficient as though it had been entered in minutes rather than hours, solve, and read the status. Optimal. Then compare the plan and the margin with the correct ones. Note that the only way you knew was that you had the right answer already.

Find the constraint that is not in the matrix

The mill has three rows. Name two limits a real feed mill would have that are nowhere in them, and say for each whether it would make the plan better or worse than the truth. Then say which of the two you would have found by staring at the model, and which needed a person who works there.

List what the objective does not charge for

Total margin is the objective. Write down three things that change when the mix changes and that appear neither in the objective nor in a row. For each, decide whether you would price it or bound it, and say what the difference does to the resulting plan.

Sort your own rows into hard and soft

Take the model you brought from session 5. For every row, answer the price question: is there an amount of money at which your organisation would genuinely accept crossing it? Anything with a yes is soft and is currently written wrong. Anything with a no is hard, and check it is a row rather than a penalty.

Write the M you would use, and the check

Pick any yes-or-no decision from your own work, write the linking row, and derive M from your constraint data rather than typing a number. Then write the two-line test: solve at M and at ten times M, and assert the objective is unchanged. Bring both to session 7, where the inputs stop being certain.

The habit to keep. When a model produces a result that is better than you expected, treat that as the primary symptom rather than as good news. Three of tonight's four faults make the answer look better than the truth, because all three of them, in different ways, hand the solver something it should not have had. A plan that beats today's practice by a wide margin deserves the five questions before it deserves a meeting.

Check yourself

Three questions

One coefficient in the milling row is entered in minutes while the rest of the row is in hours. What happens?

Arithmetic does not know what an hour is. A mixed unit rescales one column of one row, which tilts the trade-offs between products rather than pushing the model towards infeasibility, so there is no signal at all. Writing the unit beside every coefficient is the only reliable defence.

A model's first run produces a plan worth considerably more than anything the operation has achieved. What is the most likely explanation to check first?

Adding a constraint can only shrink the feasible region, so leaving one out can only enlarge it, and the optimum over a larger region is at least as good. An unpriced cost has the same effect through the objective. An attractive first run is the characteristic symptom of both, which is why it is a reason to check rather than to celebrate.

In quantity <= M * switch, what is the difference between choosing M far too large and choosing it too small?

The two failures are not symmetric. Too large is a performance problem with a correct answer at the end of it, caused by the relaxation being able to run a real quantity while holding the switch at a fraction. Too small is a correctness problem with no symptom. Derive M from the constraint data, then check that multiplying it by ten leaves the objective unchanged.

Before session 7

Homework

Thirty minutes, mostly writing rather than coding.

Run the five questions on your own model

Take the model from session 5 and answer all five gates in writing, one short paragraph each. Count how many of the five you could answer without asking somebody else, and note which ones needed a conversation with whoever runs the thing.

Write down which inputs you are least sure of

List every number in your model and mark the three you trust least. Say for each whether being wrong by ten percent would change the plan or only the objective. Session 7 stops treating inputs as known and starts drawing them, and this list is what it works on.

Covered tonight

What this session covers

✓Dimensional consistencyEvery term in a row and its ceiling in the same unit, or the answer is wrong rather than infeasible.
✓The missing constraintNever errors. Enlarges the region, so the plan looks better than the truth.
✓The unpriced costAnything free and unbounded gets consumed, because maximising is the instruction.
✓Hard against softA price question about the world, and both directions of getting it wrong cost something.
✓Indicator variablesA 0 or 1 variable standing for a yes-or-no decision a quantity cannot express.
✓Big-M as a mechanismquantity at most M times switch, and what each side of the switch does to the row.
✓Both ways of choosing M badlyToo large is slow and correct. Too small is fast and wrong, with no warning.
✓The five-question reading orderUnits, completeness, the objective, hardness, then any big-M.
◐Deriving M from the rowsThe tightest honest bound comes from the ceilings, and the check is that ten times M moves nothing.
◐Tighter formulations of the same logicNamed as the alternative to big-M, and argued about honestly rather than settled.
○Measured big-M figures for the millRavensworth's frozen numbers carry none, so the mechanism is taught without inventing any.
○Automatic model validationTools that check a formulation are not covered. The five questions are done by a person.
○Uncertain inputsEverything tonight assumes the numbers are known. Session 7 draws them instead.
Take this with you

Cheat sheet

Units. Every term in a row and its ceiling in the same unit. Wrong, not infeasible, and no signal.
Missing constraint. Enlarges the region, so the answer improves. Reality rejects the plan on day one.
Unpriced cost. Free and unbounded gets consumed. Not gaming, just the instruction you gave.
Hard or soft. Is there a price at which you would cross it? Yes is soft, no is a row. Both errors cost.
Big-M. quantity <= M * switch. Switch 0 forces zero, switch 1 opens the gate to M.
M too large. Weak relaxation, poor bound, slow search. Correct answer, eventually.
M too small. Valid plans cut off, optimal status, no warning. The one to fear.
The check. Derive M from the rows, then solve at M and at ten times M. The objective must not move.