learn-prescriptive-analytics-with-phoebe / Practitioner session 7 of 10
Learn Prescriptive Analytics with Phoebe · Practitioner session 7 of 10

Monte Carlo over the plan

Four moves, in a loop. Hold the plan still, draw the uncertain inputs, re-evaluate what that draw would have earned, and keep the number. Ten thousand times, from a stated seed, and then read the collected ten thousand rather than their average. Tonight runs that loop on Ravensworth's own plan, reproduces the five-seed table figure by figure, and takes apart the one column on it that is easiest to describe backwards.

🟡 Foundations Anyone who will run the simulation Comfortable reading a little JavaScript 45 min
0-3 · Setup 3-18 · The loop 18-30 · Seeds and draws 30-45 · Build-along and Q&A
Part 0

Four moves, and the third one is where the money goes

The plan is already decided: 33 tonnes of Standard, 63 of Dairy, no Show, worth 4,656.00 with demand held at a single figure per product. Nothing tonight re-optimises it. The plan stays exactly where it is while demand is allowed to move, and the only thing being built is the machinery that measures what happens. That machinery is four moves long and you can write it in twenty lines.

Live = presented in the 45 minutes Self-study = the full depth, after class 🔍 Every diagram zooms

★ What you walk out with tonight. The loop written out and run in the page, reproducing the five-seed figures exactly rather than approximately. A precise reading of what the sells-out column counts, which is not what its plain-English name suggests and inverts if you describe it carelessly. And two pieces of reporting discipline that cost nothing and save arguments: the seed and the draw count travel with every simulated figure, and no result is believed until it has been seen at several seeds.

LiveWhat is fixed and what moves4 min▶

The single most common way to get this wrong is to let the plan move. If each draw re-solves the problem against that draw's demand, you are measuring something else entirely: a mill with perfect foresight, which chooses its production after seeing the week. That mill earns more than any real one and the figure it produces is not an estimate of anything Ravensworth can do.

So the loop has one fixed object and one moving one. The plan is fixed, because the mill commits on Monday. Demand moves, because it is not known on Monday. Everything else follows from that order of events, and the order of events is the modelling assumption that matters most here.

Real world

Ravensworth's demand is not a mystery: Standard runs around 33 tonnes with a spread of about 6, Dairy around 63 with a spread of about 11. Those two spreads existed before the plan was computed. They were not in the problem statement, because the statement had one box per product and one number fits in a box.

Self-studyThe words, and where each one is worked4 min▶
  • Draw. One sampled value of every uncertain input, and the one evaluation that follows it. Ten thousand of them per run here.
  • Seed. The integer that starts the pseudo-random generator. Same seed, same ten thousand draws, on any machine.
  • Realised profit. What a draw actually earned, computed from what was sold rather than from what was made.
  • Sells out. A draw where every product's plan was taken in full by demand. Part 2 pulls the definition apart, because the plain reading of the phrase is the opposite condition.
  • 5th percentile. The value one draw in twenty falls below. Read off the sorted array at index floor(draws * 0.05).
  • Median. The middle draw. Not the mean, and at Ravensworth it sits above the mean.
  • Run-to-run spread. How far the answer moves when only the seed changes. The quantity that decides whether a draw count is enough.
  • Posterior sampling. A different technique with a similar name, belonging to the Bayesian course. Nothing here walks a posterior; here the inputs are drawn and a fixed plan is re-evaluated.
Part 1 · covers the loop

Fix, draw, re-evaluate, collect

15 min live

Here is the whole mechanism on one page. Three of the four moves are bookkeeping. The third move, where a draw's demand meets a fixed quantity of product, is the only place anything interesting happens, and it is the reason the collected average lands below the figure the optimiser quoted.

The loop: one thing held still, one thing allowed to move 1 · FIX THE PLAN 33 Standard 63 Dairy 0 Show Unchanged for every draw, because the mill commits first. 2 · DRAW THE INPUTS Standard ~ 33, sd 6 Dairy ~ 63, sd 11 Negative draws clipped to zero. Generated from a stated seed, so repeatable. 3 · RE-EVALUATE sold = min(plan, demand) profit = margin x sold The only move that is not bookkeeping. Profit comes from sold, never from made. 4 · COLLECT AND SORT keep every profit sort, then index Ten thousand numbers, not one. The sort is what makes a percentile readable. Move 3, drawn: one product, one draw sold = demand flat: the mill made only 33 plan = 33 0 demand drawn tonnes sold Move 4, drawn: the ten thousand, sorted WORST BEST 5TH PERCENTILE MEDIAN One draw in twenty falls to the left of the first mark. The mean is not a mark on this bar. It summarises all of it. The flat half of that line is the whole mechanism. A quantity committed before demand is known is capped on the upside and open on the downside. Light weeks are paid for in full; heavy weeks pay nothing back, because the extra tonnes were never made.
🔍 Click to zoom - move 3 is one line of code and it is the only line that bends the answer.
The loop, in the page's own solver 
var c    = [38, 54, 71];        // margin a tonne: Standard, Dairy, Show
var plan = [33, 63, 0];         // fixed before any draw is taken
var mean = [33, 63, 0];         // demand centre, per product
var sd   = [6, 11, 0];          // demand spread, per product

var r = OPT.simulatePlan(plan, mean, sd, c, 10000, 7);
//                                          draws ^    ^ seed

r.draws;       // 10000, echoed back so it travels with the figure
r.sellsOut;    // fraction of draws where plan[j] <= demand[j] for EVERY j
r.meanProfit;  // average realised profit over the draws
r.p05;         // the sorted array at floor(draws * 0.05)
r.p50;         // the median draw
r.p95;         // the sorted array at floor(draws * 0.95)

// Inside, per draw, per product, the only line that matters:
//   sold = Math.min(plan[j], demand[j]);
//   profit += c[j] * sold;
LiveWhy profit is computed from sold, not from made4 min▶

Swap sold for plan[j] in that one line and the simulation reports 4,656.00 on every single draw, with a spread of exactly zero. It will not look like a bug. It will look like a reassuring result, and somebody will put it in a pack as evidence that the plan is robust.

The reason it is wrong is physical rather than mathematical. Tonnes that nobody ordered sit in the yard; they consumed milling hours and protein, and they earned nothing this week. The margin figures of 38, 54 and 71 are margins on a tonne sold. Applying them to a tonne made is the arithmetic of a business with a guaranteed buyer, and if the mill had one, there would be no uncertainty to simulate.

LiveWhere the draws come from, and why that is not a detail4 min▶

The generator here is a small deterministic function of the seed, and each demand value is a normal draw clipped at zero. Both of those are choices with consequences. The clipping matters when a product's spread is large relative to its mean, because it quietly shifts the average of the drawn demand upward. At Ravensworth, with a mean of 33 and a spread of 6, it almost never fires; with a mean of 4 and a spread of 6 it would fire constantly and the simulation would be describing a distribution nobody chose.

The normal shape is also a choice, not a fact about feed demand. Real demand is often right-skewed, lumpy where one large order dominates, or correlated across products when a single customer buys two of them. Each of those would change the answer, and none of them is harder to simulate than what is here. The honest move is to say which shape you assumed, next to the result, in the same sentence as the seed.

Self-studyWhy the median sits above the mean4 min▶

At Ravensworth the median draw at seed 7 is 4,440.26 while the mean across five seeds runs 4,318 to 4,331, so the middle week comes in better than the average week. That ordering is not a curiosity, it is the same asymmetry read from a different angle.

The upside is capped by the plan, so the right-hand side of the distribution piles up against a ceiling: a week with wild demand earns the same as a week with comfortably sufficient demand. The downside is not capped, so the left-hand side has a long tail of weeks where a lot went unsold. A long left tail drags the mean below the median, and the size of that drag is one honest measure of how exposed the plan is.

The practical consequence is that reporting the mean alone understates the typical week and overstates the bad one at the same time. It is the wrong single number in two directions at once, which is a good reason to stop looking for a single number.

Part 2 · covers the definition and the discipline

What the sells-out column counts, and what has to travel with a figure

12 min live

About a quarter, in all five runs. Before that number can be said out loud it has to be pinned to a condition, because the obvious plain-English reading of it is the opposite condition and produces a headline that is exactly inverted. This is not a pedantic point. It is a definition that has already been written down backwards once.

One condition is counted. The other one is what the name sounds like. WHAT THE SIMULATION COUNTS AS A SUCCESS plan[j] <= demand[j] for EVERY product j Demand took every tonne the plan made. Nothing was left unsold. Some demand may have gone unserved, and usually did, which is not penalised here. This is the column labelled sells out. THE OPPOSITE CONDITION, WHICH IS NOT COUNTED demand[j] <= plan[j] for EVERY product j The plan met demand in full. Every order was served, and whatever demand did not want stayed in the yard earning nothing. Calling the column this inverts the finding. Four hand-written draws, to show what the test does with them DRAW STANDARD, PLAN 33 OF 0 TO 50 DAIRY, PLAN 63 OF 0 TO 100 THE TEST 1 37 69 SELLS OUT both plans taken in full, 4 tonnes unserved 2 28 69 NOT A SELL Standard fell below its plan, 5 tonnes unsold 3 30 58 NOT A SELL both fell short, both left tonnes in the yard 4 33 63 SELLS OUT exact equality passes, because the test allows it Both conditions hold at once only when demand lands exactly on the plan. Everywhere else they disagree, so a column described as the wrong one turns a quarter into the other three quarters.
🔍 Click to zoom - the dark marks are draws at or above the plan, the amber marks are draws that fell short of it.
LiveRead the condition off the code, not off the column name4 min▶

The generic defence is to find the line that sets the flag and read it. Here that line is if (plan[j] > d + 1e-9) allMet = false; inside the per-product loop, and the counter increments only if the flag survives every product. Written out, a draw counts when no product's plan exceeded its demand, which is exactly the left-hand condition in the diagram.

Notice that the internal variable is called allMet, which reads like "all demand was met". It is not doing that, and a name that leans the wrong way is precisely how the mislabel happens. The name in the code is not evidence about the condition; the comparison is.

The wider habit: any boolean summarised as a percentage deserves the same treatment once. Write the condition as an inequality on paper, then write its negation, then say which of the two the headline is describing. The step takes a minute and it is the difference between "the plan works a quarter of the time" and "the plan works three quarters of the time".

Two pieces of metadata, and the table that exists because of the second one mean profit about 4,318 no seed, no draw count: nobody can re-run this or check it mean profit about 4,318 over 10,000 draws better: the precision is now justified, the exact figure still is not mean 4,317.94 | plan 33/63/0 | 10,000 draws | seed 7 re-runnable by anyone, to the penny, on any machine The same plan at five seeds, 10,000 draws each: Standard mean 33 sd 6, Dairy mean 63 sd 11 SEED SELLS OUT MEAN PROFIT 5TH PERCENTILE MEDIAN 7 25.1% 4,317.94 3,539.75 4,440.26 101 25.6% 4,327.93 3,571.12 4,451.73 2024 24.7% 4,330.55 3,590.76 4,446.08 55555 24.8% 4,328.35 3,559.39 4,448.79 987654 24.4% 4,323.93 3,562.75 4,448.95 ACROSS FIVE about a quarter 4,318 to 4,331 3,540 to 3,591 The table is five rows long so the run-to-run spread can be read next to the effect being measured.
🔍 Click to zoom - the amber rows are figures that cannot be checked by anyone else. The indigo one can.
Live The real simulation, running in this page, at all five seeds at once. Drop the draw count and watch the five answers come apart.

...

...
LiveHow many draws is enough4 min▶

There is no answer in the abstract, and the box above turns the question into a measurement. Set the draw count to 100 and read the five sells-out figures. They disagree by enough that any one of them, reported alone, would be a different story. Set it to 10,000 and they agree to a fraction of a percentage point.

That is the whole method: enough draws is when the run-to-run spread is small compared with the difference you intend to claim. If you are arguing that a plan loses three hundred pounds a week, a run-to-run spread of a few pounds is plenty. If you are arguing that one plan beats another by four pounds, it is nowhere near enough, and the honest response is more draws rather than a more confident sentence.

LiveFive seeds, not an average of five seeds3 min▶

A tempting shortcut is to run five seeds and report their average as a better single number. It is very slightly better and it throws away the thing the five runs were for. The point of the repeat is not extra precision on the mean, it is the visible spread, because the spread is what tells a reader whether the finding survives a different draw.

Report the range. "Mean profit 4,318 to 4,331 across five seeds of 10,000 draws" carries both the answer and its stability in one line, and it is no longer than the number it replaces.

Self-studyWhere serious people disagree5 min▶

Nobody disagrees about the arithmetic. There is a real argument about how much of the uncertainty is worth modelling at all.

One camp holds that a two-parameter demand distribution is already more structure than the data supports. Where did a spread of 6 come from, how many weeks of history is it based on, and is it stable across a season. On this view the simulation dresses a guess in ten thousand draws and the precision of the output is entirely fictional, so the honest thing is a handful of explicit scenarios the business can argue about: a bad week, a normal week, a strong week, written down and named.

The other camp holds that a distribution is the more honest object precisely because it is checkable. A scenario set has hidden probabilities, and they are always someone's unstated judgement about how often a bad week happens. A fitted spread can be compared with the last two years of orders and found wrong, publicly. On this view the scenario approach has the same guess in it, just in a place nobody can audit.

Both are arguing about where the guesswork should sit, not about whether there is any. The thing to notice is that the two approaches fail differently: a badly fitted distribution produces confident percentiles, and a scenario set produces a plan tuned to three weeks that somebody invented.

Part 3 · build-along

Run it, break it, then write the figure down properly

15 min hands on

Six steps, using the live box above and a page console. The aim is to leave able to write the loop from memory, and unable to report a simulated number without its seed and its draw count beside it.

Reproduce the table to the penny

Leave the plan at 33 and 63 with 10,000 draws. Check all five rows against the five-seed diagram, digit for digit. If a figure differs, you have found either a browser difference or a typo, and both are worth knowing before you trust anything else on the page.

Watch a hundred draws disagree with itself

Drop the draw count to 100 and read the five sells-out figures. Write down the highest and the lowest. Then say what would have happened if the first seed you tried had been the only one you ran.

Write the condition out, then negate it

On paper, in symbols: the condition the sells-out column counts, and its negation. Then write the sentence you would put in a report for each one, and check that they describe different quarters of the ten thousand.

Break move three on purpose

In the console, call OPT.simulatePlan with a plan whose quantities are far below the demand means, say 5 and 5. The sells-out figure goes to essentially everything and the mean profit collapses. Say in one sentence why both of those happened together, and what that proves about reading the sells-out column on its own.

Find the widest gap between median and mean

Hunt for a plan where the median draw sits furthest above the mean draw. Note what that plan has in common: it is the one exposed to the longest left tail. Write down what the gap between those two numbers is telling a reader that neither number tells them alone.

Write one figure the way it should be written

Take any result from tonight and write it as one line containing the statistic, the plan, the draw count and the seed, plus the range across five seeds. Keep the line. Session 8 asks what you would do about the shape once you can see it.

The habit to keep. A simulated figure without its seed and its draw count is not a result, it is an anecdote, and a result seen at one seed has not been checked. Both cost one extra clause in a sentence. The third habit is the one people skip: read the comparison that sets any percentage flag before you describe the percentage in words.

Check yourself

Three questions

In the simulation loop, why must the plan stay fixed across all ten thousand draws?

The order of events is the modelling assumption. The plan is decided first and demand arrives after, so the simulation has to preserve that order. Letting the plan react to each draw answers a different and much easier question.

The sells-out column reads about 25%. What condition is being counted?

The test is plan[j] <= demand[j] for every j. Option A is the opposite inequality and would describe the other three quarters of the draws, which is exactly how the figure gets inverted in a write-up.

A colleague reports "mean profit 4,318 from the simulation". What is missing, and why does it matter?

Both pieces of metadata are one clause each. The draw count justifies the precision claimed, the seed makes the exact figure reproducible, and running several seeds is what separates a finding from one lucky set of draws.

Before session 8

Homework

Thirty minutes, half of it coding and half of it re-reading somebody else's numbers.

Write the loop yourself

From scratch, in any language, without looking at simulatePlan. Fixed plan, drawn demand, sold as the smaller of the two, profit accumulated from sold, results sorted, percentiles read by index. Then run it on Ravensworth and see how close you get to the five-seed table. Bring whatever did not match.

Audit one percentage you did not compute

Find a hit rate, success rate or coverage figure in something your team already publishes. Track down the comparison that sets its flag and write the condition out as an inequality. Say whether the words around it describe that condition or its negation. Session 8 turns tonight's measured shape into a question about what to plan for.

Covered tonight

What this session covers

✓The simulation loopFix the plan, draw the inputs, re-evaluate from sold, collect and sort.
✓Why profit comes from soldmin(plan, demand) is the one line that bends the answer, and why it is right.
✓Reading percentiles by index5th percentile, median and 95th off the sorted array of draws.
✓The five-seed table, reproducedMean 4,318 to 4,331, 5th percentile 3,540 to 3,591, against a promise of 4,656.
✓What sells out countsplan[j] <= demand[j] for every j: nothing unsold, which is not meeting demand in full.
✓Seed and draw count as metadataBoth travel with the figure, and the result is checked at several seeds.
✓Choosing a draw countMeasured live: enough is when run-to-run spread is small against the claim.
◐Acting on the distributionTrading a worse mean for a better 5th percentile is outlined in session 8.
◐Other demand shapesSkew, lumpiness and correlation named tonight as choices, not simulated here.
○Building or judging the demand forecastModel choice and evaluation belong to the prediction courses.
○Sampling a posteriorSimilar name, different technique. MCMC lives in the Bayesian course.
○Whether the solver itself is rightSession 9 cross-checks it and finds two faults.
Take this with you

Cheat sheet

Four moves. Fix the plan, draw the inputs, re-evaluate, collect and sort. Only the third one bends the answer.
The plan does not move. Re-solving per draw measures perfect foresight, not the mill.
Profit from sold. sold = min(plan, demand), and margin applies to tonnes sold.
The measured result. Mean 4,318 to 4,331 against a promise of 4,656, 5th percentile 3,540 to 3,591.
Sells out means nothing unsold. plan[j] <= demand[j] for every j, which is the opposite of meeting demand in full.
Median above mean. A capped upside and an open downside drag the average below the middle week.
Two clauses, always. The seed and the draw count travel with every simulated figure.
Enough draws. When the run-to-run spread is small compared with the difference you intend to claim.