The retrain-policy bench

Nine inputs: a constant, the New South Wales price and demand, the Victoria price and demand, the scheduled transfer between the two states, the time of day as a point on a circle, and a weekend flag. One linear score, fitted by ridge least squares; a score above zero means UP. There is no feature engineering, no tuning and no clever architecture, and there should not be. A retrain policy is a decision about a model you already have in production. If the model changed between runs you could not tell whether a better number came from the policy or from the model.

The first model learns from the first four weeks, 448 rows. Every run is then scored on the same 14,656 rows that follow, from June 1996 to December 1998. The floor for reference, always answering DOWN because DOWN is the more common label, scores 57.72 percent.

Accuracy is the share of the 14,656 rows the policy's model called right. It is measured: the same rows, the same labels, counted.

Retrain spend is the number of refits times a dollar figure you set, $500 by default. That $500 is not a fact about anybody's cloud bill; it stands for the whole cost of one retrain in production, which is rarely the compute. It is the validation run, the review, the change ticket, the canary from session 3, and the on-call risk of shipping a new model on a Friday. Set it to what one retrain costs your team and every spend figure on the page moves with it. The accuracy column never does.

Read them as a pair. A policy that gains a point and a half for $65,000 and a policy that gains four points for $3,000 are not "both better than never". One of them is a bargain and one of them is a habit.

The full Elec2 file on OpenML has 45,312 rows, one per half-hour from 7 May 1996 to 5 December 1998. The bench keeps every third row, in time order, which gives 15,104 rows one and a half hours apart: 16 a day, 112 a week. Each input, already scaled between 0 and 1 by OpenML, is rounded to one of 256 levels. Nothing is shuffled, so the drift is exactly where the market put it.

One thing the thinning changes, and it matters for Part 3. On the full file, a rule that simply repeats the previous half-hour's label scores 85.3 percent, a figure Žliobaitė published in 2013 and that this build reproduces at 85.33 percent. On the bench, the "previous row" is an hour and a half earlier, so the same rule scores 74.63 percent. The model runs are not flattered or penalised by the thinning in the same way, which is why the page quotes the bench's own no-model number and never the 85.3.

One more real quirk: the Victoria price, Victoria demand and transfer columns hold a single constant value for roughly the first year, because they were missing and were filled in before the file was published. They start moving in May 1997. The chart on the bench marks the date. A model trained before then has learned nothing about three of its nine inputs, which is a very ordinary way for production data to change under a model.

Never retrain costs nothing and scores 68.70 percent. Refitting on all history once a quarter, ten retrains, scores 72.44 percent: 3.75 points for $5,000, about $1,335 a point. The line under the chart shows where the gap opens. Select "Every week" and it reports a best four-week lead of +14.5 points over never retraining, in the four weeks from 19 Nov 1996, six months after launch. That slow slide is what drift looks like from the inside: no alert fired, no pipeline broke, and an uptime dashboard would have shown a healthy model. The same line keeps it honest in the other direction too. The weekly run's worst block, 58.5 percent from 10 Feb 1998, is as bad as never retraining's worst, and in the four weeks from 30 Jun 1998 never retraining beats it by 2.9 points. Retraining raises the average; it does not buy immunity from bad months.

Ten retrains a quarter apart: 72.44. Thirty-two, every four weeks: 73.11. One hundred and thirty, weekly: 73.55. Nine hundred and fifteen, daily: 73.55 again, to the row, 10,779 correct in both. Going from weekly to daily multiplies the retrain bill by seven and buys nothing. On the bench's printed rate, the quarterly policy pays about $1,335 per accuracy point over never, monthly about $3,624, weekly about $13,399.

The alarm policy sits on the knee of the curve: 6 retrains, 72.78 percent, about $735 a point. It refits only when four-week accuracy on labelled rows has fallen 10 points below where that model started. It is not the most accurate policy on the page. It is the cheapest accuracy on the page, which is usually the more useful sentence in a budget meeting.

"Weekly, recent only" retrains exactly as often as "weekly" and spends exactly the same $65,000. The one difference is the refit window: the last four weeks of labels instead of everything labelled so far. It scores 69.88 percent: +1.18 points over never retraining, against +4.85 for the all-history weekly run on the same schedule. On this stream, most of what retraining buys is more data, not newer data. That is a finding about this model on this market, not a law: a stream whose old regime is actively wrong, a fraud pattern that has been shut down, a product that no longer exists, would reward the recent window instead. Which is why the window is a dial on the bench and not a default in the policy.

The freshest model money can buy: refit before every one of the 14,656 predictions, on the last eight labelled rows, the last twelve hours. It retrains 14,655 times, $7,327,500 at the default rate, and scores 61.91 percent. That is 6.78 points below never retraining and 3.43 points below having no model at all.

The mechanism is arithmetic, not bad luck. Eight rows cannot pin down nine weights; the ridge penalty keeps the solve from blowing up, but what it learns is whatever the last twelve hours happened to look like. Every refit is a fresh overreaction. "Retrain as often as the pipeline allows" sounds like diligence and is, on a small window, a way of paying to replace a model with noise.

Keep the nuance honest. Move the label delay to 1.5 hours and the same policy scores 69.45 percent, slightly above never retraining, because a model refitted on the last twelve hours with labels one row old is partly just copying the last label. Žliobaitė showed in 2013 that on this dataset even a detector firing at random lifts accuracy for exactly that reason. A frantic policy can look good on a stream whose labels are sticky. That is a reason to compare every policy against the no-model rule, which is Part 3.

At a 1.5-hour delay the no-model rule scores 74.63 percent and beats every one of the seven model policies, the best of which, weekly on all history, reaches 73.51. Every dollar spent on training, serving and retraining that model buys a worse answer than a one-line rule. At a one-day delay the same rule falls to 65.35 percent and the weekly model beats it by 8.20 points.

The market did not change between those two runs. What changed is when the truth arrives, and that is decided by the business process around the model: how long until a loan defaults, a parcel is returned, a claim is settled, a price is published. Before a team argues about retrain schedules it should be able to answer, with a measured number, "how long after a prediction do we find out whether it was right?" Session 5 builds monitoring on that same number, because an alarm on accuracy can only look at rows whose labels have arrived.

The alarm watches accuracy on the last four weeks of labelled rows. With labels a week late it is always looking at a window that ended a week ago, and it still scores 72.61 percent on 6 retrains against 72.78 at a one-day delay. That robustness is a property of this stream, which drifts over months rather than hours, and it would not survive a stream that breaks overnight.

Alarm tuning is also less smooth than the headline suggests. On the bench, a 5-point drop threshold fires 8 times and scores 71.38 percent, below the 10-point alarm's 72.78 on 6. Fewer alarms did better because of when they fired, not how many. An alarm is a timing decision, and one good run of one threshold is not a tuning result. Try the thresholds yourself before quoting any of them.