Build - a ladder, a bench, and an answer nobody wants
One script runs everything: the ladder on the real year, the ladder against planted truth, the cadence measurement, and the attribution. Every number below comes out of that run.
A. What the year actually looked like
15 series is 15 chances a day to be surprised, and a percent-change rule takes every one of them.
The March duplicate load inflated the month by $125,738 of GMV - 18 percent of the month as booked. It lands on all 31 days at a median 24 percent, so it is a sustained uniform shift rather than a spike, and the baseline simply learns it. Removing the duplicates moves March from 18 alerts to 18 on the robust rung, and 15 to 13 on the FDR rung. Anomaly detection is not data quality. This class of defect is caught by a reconciliation check on row counts and key uniqueness, and never by a threshold on a metric.
All money on this page is GMV. Everrest takes 11 percent, so the March overstatement is about $13,800 of platform revenue - a mart defect to reconcile, not a ledger entry.
B. Two fixes that cost one line each
Before any threshold is chosen, two things about the residual decide how the monitor will behave.
actual - expected is skewed +6.6 and trips a nominal 3-sigma threshold on 5.5% of series-days. log(actual / expected) is skewed +0.15 and trips it on 1.1% - 5.1x fewer false alarms, before any threshold is touched.
C. The ladder, and why its table cannot pick a winner
Seven rungs, each fixing one named defect in the one below. Scored against 240 planted changes across 20 replicates.
| rung | the defect it fixes | recall | precision | lift vs chance | lag (days) | false alarms/yr |
|---|---|---|---|---|---|---|
| D1 week-over-week 15% | what most teams ship | 98% | 7.5% | 0.99x | 0 | 4,121 |
| D2 3-sigma on level | no notion of variability | 48% | 7.4% | 0.98x | 10 | 145 |
| D3 3-sigma on residual | the level is seasonal | 23% | 10.2% | 1.35x | 14 | 33 |
| D4 robust 3-sigma | sd is dragged by the last incident | 23% | 8.2% | 1.08x | 14 | 58 |
| D5 calibrated null | 3-sigma is not a 0.27% rule | 6% | 9.6% | 1.27x | 21 | 10 |
| D6 calibrated + FDR | many series, many chances | 7% | 8.4% | 1.11x | 21 | 13 |
| D7 FDR + persistence | a one-day wobble is not an incident | 0% | - | - | - | 1 |
D7 is dead on this board - under two alerts a year, detecting nothing. It is reported rather than dropped, because a rung that is silent is an outcome and hiding it would flatter the ladder.
7.6 percent of live series-days sit inside a planted change, so a detector alerting at random scores 7.6 percent precision. Measured precision across the ladder runs 7.4 to 10.2 percent. The best lift over chance is 1.35x and two rungs score below chance. Precision in the single digits reads as weak but real until you put the base rate next to it - at which point no rung here has demonstrated meaningful discriminative power. That is the finding, not a caveat.
D. The same ladder at a matched alert budget
The table above compares seven rules at seven arbitrary thresholds, and recall moves with the threshold. Retuning every rung to the same false-alarm budget is the only comparison that isolates the idea from its operating point - and it changes the answer.
D2 on the raw level reaches 32 percent recall at k=5, D3 reaches 22 percent, and the calibrated rungs land at 17 and 16. The week-over-week rule cannot be tuned onto the budget at all within a sane range - at a 250 percent threshold it still fires 145 times a year, which is itself the finding: a rule with no usable operating point is not a strict rule, it is the wrong rule. Comparing detectors at their default thresholds measures the thresholds.
E. Why the whole ladder is fighting the wrong battle
Before tuning anything, compute how large your real incident is in units of one bucket's noise.
A genuine commercial emergency is worth less than a third of one day's noise on this board. No threshold separates that from an ordinary Tuesday, and lowering the threshold buys every Tuesday instead. At a fortnightly bucket the same drop is 1.02 sigma. Two real mechanisms contribute: the noise per bucket falls from 0.52 to 0.20 log units, and the trailing window stops eating the signal - 57 percent of a 45-day step has already been learned away at daily cadence. A monitor that stops alerting is not evidence the incident ended. The first monitoring decision is the bucket, not the threshold, and it is usually made by accident, by inheriting a daily dashboard.
End-to-end recall by cadence is deliberately not reported. One year of Everrest yields 26 fortnightly buckets against a 44-bucket warmup, so a fortnightly monitor on this history never becomes live at all - any recall comparison would measure how much history each cadence had. The signal-to-noise above is a property of the data and survives that objection; a recall number would not.
F. When an alert is real, say what moved
Detection needs a distributional assumption and so has to respect sparsity. Decomposition is exact arithmetic on sums and does not, so attribution runs at the fine grain the monitor refuses.
Beauty lost $29.04 of its own average order value, down 20 percent, contributing -$3.88 against a net rate effect of -$1.82. Five of eight categories raised their own average and it did not matter, because the three that fell carry 39 percent of orders. The action is to go and look at Beauty - which is not what a mix-shift headline on its own would have told anyone to do. A threshold on average order value would have fired on -$0.99 and said nothing about any of this.
Both decompositions reconcile exactly - segment effects sum to the observed total to floating point, asserted in code against the source totals rather than against the frame built from them - so no slice can be dropped to make an argument.
G. The artifact
A weekly digest whose every figure is computed, that shows the rows which actually crossed the threshold, and that states how often its own rule is wrong.