AI feature metrics

The event is suggestion_shown, one row per offer. arm says which version of the assistant made it, dev says who saw it. accepted is the moment the developer took it. kept_30s and kept_600s say whether the accepted code was still there, unchanged, 30 seconds and 10 minutes later; deleted_600s says it was gone by 10 minutes. chars is the size of the offer. Every later flag is only meaningful when accepted is true, which is why the metrics divide by accepted rows, not by shown ones.

In a real product the later flags arrive later, as their own events joined back to the suggestion by an id. Storing them as columns on one row, as tonight's stream does, is the shape you end up with after that join. Plan the suggestion id before the feature ships: without it, "kept at 600 s" cannot be tied to the offer it describes.

Ziegler et al. define acceptance rate as "the fraction of completions shown to the developer that are subsequently accepted for inclusion in the source file". In code that is accepted.sum() / shown, or in pandas the mean of a boolean column. For what happens after acceptance they measure persistence at four checkpoints, 30, 120, 300 and 600 seconds. Tonight's stream keeps two of them, 30 and 600, which is enough to see the difference between a quick check and a lasting one.

Their productivity outcome came from a survey built on the SPACE dimensions (they used S, P, C and E), matched to each respondent's usage: 17,420 users were emailed and 2,047 responses were matched to usage data, between 10 February and 6 March 2022.

Acceptance rate measures the moment of the offer: did the developer think it was worth taking. Persistence measures what the code turned out to be worth: of the accepted suggestions, how many were still there, unchanged, ten minutes later. "Long" is accepted more (0.309) and survives less (0.408 kept, 0.190 deleted by 600 seconds); "short" is accepted less (0.262) and survives more (0.618 kept, 0.078 deleted). Characters kept per shown multiplies survival by size, and a block that survives less often still leaves more text behind than a line that survives more often: 20.183 against 6.486.

So the question is never "which metric is right". It is "which decision are we making". Choosing a default for developers who hate cleaning up after the tool points at persistence. Estimating how much typing the tool saves points at characters kept. Neither table proves that switching versions would change anything for real users: that needs an experiment (see Learn Experimentation with Phoebe).

At 30 seconds "long" keeps 0.693 of its accepted suggestions and "short" keeps 0.800. At 600 seconds the figures are 0.408 and 0.618. The early check flatters both arms, and in this constructed stream it flatters the block more, because the script assumes a big suggestion is more often rewritten after the first glance. Pick the checkpoint before you look at the results, and report it with the number: "kept" without a time is not a definition.

The fix is one extra line of pandas: group by developer first, take each developer's acceptance rate, then describe that distribution with its 10th and 90th percentiles. Put the band next to the arm average in every review. It changes the conversation from "long is better" to "long shifts the whole band up a little, and the band is wide", which is the honest reading.

Two cautions. First, 200 developers per arm is a small group: percentiles from groups that size move noticeably with the seed, so treat the band's edges as rough. Second, the average of per-developer rates and the pooled rate are different numbers when developers see different amounts; a heavy user counts once in the first and many times in the second. Say which one you report.

Ziegler et al. report that acceptance also varies with time: 23.5 percent at weekends, 23 percent on weekdays outside working hours and 21.2 percent in working hours. So a change in the mix of when people use the tool can move the headline rate with no change in the tool at all. A launch that happens to land near a holiday, or a team that shifts its hours, is enough. Break acceptance down by time of use before reading a week-on-week movement as the feature getting better or worse.