Where we start
Most AI feature reviews open with one number: the acceptance rate. It is easy to compute, it moves when the model changes, and the vendor dashboards put it first. It is also a measurement of a click, not of value. The best evidence available comes from GitHub's own researchers, who asked 2,047 developers how productive they felt and matched the answers to their usage data. Their headline is careful and often misread: acceptance rate tracks how productive people feel, not how much of the suggested code survives. Everything tonight hangs on holding those two things apart. One honesty note before we begin: no real AI-feature event stream is public, so the worked comparison in Part 3 runs on a constructed stream and says so wherever it appears.
★ What you walk out with tonight. A one-page AI feature metric sheet for a feature you own or are about to ship: the event chain it needs written into the tracking plan, the single decision the metric has to serve, one primary metric and one guardrail chosen for that decision, and a line naming how the primary metric could be gamed. Plus the sentence to say when someone puts the acceptance rate on a slide as if it were value.
Shown, accepted, kept, edited, deleted
5 min liveAn ordinary feature has one moment worth writing down: it was used. An AI feature has a chain. The suggestion is shown; the user accepts it or does not; and if accepted, the code, the sentence or the reply then lives on in the document where it can be kept unchanged, edited or deleted. The study we lean on tonight checked that last question at four clocks: 30, 120, 300 and 600 seconds after acceptance. Every honest AI feature metric is a ratio between two links of this chain, so the chain has to be in the tracking plan before any metric can exist.
LiveThe five events, in Object Action form3 min▶
Suggestion Shown is the denominator of every metric that follows, so it needs a stable id for the suggestion, the length of what was proposed, and the moment it appeared. Suggestion Accepted and Suggestion Dismissed are the user's answer, and Dismissed includes the quiet case where the person simply kept typing. Ziegler et al. define acceptance rate as "the fraction of completions shown to the developer that are subsequently accepted for inclusion in the source file", which is exactly the second box divided by the first.
The last link is the one most tracking plans forget: Suggestion Persisted, checked at a fixed clock after acceptance, recording how much of the accepted text is still there. The study used 30, 120, 300 and 600 seconds. Without that event, "accepted" is the end of the story, and the story then cannot tell a suggestion that shipped from one that was deleted before the developer's next coffee. Kept, edited and deleted are three values of one property on that event, not three events, following the no-variables rule from session 1.
Self-studyWhy a clock has to be in the definition2 min▶
"Kept" without a time is not a metric, because every accepted suggestion is kept at the instant it is accepted and almost none is kept forever. The clock decides what the number means. Thirty seconds after acceptance mostly measures whether the person immediately regretted it. Ten minutes measures whether it survived the first round of real work around it. Two teams comparing "kept rates" at different clocks are not comparing anything, so the clock is written into the tracking plan as a property and printed on the chart. The same discipline applied to "active" in session 3: the window is part of the definition.
The vendor says it too. GitHub's own metrics documentation describes acceptance rate as a signal that "measures how often developers accept Copilot's suggestions", and then adds the line most dashboards leave off: "Look for patterns across these signals rather than focusing on any single number". When the company that ships the feature warns you not to run on one number, the burden is on anyone who does.
Acceptance tracks how productive people feel
6 min liveIn 2022 eight GitHub researchers emailed 17,420 Copilot users and matched 2,047 survey responses to those users' real usage data, collected between 10 February and 6 March 2022. They then asked which usage metric best lined up with how productive people said they were. The answer was acceptance rate, at a rank correlation of 0.24. Every persistence measure did worse. That result is real, published and useful, and it is about perception. Read the abstract's own wording once, slowly.
LiveThe abstract, and the two words that matter3 min▶
The abstract states the result this way: "the rate with which shown suggestions are accepted, rather than more specific metrics regarding the persistence of completions in the code over time, drives developers' perception of productivity". The two words are perception of. The outcome variable in the study is a survey answer about feeling productive, scored across the SPACE framework's dimensions of satisfaction, performance, communication and efficiency. It is not lines shipped, not bugs, not review time, not revenue. The study is explicit about this and its authors were careful; the slides that quote it usually are not.
In section 4 they report acceptance rate correlating with aggregate perceived productivity at rho = 0.24, P < 0.0001, then add in the same breath that "there is still notable unexplained variance". A correlation of 0.24 is real and it is small: it means acceptance rate is a legitimate early signal of whether people like the feature, and a poor stand-in for anything else. The persistence measures, which come closest to "did the code survive", were all less well correlated with the survey. That is the part that surprises people, and it is the part to remember: the metric closest to value is the one furthest from how the feature feels.
LiveWhy acceptance is not value, in one sentence2 min▶
Accepting a suggestion costs a keystroke and commits nobody to anything; it is the cheapest possible yes. A person can accept because reading the proposal is slower than accepting and then fixing it, or because the suggestion is short and harmless, or because accepting is the path of least resistance in the interface. All of those raise acceptance rate and none of them is the feature doing the work. The sentence to keep for the product review is: acceptance measures how often the yes was cheap enough; persistence measures whether the yes was right. Both are correlations with an outcome nobody has measured directly, and neither proves the feature caused anything; the experimentation course owns that question: learn-experimentation-with-phoebe.
Self-studyWhat else the study measured, and did not2 min▶
Two more numbers from the paper belong in a leader's pocket. First, the acceptance rate in the study population was 27 percent, and the authors note by comparison that IntelliCode Compose reported a click-through rate of 10 percent, so "normal" for this metric depends heavily on the product and on what counts as shown. Second, acceptance moved with the clock on the wall: 23.5 percent at weekends, 23 percent in weekday non-working hours, 21.2 percent during working hours. If the same people accept more when they are relaxed, a week-on-week rise in acceptance rate may be a calendar effect before it is a model effect.
What the study did not do matters as much. It did not measure code quality, time to complete a task, or defects. It surveyed a self-selected 2,047 of 17,420 users, which is the usual limitation of any survey matched to usage: the people who answer are not the people who left. Treat every number above as a description of one product's users in one month of 2022, not as a law about AI features.
Eight researchers, one careful sentence. Ziegler, Kalliamvakou, Simister, Sittampalam, Li, Rice, Rifkin and Aftandilian could have titled their paper around "acceptance rate predicts productivity". They wrote "Productivity Assessment of Neural Code Completion" and put "perception" in the abstract. Between the paper and the slide deck that cites it, the word perception is what tends to go missing. The habit to build is to read the abstract of anything quoted at you in a review, and to look for the word the slide dropped.
Three metrics, two winners
4 min liveThe metric has to be chosen for the decision, and the fastest way to feel why is to watch one decision flip depending on which metric judges it. The comparison below is constructed: a seeded stream of 200 developers per arm over one week, one arm receiving short single-line suggestions and the other long block suggestions, with every rate an assumption written into the generating script. It is not a measurement of any real product. It is built so that the three most common AI metrics disagree, because in practice they do.
LiveRead the table the way the decision needs it3 min▶
The constructed stream has 119,905 shown suggestions in total, 60,023 in the long arm and 59,882 in the short arm. Long wins on acceptance, 0.309 to 0.262, and wins by a wide margin on characters kept per suggestion shown, 20.183 to 6.486, because a block carries far more text than a line even after the deletions. Short wins on persistence: 0.618 of its accepted suggestions are still unchanged at 600 seconds against 0.408 for long, and only 0.078 of short acceptances have been deleted by then against 0.190 for long. Read across a row and each metric is correct about what it counts. Read down the winners column and the "which arm to ship" decision flips twice.
| Metric | Short (line) | Long (block) | Favours |
|---|---|---|---|
| Suggestions shown | 59,882 | 60,023 | - |
| Acceptance rate | 0.262 | 0.309 | long |
| Kept unchanged at 30 s, of accepted | 0.800 | 0.693 | short |
| Kept unchanged at 600 s, of accepted | 0.618 | 0.408 | short |
| Deleted by 600 s, of accepted | 0.078 | 0.190 | short |
| Characters kept at 600 s, per shown | 6.486 | 20.183 | long |
So which arm is better? Wrong question. If the decision is "does the feature earn its place on the screen", characters kept per shown is the closest thing to a value metric on the table and long wins. If the decision is "are we teaching developers to accept things they then throw away", persistence and the deletion share are the metrics and short wins. If the decision is "do users like it", acceptance is the perception signal Ziegler found and long wins, modestly. Name the decision, then read one column.
Self-studyAverages hide the spread2 min▶
Even in a constructed stream, developers differ more than arms do. The 10th to 90th percentile of per-developer acceptance runs 0.177 to 0.467 in the long arm and 0.127 to 0.390 in the short arm, so the two ranges overlap across most of their width. An arm-level average of 0.309 versus 0.262 is a small gap sitting on top of a wide spread, and in a real product the spread would also carry seniority, language, task type and time of day. When an AI metric is reported for a team of fewer than a few hundred people, ask for the spread before believing the average; a group of 200 is small enough for a handful of heavy users to move it.
A common pattern, no numbers attached. A team ships longer suggestions because acceptance rose in the release notes, and two months later engineers complain the assistant "writes stuff we delete". Nobody lied: the team optimised the metric it was shown. The fix is the sheet you write in the build-along: a primary metric chosen for the decision, and a guardrail that would have caught the deletions before they became a complaint.
The AI feature metric sheet
★ 22 min · everyone buildsOne page, five rows, for an AI feature you own or are about to ship. Pick a real one: a draft reply, a summary, an autocomplete, a suggested tag. The example row below is a constructed "Draft Reply" feature in a support tool, named so it does not collide with any real product; your rows will differ. The five rows are the same for every AI feature: its event chain, the one decision the metric serves, the primary metric, the guardrail, and what would game it.
Draw the chain for your feature, as events. Shown, accepted or dismissed, persisted at a clock. Write the clock down: for code it was 30 to 600 seconds in the study, for an email draft it might be "at send". If your product does not record the persisted event, write "not tracked" in the row and keep going; the gap is the finding.
Write the one decision the metric has to serve. Not "is the feature good". A decision has two branches and a date: keep it on by default or make it opt-in; ship the long variant or the short one; renew the vendor contract or not. If you cannot name the decision, you are about to choose a metric for a slide, and the slide will choose acceptance rate.
Choose the primary metric for that decision. Perception decision, acceptance rate. Value decision, something kept, per shown, at a stated clock. Quality-of-habit decision, persistence or the deletion share. Say in one line why this metric answers this decision and the others do not.
Choose a guardrail that moves the other way when the primary is gamed. If the primary rewards volume, the guardrail watches waste: the share of accepted suggestions deleted by the clock. If the primary rewards persistence, the guardrail watches reach: the share of users who ever accept. A guardrail is a metric you promise to look at even when the primary is up.
Write what would game it. One honest line. Every AI metric can be lifted by changing what gets shown rather than making the feature better: show fewer, safer suggestions and acceptance rises; show longer ones and characters rise. Naming the cheapest way to move the number is what makes the number defensible in session 6.
The denominator is where value hides. "Kept, of accepted" tells you about the quality of the yeses. "Kept, of shown" tells you what the whole feature delivered per interruption, including every suggestion nobody wanted. They can move in opposite directions, and in the constructed stream they do. When two teams quote a "kept rate", the first question is which denominator.
Before session 6
30 min- Find the AI metric your organisation already quotes, in a deck, a vendor dashboard or a renewal case. Write down its numerator, its denominator and its clock. If any of the three is missing, the metric is not yet defined, and that is the first line of your review.
- Check whether the persisted event exists. Ask the analyst or the vendor whether anything records what happened to an accepted suggestion afterwards. "No" is a common and honest answer; write it in the blank-pages tab of the tracking plan from session 1.
- Read one abstract. Take the study or benchmark most often cited for your AI feature, read only its abstract, and underline the outcome it actually measured. Compare the underlined words with how the result is quoted internally.
- Bring the metric sheet. Session 6 puts your product's numbers, including this one, through the review question set: definition, cohort, denominator, sampling, bots, cause. The AI row is usually where the denominator question bites first.
Three questions
Clear all three and leader session 5 gets stamped in your passport.
1 · What did Ziegler et al. (2022) actually find about acceptance rate?
The outcome in the study was a survey of how productive people felt, matched to usage data for 2,047 respondents. Acceptance rate lined up with that feeling best, modestly, and the authors said so in the abstract with the word "perception". Nothing in the study measured code quality or shipped output, and 27 percent is simply what the acceptance rate was in that population.
2 · Why is acceptance rate not a measure of the value an AI feature delivers?
Acceptance is a ratio of the first two links of the chain and never looks at the last. In the constructed stream the arm with the higher acceptance also had the higher deletion share, 0.190 against 0.078. The metric is right about what it counts; it just does not count value.
3 · Your decision is "ship long block suggestions or short line suggestions". In the constructed comparison, acceptance favours long (0.309 vs 0.262), persistence at 600 s favours short (0.618 vs 0.408) and characters kept per shown favours long (20.183 vs 6.486). What should you do?
Counting winners treats three different questions as votes on one. A value-per-interruption decision reads characters kept per shown with deletions as the guardrail; a habit-quality decision reads persistence with reach as the guardrail. The decision picks the column, and the guardrail is the promise to keep watching the column that disagreed.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Leader session 5 cheat sheet · pin this
You now have a tracking plan, an activation candidate, an engagement curve you can defend, an adoption table and an AI metric sheet. Session 6 is the product review: one dashboard, six questions asked in order, definition, cohort, denominator, sampling, bots, cause, worked on real numbers from GitHub's public diary, including the ones on these pages. Bring every artifact you have built.