The question behind the question

The first three slots make a question computable. The fourth decides whether computing it is worth anybody's afternoon, and it is the one almost nobody asks about.

Ask it plainly: what will you do differently depending on what I find? Three answers are common, and all three are useful. Sometimes it is specific, which tells you exactly how precise the answer needs to be. Sometimes it is "nothing, I am just curious", which is a fine answer and tells you to spend twenty minutes rather than two days. And sometimes the honest answer is "we have already decided, I need the number for the deck", which is also worth knowing, because that is not analysis and pretending otherwise wastes everybody's time.

The version of this question with the best pedigree is from DARPA, where the Heilmeier Catechism asks a programme manager what they are trying to do in plain language, who cares, what happens if they succeed, and how long it will take. The two that transfer directly into an analysis request are who cares and what difference will it make, and asking them at the start of a piece of work costs one sentence.

This is practitioner craft rather than a research finding, and it is worth being straight about that. There is a long-standing process model for analysis work, CRISP-DM, published by a cross-industry consortium in 2000, and its first phase is business understanding rather than data understanding, which encodes the same idea: you are supposed to establish what decision the work serves before you touch the data. It has six phases in total and it is the ancestor of nearly every process diagram you will see in this field.

Whether it is still the most used process model is a weaker claim than it is usually given. The evidence for the "most used" line rests mostly on two self-selected online polls with a couple of hundred respondents between them; there is a stronger academic literature review showing it dominates among published workflow adaptations specifically, which is a narrower statement. Microsoft's own analytics guidance gives a similar instruction in fewer words, telling you to ask a sharp question, meaning one whose answer is a name, a number or a yes.

None of this is measured. No study shows that teams filling four slots produce better analyses than teams who do not, and nobody appears to have tried. What can be said honestly is that the failure mode is real, common and expensive, and that every process model in the field independently puts a version of this step first.

Somebody followed the citations properly. A product marketing page cited a 2018 blog post, which cited a 2017 blog post, which cited a 2016 industry survey. Four links, and nobody along the way measured anything new.

At the end of the chain, the 2016 survey asked people what they spend the most time doing and reported cleaning and organising at 60 percent and collecting data sets at 19 percent. Eighty percent appears only if you add those two together, which is not how the survey framed them. Every other year gives a different figure, and an independent survey run by a different organisation put the combined total nearer a quarter. The oldest version of the claim, from a 2014 newspaper article, gives a range of fifty to eighty percent sourced to interviews.

So the honest summary is this: data preparation is a large part of the job, every source agrees on that, and there is no measurement anywhere of what fraction of a day it takes. That sentence is both truer and more useful than the number. It also survives contact with someone who has the survey open in front of them, which the number does not.

One caution about my own sources here, in the spirit of the session: the archaeology above comes from a named practitioner's write-up and a reposted press release. The original 2016 survey PDF would not load on any attempt, so nobody in this chain, including this page, has read the primary document. That is the honest tier, and it is stated rather than hidden.

Who measured this, and how? Self-report and observation are different things, and a survey asking "what takes you the longest" is not a time-use study. Most circulating statistics about work are the former dressed as the latter.

What exactly was the denominator? Percentage of what: of people, of time, of projects, of respondents who answered a particular way? The 2019 figure in the table is a proportion of respondents, and the 2016 one is a proportion of respondents naming a task. Neither is a share of a working day, and both get quoted as though they were.

How many links back is the original? If a claim has a source, check whether that source has a source. Two hops is usually enough to find either a real study or a dead end, and finding the dead end takes about five minutes.

Notice what happened when the four slots were filled in. Everybody said "did the promotion work". Only when you write the timeframe down does it become obvious that the founder is asking about a period the data does not cover, because the rows stop twelve weeks after the offer and a renewal cycle has not happened yet.

That is not a failure. It is the single most valuable thing you can produce on day one, because it arrives before anybody spends a week on it, and because it converts into a concrete request: extend the collection window, or wait, or accept that this question gets answered in the spring rather than now. Session 2 turns that into a written gap list.

The failure mode is answering a nearby question instead. The founder asks about retention, the data supports activation, so the deck reports activation and the sentence "the promotion worked" is said out loud. Nobody lied. Nobody checked. The organisation now believes something it has no evidence for, and it will plan the spring around it.