Interviews and coding at scale

Every qualitative AI feature on the market has the same shape: it applies your existing tag structure to new material at speed. Read that sentence twice. If your codebook encodes the frame you arrived with, a tool that applies it perfectly will return a beautifully consistent confirmation of the frame you arrived with - and it will do it across a hundred transcripts, which makes the confirmation look like weight of evidence.

Three practical rules stop this, and none of them are expensive:

• Code a sample by hand first. Six to ten transcripts, no tool, before the codebook is finalised. This is the only phase where you can still be surprised, and being surprised is the whole point of talking to people.
• Keep an explicit "does not fit" bucket. Every segment that resists your codes goes there, and the bucket gets read at the end. A codebook with no residual is not comprehensive, it is closed.
• Check for tool-only codes. Pull the list of codes that never appear in your hand-coded sample and only appear where the tool applied them. Each one is either a genuine discovery you missed or a pattern-match artifact, and you cannot tell which without going back to the audio.

The order matters more than the tooling. Human sample, then codebook, then tool, then residual review. Reverse any two of those and you have automated your own priors.

Asking a model for themes gets you themes, always, whether or not the data has any. Ask for the opposite:

All three outputs are checkable against the transcript, which is what makes them usable. "The key themes are..." is not checkable, and a theme you cannot check is a theme you cannot defend when someone asks how many participants actually said it.

Cohen's kappa measures agreement between two coders above what chance alone would produce:

κ = [Pr(a) - Pr(e)] / [1 - Pr(e)], where Pr(a) is observed agreement and Pr(e) is the agreement expected by chance.

The chance correction is the point. Two coders applying a codebook where 90% of segments get the same common code will agree most of the time by construction, and raw percentage agreement will flatter them. Kappa strips that out.

The thresholds everyone quotes come from Landis and Koch (1977). The caveat almost nobody quotes comes from McHugh (Biochem Med 2012;22(3):276-282), who argues those bands are too lenient when the coding feeds a consequential decision, and proposes a stricter reading in which 0.60-0.79 is moderate and 0.80-0.90 is strong. Both readings are in the table below. Report the number and say which ladder you are reading it against - that single sentence pre-empts the entire argument.

Guest, Bunce and Johnson (Field Methods 2006;18(1):59-82) ran 60 interviews and reported code saturation by the 12th, with about 80% of all codes appearing within the first 6. It is the most-cited number in applied qualitative work and the most misused.

• What it licenses. Planning honestly. If you are scoping a study and someone asks why 15 interviews rather than 50, you now have a published basis for the shape of the curve and a reason to build in a check.
• What it does not licence. Stopping at six because 80% is most of them. The last 20% of codes are frequently the ones that disagree with you, and the study that produced this finding ran 60 interviews to be able to say it.
• What to report. Not "we reached saturation" as an assertion. Report where new codes actually stopped appearing in your data: "no new codes appeared after interview 14 of 18." That is a checkable sentence, and it is the qualitative equivalent of recording your number's location.

One more honesty note. Code saturation and meaning saturation are not the same thing. Running out of new labels is not the same as understanding the labels you have, and a tool that stops surfacing new codes tells you about the tool's sensitivity as much as about your data.

Dovetail documents its stack unusually well, which makes the gap easier to see:

• Magic Highlight auto-applies your existing tag structure to new material. That is the exact mechanism Part 1 warned about, described accurately by the vendor.
• Magic Cluster groups highlights; chat features are "Powered by the latest release of Claude"; transcription runs on Amazon Transcribe and AssemblyAI.
• Dovetail states that customer data is not used for training, which is a real and checkable commitment and matters for Part 4.
• The gap: its AI documentation carries no accuracy figure and no instruction to verify output. You are told what model family sits behind the feature and nothing about how often the feature is right.

Looppanel claims "95%+ accurate transcripts", scoped to English - and its own pages elsewhere say 90% and 93%+, so the figure is inconsistent across the site. It supports 96 further languages, but translation and non-English AI notes are gated to Enterprise. So the headline number applies to one language, and the multilingual capability the pricing page advertises is not the capability the accuracy claim describes.

Marvin offers cited answers and PII blur, with no accuracy claim and no published language list. Cited answers are genuinely useful - a citation is a location you can check - but "cited" is a property of the interface, not evidence about the classification underneath it.

Neither pattern is disqualifying. Both are things you write down next to the tool's name before you let it near a claim.

Tools change monthly; these questions do not. Ask them at evaluation, write the answers into your methods appendix, and re-ask them at renewal:

• What exactly does the feature apply? My codebook, its own inferred categories, or a generic sentiment scheme? Only the first is auditable against my frame, and only if I read the frame first.
• Is there a published accuracy figure for the coding step, and on what data? A number without a methodology is positioning, not evidence. Absence of a number is information too - record it.
• Where does the transcript go, who can see it, and is it used for training? This is Part 4's question, and it is the one with legal consequences.
• Can I export the full mapping of segment to code? If you cannot get the assignments out, you cannot compute agreement, spot tool-only codes, or reproduce the analysis - and your audit trail has a hole where the analysis should be.

This is the single most consequential misunderstanding in applied qualitative research. Article 4(5) defines pseudonymisation, and Recital 26 makes the consequence explicit: data which could be attributed to a natural person by the use of additional information should be considered information on an identifiable natural person.

What that means in practice: replacing "Sarah, Head of IT at a named university" with "P07" before pasting the transcript into a third-party tool does not take the transcript outside GDPR. You still hold the key. The role, the institution type, the specific procurement anecdote and the speech patterns are all still in there. Pseudonymisation is a genuine and expected safeguard - it is just a safeguard applied to personal data, not an exit from the regime.

The operational consequence is small and specific: the tool you paste into needs the same treatment as any other processor of personal data. Know where the text goes, know whether it trains anything, know how long it is kept, and record what you knew at the time.

The Second Schedule research exception permits use without consent, but only where all of the following hold:

• the research purpose cannot reasonably be accomplished without individually identifiable data;
• there is clear public benefit;
• the results will not be used to make decisions affecting the individual; and
• published results do not identify the individual.

Read the third condition against a commercial insights project. If your interviews feed a decision about the participant's own account, pricing or renewal, the research exception is not the route you are on. That is a design constraint worth discovering before fieldwork rather than during review.

For anyone doing market or UX research under the professional code, three provisions apply the moment AI enters the workflow:

• Article 7(f) - research data used with AI must remain confidential, with access "strictly limited to a secure and controlled environment". A consumer chat window is not that environment, and the article is what you point at when someone suggests pasting transcripts into one.
• Article 6(c) - guard against re-identification by inference. Note how directly this speaks to Part 1: a rich thematic summary of a small sample can re-identify a participant without ever naming them.
• Article 4(a)(ii) - where a synthetic persona collects data, that must be clearly notified to the data subject at the beginning of the research. Not in a debrief, not in a footnote.

One rule with no exceptions: anything you present as a participant's words must be verbatim from the transcript. Never a model's paraphrase, never a tidied composite, never a "representative" quote assembled from several people.

This is easy to breach accidentally now. Ask a tool to summarise a code and it will hand back fluent sentences in the participants' register, and those sentences will be extremely quotable. They are not quotes. A summarising model is optimised to sound like the source, which is precisely what makes its output dangerous in quotation marks.

The workflow that prevents it: quotes are copied from the transcript with a line or timestamp reference recorded next to them, and the reference travels with the quote into the deck, the memo and the paper. Light edits for readability are normal practice - mark them, keep them to disfluencies, and never let an edit change what the person meant.

AAPOR's position is direct: margin of sampling error "does not apply to opt-in online surveys", and "it is impossible to develop statistically valid margins of sampling error from nonprobability surveys". The substitute is a model-based credibility interval, which is valid only if the adopted model is valid.

Practically, that draws a line you can hold: directional signal and within-sample contrasts, yes; probability-grade projection to a population, no. "Of the 400 lecturers we surveyed, 62% said their department chooses the tool" is a defensible sentence. "62% of lecturers, plus or minus 4.9 points" is not, and it is the version that gets quoted back at you.

Someone will offer to replace the panel with model-generated respondents. The best available evidence on that is Bisbee et al. (Political Analysis 2024;32(4), DOI 10.1017/pan.2024.5), who compared persona-generated responses against the American National Election Studies. Averages corresponded closely - and that is the trap, because averages are what you check first. But "48% of coefficients... are statistically significantly different" from their ANES counterparts, and among those cases the sign of the effect flips 32% of the time.

Read that as a specific warning rather than a general one. If you only need a mean, synthetic data may look fine. If you need a relationship - does budget ownership predict willingness to pay? - roughly half the coefficients are wrong and a third of those point the wrong way. Relationships are exactly what research questions are made of.

On the professional-code side: ESOMAR's 2025 guidance covers augmented synthetic data only, meaning boosting and imputation, and it does not endorse fully synthetic respondents as a replacement for human samples. If a vendor implies otherwise, ask them to point at the passage.