Analysing qualitative data without finding what you wanted

The hardest part of research is reading the data fairly once you’ve got it. You’ve got five interviews or ten usability sessions, and somewhere in those notes is the answer. The answer you were hoping for is in there too, and the two might well be different. A lot of the discipline is stopping yourself reaching for that second one.

The method is called thematic analysis, and it’s less complicated than it sounds. You take every observation, every quote, every moment someone stalled, and give it a short descriptive label, such as “Confused by delivery cost”, “didn’t understand ‘standard’” or “wanted to speak to someone”. Conclusions come later. Then you group the labels. The groups are your themes, and the themes are the finding. What matters most is the order: you tag before you know what the themes are. If you tag with a conclusion already in mind, you end up sorting evidence into boxes labelled with your own opinions.

That order is what protects you. Start from the conclusion and every ambiguous quote resolves in its favour, so the data ends up agreeing with whatever you believed on Monday. Start from the labels and the data has a chance to surprise you. A theme you didn’t expect, and didn’t want, is worth ten that flatter you, because it’s the one you’d otherwise have missed. A good sign your analysis was fair is that you can name the finding you didn’t come looking for.

The thing to watch for is the pulled quote, the one perfect sentence lifted out of five hours of tape to prove a point. It’s the most common form of research dishonesty because it feels harmless. The quote is real, you didn’t make it up, it’s just out of context. But a quote chosen because it supports a view can’t then be used as evidence for that view. There’s a simple check: could the same data have produced a different quote? If it could, and you only pulled the one you liked, you’ve been doing archaeology, digging until you find the artefact that proves the story you’d already told.

So do the tagging, then the grouping, then write down what the data shows, including the awkward bits. The findings that surprise you are the ones worth the effort, because you couldn’t have got them by sitting at your desk and thinking hard about it.

Put it to work

Take your last batch of interview or test notes. Tag each observation with a short label, then group the labels. Write down the theme the grouping shows, even if it is not the one you expected.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. NN/g, Analyzing Qualitative UX Data