Watching recordings without fooling yourself

Session recordings and heatmaps solve the cost problem of research. A live usability test means one person, a room and an afternoon. A recording tool lets you watch a hundred real sessions from last week, from anywhere, for the price of the subscription, and heatmaps add those sessions up into a picture of where people click, scroll and hover. For spotting friction at scale you won’t find anything cheaper.

Heatmaps are best for the broad question: where attention lands, what people try to click that can’t be clicked, what gets scrolled straight past. Someone tapping a phone number with no link on it is a finding, and so is a form field that gets a burst of rage-clicking. The heatmap won’t explain any of it, but it gives you a shortlist of suspects, and that’s most of the value.

Recordings are best for the specific question, and they only work if you decide the question before you watch. Open a recording tool without one and you’ll watch people scroll for an hour and come away with a feeling. Decide first, say “why do people who add to the basket not finish?”, then watch twenty sessions where that happened and tally what you see. The tally is your analysis.

The trap is built into the tool. It lets you watch anything, which means it lets you watch only what you want to see. If you already believe the problem is the price, you’ll find the three sessions where someone muttered about the price and call it proof, and skip past the seventeen where they gave up at a broken form. That’s confirmation bias, and a recording tool pointed the wrong way will feed it all day. The fix is mechanical: pick the question, fix the sample and tally the behaviour, all before you let yourself form a view.

So treat recordings as the volume layer. Live usability testing finds a problem in five people. Recordings tell you whether it shows up in five hundred, and whether it’s one segment or everyone. What they can never tell you is what the person was thinking. The scroll-back, the hover, the half-typed email someone abandoned, that’s all behaviour, and it’s good evidence of what happened, but the gap between what someone did and why they did it is still where most of the insight is. If anyone tells you a recording explained a customer to them, they’re guessing. They might be right, but it’s still a guess.

Put it to work

Pick one question you want recordings to answer before you watch any. Then watch a fixed number, say twenty sessions, and tally what people did.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. NN/g, Heatmaps and Visual Analytics for Website Optimization