Writing survey questions that give you usable numbers

A survey is a measuring instrument, and like most instruments it reports whatever the operator points it at. The trouble is that the operator is usually the person who wants a particular answer, and the survey will happily give it to them. A lot of good survey design comes down to getting out of your own way.

The two most common mistakes are easy to spot once you’re looking. A double-barrelled question asks two things at once. “Was the checkout fast and easy?” tells you nothing when it was fast but confusing, so split it. A leading question pushes the answer. “How much did you like our new checkout?” presumes they liked it. Ask it plain, “how would you describe the checkout?”, and let the answer land where it lands. A self-reported answer is already a weak proxy for behaviour, so don’t make it weaker by steering it.

Then there’s the problem you can’t fix with wording, which is who answers. A survey that goes out to anyone comes back from the people who care enough to reply, so the results mostly measure how enthusiastic people are about you. The loudest respondents are the ones with a grievance or a crush on the brand, and neither group is a representative sample. That’s why survey numbers deserve a healthy dose of suspicion. A result saying 80% of respondents would buy means nothing if the respondents were the people who already wanted to.

Used well, a survey does a narrow job, and that’s enough. Surveys are good at sizing a problem you already understand, “of the people who left, how many left because of delivery cost?”, and at checking a specific fact across a lot of people. They’re bad at discovery, because you can’t ask people to name a problem they haven’t consciously noticed. Nobody answers “what frustrates you?” with the thing that frustrates them most, because by now they’ve stopped noticing it. So for discovery, watch people, and save the survey for when you need scale.

Watch out for the survey written as a press release, designed to produce a number that looks good in a deck. The questions are warm, the sample is everyone who follows the brand, and the conclusion is that customers love the thing. It feels good and tells you nothing, and you can usually tell because it was written by someone who already knew what they wanted it to say. One rule guards against it: before you send a survey, write down what result would change your mind. If no result would, all you’re doing is collecting reassurance.

Put it to work

Find one question in a survey you have run or seen. Check it for two failures: it asks two things at once, or it pushes the answer one way. Rewrite it plain.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. NN/g, Survey Design and Execution
2. NN/g, Writing Good Survey Questions