Good test ideas start with a problem

A backlog is a list of things you believe will help, and how you fill it decides whether you’re doing research or collecting opinions. Done well, it fills from below: a usability test that showed people can’t find the delivery options, a funnel that shows half the baskets are abandoned, a support inbox full of the same confused question. Each of those is a problem first, and the test idea is a proposed fix for a problem you have evidence for.

The other way fills from above. Someone in a meeting says the button should be bigger, or the competitor’s site does something, or the founder never liked the homepage. The change exists first and the justification gets written afterwards. One question tells the two apart: what happened before the idea? If the answer is a meeting or a feeling, it’s opinion. If it’s an observation or a number, it’s research. In the list they look identical, which is why you have to ask.

The most common source of bad ideas is the tool itself. Visual editors make it very easy to move a button, change a colour or swap a headline, so the backlog fills with exactly those changes, because they’re what the tool makes cheap. That’s tool-led ideation, and it runs the process backwards: the change comes first because it’s easy, and the research never gets a look in. The fix is a rule that no idea reaches the backlog without some evidence attached, whether that’s an observation, a support pattern, a drop-off, or something else you saw before you thought of the fix.

A healthy pipeline has several sources feeding it, and they balance each other: research findings, analytics, support tickets, session recordings, past test learnings. Part of the point is that they disagree, and the disagreement is where the interesting problems live. An unhealthy one has a single source, usually the senior person’s instinct, and a backlog that’s their wish list with the word hypothesis typed over the top. You can tell because every idea in it was decided before any of it was researched.

The habit that fixes most of this is small: for every idea, write down the problem it solves and the evidence that the problem is real. If you can’t fill in the second column, treat it as a candidate for research rather than a test idea. Run the research, find the problem, and then the idea has somewhere to stand.

Put it to work

Take the last five ideas in your backlog and write next to each whether it came from an observed problem or from a preference. Notice the ratio.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. CXL, Research for Conversion Optimization
2. NN/g, Discovery: Building the Right Thing