When a test is the wrong tool

The testing tool is so good at producing confidence that it’s easy to forget it’s a tool with a place, and that it won’t settle every decision. Some decisions aren’t experiments. Running a test where one doesn’t belong produces a precise number for a situation the number can’t capture, and then everyone feels better about a call that was always going to be judgement.

The clearest case is traffic. A test needs enough people through each version to see a real effect, and most businesses, and most pages, don’t have that. Run a test on a page that gets fifty visits a week and you’ll wait months for an answer you still can’t trust, because the sample is too small to tell a real effect from noise. It’s better not to pretend. Make the change, watch it with good analytics, and say plainly that it was a judgement call. A judgement call made openly is worth more than a test performed badly, because the bad test looks like evidence.

The second case is irreversibility. Some changes can’t be taken back: a pricing change customers have already been told about, a brand decision that’s already gone to print, a data migration that’s already started. A test is a way of finding out whether a change works before you commit to it. If you’ve already committed, the test is theatre, and the time to run it was earlier.

The third is the decision that comes down to something other than the number. A test tells you whether a change moves a metric. It won’t tell you whether the change is right, whether it’s kind, or whether it respects the person on the other end. Some calls are about what kind of company you want to be, and no amount of significance will answer that. This is where the method and the ethics part ways, and it’s worth saying so out loud, because if you pick the metric carefully enough a test can be rigged to justify almost anything.

There’s also the change that isn’t worth the cost of testing. Building the test, running it and waiting for it all cost something, and for a small, low-risk change that cost can be more than knowing is worth. So test the things that matter and use judgement for the rest. A programme that tries to test everything spreads its effort too thin to do any of it properly.

The last trap is testing as performance: the test run because it looks rigorous, because the deck needed a number, because the stakeholder expects one and nobody wants to be the person who says this one is a judgement call. What you get is a precise-looking answer to a question nobody needed answered, and a programme that’s busy without being useful. Being clear about when not to test is what keeps people believing the results on the occasions a test is the right tool.

Put it to work

Look at your current backlog and flag any test that lacks the traffic, or the reversibility, to be a real test. Note what you would do instead.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Ronny Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments