How many tests should you be running?

Every programme owner asks this hoping for a number, four a month or maybe ten. What you get instead is a constraint, and your own traffic has already set it.

The first limit is traffic. Every test needs a sample size to stand a fair chance of detecting a real effect, which comes from your baseline rate, the smallest effect you care about, and the power you want. Your weekly conversions are finite, so the number of adequately powered tests you can finish in a quarter is finite too. A site with two hundred conversions a week and one with two hundred thousand aren’t playing the same game, and no amount of enthusiasm changes that.

The second limit is ideas. You can only run as many good tests as you can design, and testing a stack of weak hypotheses quickly mostly produces noise. Sharp tests come from research, from analytics and from talking to users, and that work takes the time it takes.

Trouble starts when someone decides tests per month is the metric and the team starts manufacturing tests to hit it, because win rates are low. Optimizely reports that its clients win around one test in five on average, and only one in ten of the tests tied to revenue. The big experimentation teams at Google and Bing sit around ten to twenty per cent, and Microsoft’s celebrated one in three came only after heavy upfront work scoping ideas before they ever ran. A fair planning assumption is somewhere between ten and thirty per cent, leaning to the lower end. Volume does help you find the wins hiding in that pile, but only powered volume. Underpowered tests run to make up the numbers produce false positives and bad ship decisions instead of wins, and that’s worse than running nothing.

How many wins is that a year?

Most tests do not win, so the wins come from volume of good ones. Set your pace and a win rate to see what a year of testing tends to return.

Tests per month4

Win rate15%15% is a fair planning average for most programmes.

≈ 7 winning tests a year

From 48 tests a year at a 15% win rate. Over three years, roughly 22 wins, the kind of compounding that separates busy programmes from effective ones.

Win rate climbs with better research and sharper hypotheses, not by running more weak tests, and you can only run as many as your traffic can power. The number here is a planning estimate, not a promise.

It’s more useful to stop counting tests and count two other things instead: validated learning and shipped wins. Size each test to your traffic, prioritise ruthlessly with whatever framework you like, and protect the quality of the hypothesis above all. A programme running four sharp, well-powered tests a month that ships one real win is beating one running twelve underpowered guesses, every time, even though the second one looks busier in the board deck.

One lever does raise the ceiling, and that’s running tests at the same time rather than one after another, which is the subject of its own lesson. Short of that, run as many tests as you can power properly and design well, and not one more. Pushing past that just fills your roadmap with tests that were never going to tell you anything.

Put it to work

Estimate your own ceiling. Work out how many adequately powered tests your weekly conversions support, then plan to that.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments
2. Optimizely, Experimentation metrics for program success
3. CXL, 5 things we learned from analyzing 28,304 experiments
4. Georgi Georgiev, Analytics-Toolkit