How much traffic a test needs, and what the minimum detectable effect is

Most test plans have a sample size somewhere near the top, something like 52,000 visitors per variant, and most people commissioning the work nod at it and skip to the designs. That number rests on one assumption that decides whether the test can answer your question at all, and you can check it in a couple of minutes.

The assumption is the minimum detectable effect, or MDE. It’s the smallest change the test is designed to catch reliably, if that change exists. With the usual settings of 80% power and a 5% significance level, a test sized for a 10% lift will spot a real 10% lift about four times in five. A real 4% lift will mostly slip through and come back as not significant, and then gets reported as “didn’t work”. If power is a new word, the core concepts cover it.

The part that catches people out is how quickly the numbers grow. The sample you need rises with the square of how small the effect is, so halve the effect you want to detect and you need roughly four times the visitors.

A worked example

Evan Miller gives a rule of thumb that is close enough for checking a plan. Visitors needed per variant is sixteen times the variance, divided by the square of the absolute effect. For a conversion rate the variance is the rate times one minus the rate. The sixteen bakes in 80% power and 5% significance.

Say the test runs on a product page that converts 3% of visitors to an order. You want to catch a 10% relative lift, from 3% to 3.3%, so the absolute effect is 0.3 percentage points, or 0.003. Two variants, split evenly.

The variance is 0.03 times 0.97, which is 0.0291. Sixteen times that is 0.4656. The effect squared is 0.000009. Divide one by the other and you need about 51,700 visitors per variant, so around 103,000 in total. A full calculator such as Evan Miller’s gives about 53,200 per variant for the same inputs, because it allows for the variant’s rate being a little higher. The rule is slightly optimistic, which is fine for a sense check.

Now hold everything else and change only the MDE, assuming the page gets 40,000 visitors a week:

Lift you want to detect	Variant rate	Visitors per variant	Weeks to reach it
20% relative	3.6%	about 12,900	under 1
10% relative	3.3%	about 51,700	about 2.6
5% relative	3.15%	about 207,000	about 10.3

Going from a 10% lift to a 5% lift turns a three week test into a quarter. And the 20% row doesn’t mean a five day test, because a test still needs whole weeks of traffic, which how long to run a test covers.

Checking an agency’s estimate

You don’t need to redo the statistics. You need the four inputs, and each one can hide a problem.

Ask for the baseline rate and where it came from. It should be the rate on the page in the test, for the metric in the test, over a recent normal period. A site-wide conversion rate used for a test on one product page will give you the wrong answer.

Ask whether the MDE is relative or absolute. A 10% relative lift on a 3% rate is 0.3 percentage points. A 10 percentage point lift would take you from 3% to 13%, and calculators offer both, so it’s an easy mix-up to make and an expensive one to miss.

Ask for the power and the significance level. If they’re not 80% and 5%, ask why, and get the answer in writing.

Ask how many variants there are, and count the visitors who will see the change, not everyone on the site. Each extra variant needs its own sample. And if the change sits below the fold or behind a click, the traffic that counts is the traffic that reaches it. Microsoft’s experimentation team make the same point, that the power calculation sets a floor on how many people have to be in the test.

Then run the rule of thumb yourself. If your answer lands within about 10% of theirs, the arithmetic is fine and you can move on to whether the MDE is believable.

That’s where most plans go wrong. The usual pattern is an MDE worked backwards from the timeline. Someone wanted the test done in a fortnight, the calculator was fed whatever lift made that true, and the result was a test that can only detect a 25% jump on a page where nothing you’ve shipped has ever moved it by more than a few per cent. It’s an understandable thing to do under deadline pressure, and it means a likely real but modest effect gets thrown away as a loss.

When you don’t have the traffic

Bigger expected effects are the only lever that shrinks the sample a lot, so the first option is to test bolder changes. A rewrite of how delivery is explained is more likely to move people than a new shade of button.

You can move the metric up the funnel, to add to basket instead of orders, where there are more events to count. That buys speed at the cost of measuring something one step from the money, which choosing metrics and guardrails goes into.

You can test on a template that many pages share, so more traffic sees the change. Some tools reduce noise with pre-test data, often called CUPED, which lowers the sample you need, and questions to ask your testing tool covers how to find out if yours does.

You can accept a lower confidence level, as long as you write down the higher false positive risk and everyone signs up to it.

Or you can decide this one isn’t a test. Some changes are better made on the strength of research and watched carefully afterwards.

Whichever you choose, settle it before the build. An MDE is a statement about the smallest change the business would act on, and if your traffic can only see a lift three times bigger than anything you’re likely to get, it’s far cheaper to find that out from the plan than from the readout.

Put it to work

Take the sample size from the last test plan you were sent and check it with the rule of thumb, sixteen times the baseline rate times one minus the baseline rate, divided by the absolute effect squared.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Evan Miller, Sample Size Calculator
2. Evan Miller, How Not To Run an A/B Test
3. Microsoft ExP, Patterns of Trustworthy Experimentation, Pre-Experiment Stage
4. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments