The frequentist test, and the temptation to peek

Most A/B tests you’ll meet are frequentist, even when nobody says so. The logic runs like this. Assume there’s no real difference between control and variant, the null hypothesis, then ask how surprising your data would be if that were true. If it’s surprising enough, you reject the idea of no difference.

Before any traffic flows, you do a power calculation. You feed it the current conversion rate, the smallest change you’d care about, your power (usually 80%) and your significance level (usually 5%). Out comes a sample size. Treat that number as a promise: collect this much, then look.

The p-value is where good practitioners still trip. It’s the probability of seeing data at least this extreme if there were no real difference. It isn’t the probability that the variant is better, and treating one as the other is behind a lot of confident, wrong calls. A p-value of 0.04 does not mean a 96% chance the variant wins.

Now the big one, and the most common mistake in the discipline: optional stopping, better known as peeking. Every time you check an in-flight test and reserve the right to stop when it looks significant, you give random noise another go at crossing the line. A test with no real effect, peeked at once a day for a fortnight, can show “significance” a third of the time or more. The result feels earned, but mostly it’s down to how often you looked.

It’s worth being fair about why this happens. The dashboard updates in real time, a stakeholder wants the answer, and a winner showing today is hard to leave alone. The tooling invites it and the pressure rewards it, so trying harder not to look rarely works. What works is setting the test up so you don’t have to rely on willpower.

So commit up front. Run the power calculation, fix the sample size and the duration, name the primary metric before you start, and wait until the planned end to decide. If you do need to look early and act early, don’t bodge it onto a frequentist test. Use a method built for that, the sequential approach covered in its own lesson. And if you’re tracking several metrics, correct for it, because testing five things at 5% each is no longer a 5% risk.

Frequentist testing earns its place. It’s simple, every tool supports it, and the wider team broadly understands it. Its weaknesses are real too: it needs a fixed finish line, the p-value is misread constantly, and it never gives you what people usually want, a straight probability that B is better than A. The Bayesian approach sets out to fill that gap, and it has temptations of its own.

Put it to work

Before your next test, run a power calculation, fix the sample size and duration, and write them down. Then do not look early.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Evan Miller, How Not To Run an A/B Test
2. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments