How long to run a test, and when to stop it

The test hit its sample size on a Wednesday, the variant is ahead, and the agency would like to call it. The sample size was the right thing to plan around, and the lesson on it shows how to check one. But it only tells you how many visitors the test needs. It says nothing about which visitors, and a test that stops midweek has seen more of some kinds of customer than others.

Whole weeks and business cycles

People behave differently on different days. On a lot of sites the weekday crowd and the weekend crowd differ, and many have a payday pattern on top. Your own analytics will show you which days differ and by how much. If a test runs from Monday to Wednesday of one week and Monday to Sunday of the next, the weekdays count twice and the result leans towards weekday shoppers.

So run in whole weeks. Microsoft’s experimentation team say their typical test runs for 7 days, and they recommend splitting every analysis by date, so a single odd day stands out instead of hiding in the total. Your first A/B test suggests fourteen days as a floor, which gives you two of every weekday and two weekends, so one unusual Saturday can’t carry the result on its own.

Then think about your own cycle. If your customers take a fortnight to decide on a sofa or a holiday, a one week test mostly counts the fast deciders. People who first saw the variant in week one and came back to buy in week three never make it into the numbers. Run long enough to cover at least one full buying cycle, and ask the agency what they think yours is and how they know.

Novelty and primacy

Some changes win at first because they’re new. Others lose at first because they’re unfamiliar. Microsoft researchers define the novelty effect as the urge to try something new, which fades over time, and the primacy effect as engagement that grows as people get used to a change.

Microsoft give a clear example of novelty. Microsoft News swapped one button on its homepage for another and clicks on it went up 28%. When the team broke the results down by day, the clicks were falling fast, day after day. They put it down to novelty, people confused and clicking more than once, and didn’t ship the change.

Primacy runs the other way. Returning customers who know the old layout can do worse with a new one for the first week or two, and then recover or overtake. A test stopped early would call that a loss.

Both effects mostly affect returning visitors, because a first-time visitor has nothing to compare against. So they matter most on sites people use often, such as a grocery shop, a bank or a software dashboard. The check is the same in both cases. Ask for the effect plotted by day or by week. If it’s steady, fine. If it’s shrinking or growing, the overall number is an average of a moving target, and the Microsoft researchers’ advice is to run longer and let it settle.

Sales, seasons and launches

A test run during a big sale tells you how sale shoppers respond. They arrive with different intent, often from different channels, and they may never behave like that again. The same goes for the run-up to Christmas, a TV campaign or a product launch that sends a crowd of new visitors.

Where you can, avoid launching across one. Where you can’t, note it in the plan, look at the results with and without that period, and treat the result as provisional until a test in a normal period agrees with it.

Don’t solve this by running everything for months. Longer tests have their own problems. A Microsoft paper on long-running experiments lists several, including cookies being cleared so the same person gets counted in both variants, and the people who stay in a test for months being a different group from the ones who drop out.

A/A tests

An A/A test splits traffic between two identical versions. Nothing has changed, so if the setup is sound the result should be dull. Optimizely’s guidance is that most A/A results should come back inconclusive. At a 5% significance level, about one comparison in twenty will still look significant by chance, so one surprising A/A result proves little on its own. Repeated ones point to a fault, and so does a sample ratio mismatch, which the quality checks lesson explains.

Ask for one when you start with a new tool, a new site or a major change to tracking. It costs a couple of weeks of traffic, and it tells you how much the numbers move from day to day when nothing has changed, which makes every later result easier to read.

When to stop

Agree the stopping rule before launch and write it down. A sound one has four parts. The test stops when it has reached its sample size, covered at least two full weeks and one full buying cycle, and passed the split check. It can stop early only if a guardrail is clearly breached or something is broken. And it has a maximum length, so a flat test doesn’t drift on indefinitely.

The two things to refuse are stopping the moment a winner appears and extending a flat test in the hope it turns significant. Both give chance another go at producing a result, which is why peeking does so much damage. Evan Miller makes the case well. If you do need to stop early and safely, use a method built for it, which is what sequential testing is for.

When an agency proposes a duration, ask how they got to it, whether it runs in whole weeks, what’s in the calendar during it, what would make them stop early, and whether you’ll see the effect by day. If the answers come back as a date and no reasoning, the date came from the calculator alone, and the calculator doesn’t know when your customers get paid.

Put it to work

For your next test, set the end date as the later of two dates, the day it reaches its sample size and the end of its second full week. Then add one line on any sale, promotion or holiday that falls inside that window, and what you'll do about it.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Microsoft ExP, Patterns of Trustworthy Experimentation, During-Experiment Stage
2. Sadeghi and others, Novelty and Primacy, A Long-Term Estimator for Online Experiments
3. Dmitriev, Frasca, Gupta, Kohavi and Vaz, Pitfalls of Long-Term Online Controlled Experiments
4. Optimizely, Run and interpret an A/A test
5. Evan Miller, How Not To Run an A/B Test
6. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments