Can you run multiple tests at once?

The worry sounds responsible. If two tests run at the same time, surely they contaminate each other, so you should queue them and run one at a time to keep things clean. It’s an understandable instinct, and one of the most expensive habits in the discipline.

Start with why concurrency is usually fine. When you randomise properly, every test you run splits its traffic evenly across the variants of every other test. So test B’s variants land in roughly equal measure on test A’s control and on test A’s treatment, and B’s effect averages out within A’s comparison. Each test still gets a clean read of its own change. The large experimentation programmes at Microsoft, Google and Booking.com run hundreds or thousands of overlapping experiments at once for exactly this reason.

Two tests, one audience

Run Test A and Test B at the same time. Each visitor is dropped independently into a group for both, so every person lands in one of four buckets. Send some visitors and watch where they fall.

	Test B control	Test B variant	Test A total
Test A control	0 0%	0 0%	0
Test A variant	0 0%	0 0%	0
Test B total	0	0	0

Send some visitors to see how they split. Ink bars are Test B control, red bars are Test B variant.

Send 50 visitorsReset

What about interaction effects, where two changes amplify or cancel each other? They’re real, but they’re rare and usually small. They matter mainly when two tests touch the same thing, the same element or the same step of the funnel. Two tests both rewriting the same call to action, or one redesigning the basket while another redesigns the checkout right after it, can interfere, because a visitor meets both and the changes aren’t independent.

The mistake comes in two opposite flavours. The first is the over-cautious serialiser, who runs one test at a time and turns what could have been a year of learning into a fraction of it. The second is the reckless overlapper, who runs two tests on the same button and then can’t say which change moved the needle. There’s a less obvious third one too. Run enough tests and metrics at once without minding multiple comparisons, and one of them will cross the line by luck alone.

So run concurrently by default. Run your tests at the same time, randomise properly, and save the caution for the cases that need it. When two tests share an element or a funnel step, do one of three things: coordinate them, put them in mutually exclusive audiences so no visitor sees both, or fold them into a single multivariate test designed to measure the combinations. And keep multiple comparisons in check by deciding your primary metric in advance, so you aren’t trawling the report for anything that looks significant.

Running everything one at a time feels safe, but it buys a kind of cleanliness you almost never needed, and you pay for it in learning you’ll never get back.

Put it to work

List your live and queued tests. Flag any two that touch the same element or funnel step, and isolate only those.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments
2. Georgi Georgiev, Analytics-Toolkit on concurrent experiments