Checking a test is sound before you trust its result

Kohavi, Tang and Xu give a whole chapter of their book to Twyman’s law, the old rule from audience research that any figure that looks interesting or unusual is probably wrong. It’s the right instinct for a results meeting. When a test comes back with a big, surprising lift, the first thing to rule out is that the test was broken, and most of the ways a test breaks can be caught by someone who never touches the code, as long as they know what to ask for.

Before launch

Your first A/B test covers the basics of QA. As the commissioner, your job is to see the evidence, which usually means asking for five things.

Screenshots or a recording of every variant on the browsers and devices your customers use, including a real phone. Proof that the conversion event fires in every variant, ideally a test order or form submission that turned up in the tool’s report. The targeting rules in plain words, which pages, which visitors, and whether staff and internal traffic are excluded. Confirmation that anything downstream still works, such as forms arriving, payments going through and confirmation emails sending. And a preview link so you can click through it yourself.

None of this is hard. It gets skipped because launch dates slip and QA is the last step before the deadline. Asking for it as a matter of routine takes the pressure off the person doing it.

Sample ratio mismatch

Once a test is live, the most useful single check is whether the traffic split matches what you set. If you asked for 50/50 and got something noticeably different, that’s a sample ratio mismatch, or SRM, and it usually means something is broken. Microsoft’s experimentation team report that about 6% of their A/B tests have one, and they treat an SRM as something that typically invalidates the result.

The causes are rarely random. A variant that redirects people to a new URL can lose some of them on the way. A variant that throws an error on one browser drops those visitors before they’re counted. Bucketing bugs, broken logging and uneven ramp-ups all show up the same way. In one Microsoft example the variant made people more engaged, a bot filter mistook the most engaged people for bots and removed them, and fixing it flipped the result to positive.

The check is a chi-square test, and the sum is short enough to do by hand. Say a 50/50 test reports 50,000 visitors in control and 48,800 in the variant.

1. Add them up to get 98,800. Half of that is 49,400, which is what each variant should have had.
2. Each variant is 600 away from 49,400. Square that to get 360,000.
3. Divide by the expected count. 360,000 divided by 49,400 is about 7.29.
4. Do the same for each variant and add them up. Here both are the same, so the total is about 14.6.

Then compare it with a threshold. Microsoft use a deliberately strict cut-off of a p-value below 0.0005, to avoid false alarms when every test is being checked. For a test with two variants, that works out at a total above about 12.1. This one is 14.6, a p-value of roughly 0.0001, so it fails, even though the variant is only 2.4% short and looks like a rounding error on the dashboard.

Compare 10,000 against 9,850. The expected count is 9,925, each is 75 out, 75 squared is 5,625, divided by 9,925 is about 0.57, and the total is about 1.1. Comfortably fine. The ratio alone can’t tell you which is which, because the same percentage gap means more in a big sample than a small one.

For an uneven split, such as 90/10, use the expected share for each variant instead of half. For three or more variants the threshold changes, so use a calculator such as Lukas Vermeer’s free SRM checker, which takes the counts and the split you intended.

If a test fails the check, don’t read the result. Find the cause, fix it, and restart. Microsoft’s advice is to check for SRM early and set up alerts, so a broken test can be stopped before it wastes weeks of traffic.

Flicker

Most client-side testing tools change the page after the browser has started to show it. If the script arrives late, visitors see the original for a moment before it swaps to the variant. That’s flicker, and it means the variant group is seeing your change plus a page that jumps about, so you’re measuring both at once.

The usual fix is an anti-flicker snippet that hides the page until the test has loaded. That trades flicker for a blank screen. DebugBear looked at one company’s homepage that stayed hidden until about six seconds, and removing the hiding brought the main content in at about 2.7 seconds. A slower page in both variants can drag your baseline down without anyone noticing.

Ask how the tool loads, whether anyone has watched the variant load on a slow phone connection, and whether big changes would be better built server-side, where there’s no swap to see.

Broken tracking

Variants change the page, and analytics tags often hook onto the page’s structure. Rename a button or rebuild a form and the tag that counted conversions can stop firing in one variant only. The result is a huge loss or, if a duplicate fires, a huge win, and both look like news.

Ask to see the daily count of the primary conversion in each variant for the first few days, next to the same count from your main analytics. If the tool and your analytics disagree sharply for one variant but not the other, something is broken.

Bots

Automated traffic inflates visitor counts and dilutes any real effect. That’s a nuisance when it hits both variants evenly. It becomes a problem when the variant changes how bots behave or how the filter treats people, as in the Microsoft example, and then it often shows up as an SRM. Ask how the tool identifies bots and whether the same filter applies to every variant.

What to ask for in the readout

Before the result, ask for the split check with its numbers, the daily conversion counts by variant, and a note of anything that changed while the test ran, such as a site release, a tracking change or a campaign. A result that comes with those is one you can defend in a meeting. A result that doesn’t is a result you’re taking on trust, and Twyman’s law says to hold off until you’ve seen them.

Put it to work

Take the visitor counts from your last finished test and run the sample ratio check. Work out the expected count per variant, square each difference from it, divide by the expected count, add them up, and see whether the total goes over 12.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Microsoft ExP, Diagnosing Sample Ratio Mismatch in A/B Testing
2. Fabijan and others, Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (KDD 2019)
3. Lukas Vermeer, SRM checker
4. Microsoft ExP, Patterns of Trustworthy Experimentation, During-Experiment Stage
5. Matt Zeunert, DebugBear, Anti-Flicker Snippets From A/B Testing Tools And Page Speed
6. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments