Checking a results deck before you believe it

The deck arrives the night before the quarterly review. Slide four says the new product page lifted conversion by nine per cent and is worth four hundred thousand pounds a year. Everyone in the room will remember that number, and most of them will repeat it. Before you do, it’s worth fifteen minutes with the questions below. You don’t need to do any of the statistics yourself. You need to know which questions to ask, and what a good answer sounds like.

Kohavi, Tang and Xu put the right attitude in one line, borrowed from Twyman’s law: any figure that looks interesting or different is usually wrong. They encourage double-checking results, especially breakthrough positive ones. A big number earns more scrutiny, and you can be friendly about asking for it.

The checklist

1. Were the metric and sample size fixed before launch? Ask to see the test plan, with its date. It should name one primary metric, the guardrails, and the sample size or run length. If the metric on the results slide differs from the plan, ask why. Switching to whichever metric moved is one of the most common ways a flat test becomes a win.

2. Did the test run to completion? A classic frequentist test has to run to its planned sample and be read once. Stopping the moment it crosses significance inflates the chance of a false positive, and Evan Miller’s well-known piece on the subject shows how quickly that happens. Some tools use sequential methods built for early stopping, so ask which method the tool used and whether the stopping rule was followed. See the frequentist test, sequential testing and questions to ask your testing tool.

3. What’s the interval behind the headline lift? A lift of nine per cent is a best guess with a range around it. If that range runs from minus two to plus twenty, the test is consistent with no effect at all, and the slide should say so. If it runs from six to twelve, you have something much firmer. Ask for the interval on every result, wins and losses alike. See reporting wins and losses the same way.

4. Was the winning segment planned in advance? “It lost overall, but won for returning mobile visitors” is a sentence to slow down on. Every slice of the data is another chance to find a fluke, and a team that looks through twenty segments will usually find one that shines. Researchers at Microsoft and Airbnb note that even experts filter down to the significant results across metrics, segments and variants, and that doing so biases the estimate upwards. A segment result that wasn’t in the plan is a hypothesis for the next test, and it shouldn’t be shipped on its own. See multiple comparisons.

5. Did the traffic split come out as planned? If the test was set to split fifty fifty, the visitor counts for each version should be very close to equal. When they’re noticeably off, that’s a sample ratio mismatch, and it usually means something broke, such as a variant failing to load for some browsers or a redirect losing people. It happens more often than most people expect. A study of experiments at Microsoft found about six per cent showed one, and one case came down to a bug that set a fifty fifty test to 49.9 and 50. Ask for the visitor counts per version and whether the tool checked the split. A test with a mismatch can’t be trusted until someone explains it, however good the result looks.

6. What happened to the guardrails? A win on the primary metric alongside a rise in returns or cancellations needs both on the same slide. If the guardrails aren’t reported, ask for them. See what counts as a win.

7. How was the revenue projection calculated? The annualised figure is usually the lift multiplied by a year of revenue, and it nearly always runs high. Four reasons, all worth asking about.

• It uses the best guess. A projection built on the bottom of the interval is often a fraction of the one built on the headline.
• It has a selection problem. Tests get reported as wins because their result came out high, and some of that height is luck. Airbnb’s experimentation team described this as a winner’s curse and showed that adding up the measured effects of the tests you launched overstates their combined impact, even when every one of them was a real improvement.
• It assumes the whole audience. If only visitors on one page or one device saw the change, the effect has to be diluted. Kohavi and his co-authors give the example of a five per cent improvement for ten per cent of users, which comes to roughly half a per cent overall.
• It assumes the effect lasts. Novelty wears off, seasons change, and other changes ship on top. A year is a long time to hold a number still.

Ask for the projection built on the lower end of the interval, scaled to the audience that saw the change, and treat the result as an upper bound for planning rather than a forecast.

8. What else ran this period? A deck that shows three wins should also say how many tests ran in total. If it was three out of four, ask how a win was defined, because published figures are much lower. Kohavi, Tang and Xu cite one third of ideas at Microsoft improving their target metric, and ten to twenty per cent on some measures at Bing and Google.

What to do with the answers

Most decks will pass some checks and fail others, and that’s normal. A missing interval or an unreported guardrail is usually an oversight that a good team will fix the same day. The answers that should make you hold off on a decision are a metric changed after launch, a test stopped early under a fixed-horizon method, an unexplained traffic mismatch, or a winning segment nobody planned for. Any of those leaves the question open, whatever the change turns out to be worth.

Ask the questions the same way whether the result is a win or a loss, and ask them every time. After a few rounds, the decks start arriving with the answers already in them.

Put it to work

Take the last results deck you were shown and run it against the eight checks in this lesson. Write down each one you couldn't answer from the deck itself, and send those as questions before the next meeting.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments, chapter 1
2. Evan Miller, How Not To Run an A/B Test
3. Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (KDD 2019)
4. Deng, Li, Lu and Ramamurthy, On Post-selection Inference in A/B Testing (KDD 2021)
5. Shen and Lee, Selection Bias in Online Experimentation (Airbnb Engineering)