How to make the business case for an experimentation programme

The usual business case for testing goes like this. Take the conversion rate, assume a programme will lift it by some percentage, multiply by annual revenue, and put the result next to the cost. The return looks enormous, the budget gets approved, and twelve months later someone adds up the wins in the results decks and asks why revenue didn’t move by the same amount. The programme spends its second year defending itself against a number it never had a chance of hitting.

A case that survives that review is built differently. It costs everything, it values more than wins, and it plans on the win rates that large programmes publish.

Cost the whole programme

Most cases count the tool and an agency fee and stop there. A fuller list:

• People. Someone to run the programme, someone to do research, someone to do the analysis, and design and development time to build variants. In a small business these may be parts of two or three people’s jobs, or an agency. Either way, cost the hours.
• Tools. The testing platform, analytics, session recordings, a user testing or survey tool. Get real quotes, because pricing often scales with traffic.
• Development time after the test. A test is often built as a quick front-end change. Making a winner permanent, properly built into the site, can cost more than the test did. Leave this out and winners queue up unshipped, which makes the programme look as if it isn’t delivering.
• Stakeholder time. Sign-off, results meetings, ideation sessions. It’s real time from senior people and it belongs in the cost.
• Traffic. A test uses visitors you can’t use for anything else in that part of the site while it runs, which limits how many things you can try in a year. See how many tests should you be running.

Value it as better decisions

The business already changes its website. The argument for a programme is that it finds out which of those changes help, and the published evidence says most don’t. Kohavi, Tang and Xu report that only a third of the ideas tested at Microsoft improved the metric they were designed to improve, with ten to twenty per cent on some measures at Bing and Google. A business that ships changes without testing is getting roughly that hit rate already. It just can’t tell which change was which.

So there are at least three kinds of value to put in the case.

Wins found and shipped. The obvious one, and the one to be most careful with, for reasons below.

Losses avoided. A change that would have shipped anyway, tested first and found to hurt. The money it would have cost stays in the business. This is the easiest kind of value to forget, because nothing visible happens.

Bigger decisions made better. Kohavi, Tang and Xu describe a strategy at Bing, integrating social media into search results, that cost more than twenty-five million dollars over about two years before it was dropped for showing no significant impact on key metrics. It was experiments that showed the strategy wasn’t working, and the sooner a business looks at results like those, the less a strategy like that costs it. The same authors point out that many organisations measure delivery as the percentage of the plan completed, without checking whether anything shipped had an effect. A programme gives you the other measure.

If you can, name a real decision from last year that a test would have changed. One local example usually does more for the case than any industry figure.

Plan on realistic pace and win rates

Work out how many properly powered tests your traffic supports in a year, then apply a win rate of ten to thirty per cent, and build the case around the bottom of that range. If the programme only makes sense at the top of it, the case is too thin. Expect the wins to be small. Kohavi, Tang and Xu note that improvements to key metrics usually come from many small changes, in the range of 0.1 to 2 per cent each. A plan that assumes a run of double-digit lifts is planning on luck.

The occasional large win does happen. Kohavi and Thomke describe a change to how Bing displayed ad headlines that raised revenue by twelve per cent, worth more than a hundred million dollars a year in the United States, and it had sat in the backlog for months because nobody rated it. Mention it as a reason to keep testing ideas people undervalue, and keep it out of the forecast.

Why summed wins overstate the return

Adding up the projected value of every winning test gives a total that runs high, for a few reasons.

• Selection. Tests get called wins partly because their measured result came out high, and part of that is luck. Airbnb’s experimentation team showed that summing the measured effects of the tests you chose to launch overstates their combined impact, even when every one was a real improvement. Researchers at Microsoft and Airbnb describe the same bias wherever results are filtered down to the significant ones.
• The headline instead of the range. Most projections use the best guess. The lower end of each interval gives a much smaller total.
• Overlap and decay. Two wins on the same page don’t always stack. Effects can fade as novelty wears off, and the rest of the business keeps changing around them.
• Audience. A lift measured on one page or one device has to be scaled down to the share of customers who see it.

The better approach is to keep a small holdout group, a slice of visitors who don’t get the shipped winners for a set period, and compare them with everyone else. It costs a little traffic and gives you a measured total instead of a summed one. If a holdout isn’t practical, report the summed figure as an upper bound, show the version built from the lower end of each interval beside it, and say which you’d plan on.

Ask for a pilot, twelve months is a reasonable length, measured on tests run to completion, decisions changed and, if you can manage it, a holdout. A case that asks to be judged on things the programme controls is far easier to keep making in year two.

Put it to work

Draft a one-page business case for your programme, or the one you want, with a full cost line, a low case built on a ten per cent win rate, and at least one decision the programme would have changed last year.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments, chapter 1
2. Kohavi and Thomke, The Surprising Power of Online Experiments (Harvard Business Review)
3. Shen and Lee, Selection Bias in Online Experimentation (Airbnb Engineering)
4. Deng, Li, Lu and Ramamurthy, On Post-selection Inference in A/B Testing (KDD 2021)