Testing beyond the website, from email to operations

Most programmes start on the website because that’s where the tool is. Then someone in a planning meeting asks whether the welcome emails do anything, or whether next-day delivery is worth what it costs, or what would happen if the premium tier cost a little more. All of those can be tested. The method has to move with the channel, and three things change when it does.

What changes in the method

The first is the randomisation unit, the thing you assign to control or variant. On a website it’s usually a cookie, which breaks when someone switches device or clears their browser, so the same person can land in both groups. In email it’s the subscriber, which is cleaner. In an app it’s usually the account. For a delivery promise or a store layout it might be a region, a branch or a week. The rule of thumb is to randomise at the level where the change is experienced and where a customer could notice the difference. Test a price by cookie and the same customer can see two prices on two devices, which is a bad experience and a contaminated test.

The second is the holdout. On a website you’re usually comparing two versions of a page. Outside it, the useful question is often whether something works at all. Email platforms report opens and clicks, and the clicks get credited with the sales, but plenty of those people would have bought anyway. The only way to find out what a welcome series or a monthly newsletter adds is to hold a random share of people back from it entirely and compare what both groups spend. Paid media works the same way at a larger scale. Vaver and Koehler at Google describe geo experiments, where whole non-overlapping regions are randomly assigned to see the advertising or not, and the regions are compared.

The third is interference, where one group’s treatment changes another group’s outcome. It barely matters on a product page, and it matters a great deal anywhere customers share something finite: stock, delivery slots, drivers, appointments, staff time. Give a discount to half your customers and they may buy the stock the other half would have bought, so the control looks worse than it would have, and the discount looks better than it is. Holtz and colleagues measured this in Airbnb pricing experiments and estimated that about a third of the effect measured by ordinary listing-level randomisation came from interference. The usual fixes are cluster randomisation, where you randomise groups that mostly interact with each other, and switchback tests, where a region alternates between treatment and control over time. Bojinov and colleagues set out how to design those. Both leave you with fewer units to compare, so tests need to run longer.

Channel by channel

Email is the easiest place to start, because the unit is clean and the list is yours. Judge a test on what happens after the click, ideally revenue per recipient, and use a holdout to judge whole programmes rather than single sends.

Apps bring release cycles and app store review, and some users stay on old versions for months. Server-controlled feature flags let you change what the app does without shipping a new version. App store listings can be tested too. Apple’s product page optimisation lets you test icons, screenshots and preview videos against your original, up to three treatments, one test at a time, for up to 90 days.

Product features are usually tested with feature flags. Pete Hodgson’s long article on martinfowler.com calls these experiment toggles, which route each user consistently down one code path so the groups stay stable for the length of the test. The same article warns that teams should treat flags as inventory with a carrying cost and remove them once the decision is made. Ask your product team how many flags are live and how many belong to tests that finished months ago.

Pricing is the most consequential test you can run and the one that needs the most care. Randomise by account, keep it short, decide in advance what you’ll do for customers who paid the higher price, and assume customers will compare notes. Price presentation, such as monthly versus annual framing, is often a safer first test than the price itself. When personalisation pays covers the legal side, which applies here too.

Paid media platforms have their own A/B tools, and they answer a narrower question than they appear to. Braun and Schwartz, writing in the Journal of Marketing, showed that platform algorithms deliver each ad to a different mix of people, so a winning ad may have reached more responsive people rather than carried a better message. That’s fine if you want to know which ad performs best on that platform. It’s weak evidence about which message works with your customers in general, so don’t carry a paid social winner over to the homepage as if it were proven.

Offline and operations work, such as store layouts, call scripts, packaging inserts and delivery promises, usually means randomising stores, regions, agents or weeks. You’ll have far fewer units than a website has visitors, so expect to need matched pairs, longer runs and a statistician’s help. It can still beat rolling a change out everywhere and arguing about the weather afterwards.

What to ask before it launches

Whoever is running the test, internal team or agency, should be able to answer five questions. What is the randomisation unit, and could one customer end up in both groups? Could the treated group affect the control group’s results through shared stock, staff or budget? Is there a holdout, and how long will it stay in place? How long does the effect take to show up, and could it linger after the test ends? And who can switch it off, and how quickly?

With clear answers to those, the result will mean what it says. Without them, the test may still run and produce a number, and you’ll have no way of knowing which part of it came from the change you made.

Put it to work

Pick one recurring decision outside the website, such as a welcome email, a delivery promise or a sales script. Write down what you would randomise, what the control would be, and whether the two groups could affect each other.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Pete Hodgson, Feature Toggles (aka Feature Flags), martinfowler.com
2. David Holtz, Ruben Lobel, Inessa Liskovich and Sinan Aral, Reducing Interference Bias in Online Marketplace Pricing Experiments
3. Iavor Bojinov, David Simchi-Levi and Jinglong Zhao, Design and Analysis of Switchback Experiments
4. Jon Vaver and Jim Koehler, Measuring Ad Effectiveness Using Geo Experiments, Google Research
5. American Marketing Association, on Braun and Schwartz, Where A/B Testing Goes Wrong (Journal of Marketing)
6. Apple Developer, Product page optimization