The Bayesian test, and the prior you forgot to mention

The Bayesian approach answers the question most people are asking. Instead of a p-value, you get a posterior, a full distribution of what the true effect might be once the data is in. From it you can say plainly that the probability B beats A is, say, 96%, and you can compute the expected loss, how much you stand to give up if you back the wrong horse. For a decision-maker that’s a far more useful pair of numbers than “p equals 0.04”.

To get there you supply a prior, your belief about the effect before the test ran. Most of the time it should be weakly informative, a gentle nudge that says large effects are rare and otherwise leaves it to the data. A reasonable decision rule might be: ship when the probability B beats A clears 95% and the expected loss sits under a threshold you set in advance.

The thing to watch is that the prior is a dial, and you can turn a dial to taste. Start optimistic and you’ll reach an optimistic conclusion sooner, and on small samples the prior has a big say in the result even if you didn’t intend it to. So pick the prior before you see results, keep it modest, and state it plainly so anyone reading the call can see what went in.

A second trap gets sold as a feature: “It’s Bayesian, so I can watch it and stop the moment it looks good.” That’s tempting, but it’s mostly wrong. Continuous monitoring still needs a disciplined stopping rule, and a confident-looking posterior on a few hundred visitors is largely your prior reflected back at you. Bayesian maths doesn’t change the fact that small samples are noisy. People who fall for this have usually been sold “no more waiting” by tools that gloss over the small print, so it’s an easy mistake to make.

Used carefully, it’s a strong approach. Use weakly informative priors and disclose them. Decide your probability threshold and your expected-loss cap before launch. Gather enough data for the posterior to settle before you act, and don’t act on the first encouraging wobble.

There are trade-offs on both sides. Bayesian outputs are intuitive, they handle small effects gracefully, and the expected-loss framing turns a statistical result into a business decision. Against that, the prior is a responsibility you have to own and declare, the tooling is less of a default than frequentist kit, and the freedom it offers gets abused as a licence to peek. If what you want is valid early stopping, there’s a family of methods built for exactly that, which is where sequential testing comes in.

One last practical note. Whether you can set a prior at all depends on your testing tool, and most don’t let you. To find out how yours handles the statistics, see Questions to ask your testing tool.

Put it to work

For your tool, find out whether you can set a prior. If you can, set a sceptical one and disclose it. If you cannot, note which prior it is using for you.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Chris Stucchio, Bayesian A/B Testing
2. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments