Trusting the results

The frequentist test, and the temptation to peekThe default way to call an A/B test is sound, as long as you fix the sample size and wait.Lesson 1 · 5 minHow many tests should you be running?Test velocity matters, but your traffic and your ability to write good hypotheses put a limit on it.Lesson 2 · 4 minThe Bayesian test, and the prior you forgot to mentionBayesian testing gives you the intuitive answer that frequentist stats won't.Lesson 3 · 5 minCan you run multiple tests at once?Concurrent tests almost never interfere if you randomise properly.Lesson 4 · 4 minSequential testing, or how to peek without lyingSequential testing lets you look early and stop the moment the evidence is real, with the error rate kept in check.Lesson 5 · 5 minHow much traffic a test needs, and what the minimum detectable effect isThe minimum detectable effect is the smallest change a test is built to catch, and it decides how much traffic you need.Lesson 6 · 5 minChoosing what a test is judged on, before it launchesEvery test needs one primary metric, a handful of secondary ones, and guardrails that stop a win doing damage somewhere else.Lesson 7 · 5 minChecking a test is sound before you trust its resultMore bad results come from broken tests than from bad statistics.Lesson 8 · 6 minHow long to run a test, and when to stop itA test needs enough visitors, and it needs them spread across the days, weeks and kinds of customer you'll get once the change is live.Lesson 9 · 5 minNext track: Championing experimentationTalk about testing inside the business, report results honestly, and answer the objections.Track 5