Sequential testing, or how to peek without lying

The frequentist lesson ends on a frustration. You’re told to fix the sample size and wait, but you have real reasons to look early. A losing variant is costing money and, worse, giving real users a worse experience every hour it runs. You want to kill it fast. Sequential testing is the method that lets you, without the inflated error that comes from peeking at a fixed-horizon test.

The idea is simple in spirit. If you’re going to look many times, the bar for “significant” has to be raised to account for all those looks. Two main families do this. Alpha spending designs, the group sequential methods with names like O’Brien-Fleming and Pocock, parcel out your error budget across a set of planned interim looks. Always-valid inference and confidence sequences go further and hold at every single moment, so you can watch continuously and stop whenever the evidence crosses the line. Either way the false positive rate stays controlled, because the threshold already knows how often you’re looking.

The common problem is a cheap imitation. A team says “we run sequentially” but in practice someone is eyeballing an ordinary p-value and stopping on the first dip below 0.05. That’s peeking with better vocabulary, and it carries the same inflated error as any other peeking. A close cousin is stopping the instant a boundary is touched on a noisy guardrail metric the design was never set up to judge. And there’s a subtler one: trusting a vendor’s “stats engine” without ever checking whether it’s properly sequential or just a dashboard that refreshes. Usually it’s well-meaning people under deadline pressure, trusting the vendor.

Doing it properly means being specific. Use a real sequential method with proper boundaries, or always-valid intervals. Pre-specify your interim looks, or use a method that’s valid at all times so you don’t have to. Still set a maximum duration, because the option to stop early shouldn’t turn into running forever in hope. And only stop early on the metric the design covers.

Now the trade-offs, and why all three of these lessons matter together. Sequential testing is efficient and, to my mind, the most ethical of the three, because stopping a harmful or losing variant sooner spares both budget and real people. It gives you the valid early stopping the others can’t. The cost is complexity, a dependence on the right tooling, and a subtler risk: the freedom to stop early tempts teams into deciding too often and too soon. So choose deliberately. Frequentist for simplicity and a clean fixed horizon. Bayesian for intuitive, decision-shaped answers when you’re willing to own your priors. Sequential when looking early and stopping fast matters, and you have the method to do it properly.

Put it to work

Check whether your testing tool offers a proper sequential mode. If it does, use it when stopping early matters. If it does not, hold to a fixed horizon.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Georgi Georgiev, Analytics-Toolkit on sequential testing
2. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments