Questions to ask your testing tool

Every testing tool makes statistical choices on your behalf. Most of them are sensible. A few of them change what a result means, and they rarely sit on the results screen where you would notice.

You don’t need to know which vendor is best. You need to know what yours is doing. Each question below has an answer somewhere in your tool’s statistics documentation or settings. If you can’t find it, ask your vendor or whoever runs your programme. If the answer comes back vague, make a note of that too.

1. Which statistical engine is this test using?

Frequentist, Bayesian, or sequential. Many tools offer more than one, and some let you switch per experiment, so the answer can differ between two tests in the same account.

Why it matters: each engine answers a different question and has different rules about when you’re allowed to read it. See the frequentist test, the Bayesian test and sequential testing.

2. Can I look at the results before the test ends?

If the engine is fixed-horizon (most classic frequentist tests), the answer is no. You plan a sample size, run to it, and read once. Checking daily and stopping when it looks good inflates your false positives. Sequential engines are built for continuous monitoring. Bayesian engines often let you watch, but that isn’t the same as a formal guarantee that early stopping is safe.

Why it matters: more “winners” turn out to be noise this way than any other.

3. If it’s Bayesian, what prior is it using, and can I change it?

Most Bayesian tools use a built-in prior you can’t see or edit. A few let you set your own. Very few let you set it per metric.

Why it matters: on small samples the prior does much of the talking. You should know what it is, even if you can’t change it. If you want a different prior, check whether you can export visitors and conversions per variant and do the sums yourself.

4. What exactly is being counted as a conversion?

A click on the button, or the page that only loads when the action succeeds? Once per visitor, or every time? Over the whole session, or only after the visitor saw the variant?

Why it matters: if the goal sits on whatever was easiest to track, failed attempts can be counted as wins without anyone noticing. See your first A/B test.

5. How does it check the traffic split is healthy?

Ask whether the tool warns you when control and variant receive noticeably different numbers of visitors from what you set (often called a sample ratio mismatch).

Why it matters: an uneven split usually means something is broken, such as targeting firing too early or a variant failing to load for some browsers. A broken test can’t give a trustworthy result, however significant it looks.

6. How does it handle several variants and several metrics?

Some tools correct for the extra chances to find a fluke when you test many variants or watch many metrics. Others report each comparison as if it were the only one.

Why it matters: watch twenty metrics with no correction and you should expect one of them to “win” by chance. See what counts as a win.

7. Does it adjust results using pre-test data?

Some tools reduce noise using how visitors behaved before the test (CUPED is a common name for this). It can let tests finish faster.

Why it matters: it’s a good feature, but you should know whether it’s on, because it changes the sample size you need and makes results harder to reproduce outside the tool.

8. Can I export the raw numbers?

Visitors and conversions per variant, per day, at a minimum.

Why it matters: with raw counts you can check any result yourself, apply your own prior, or settle a disagreement. Without them, you’re trusting a results screen you can’t inspect.

---

Write the answers down once for your tool and keep them with your test documentation. They change rarely, but they do change, usually when a vendor updates its statistics engine, so check again if your vendor announces one.