Running your first A/B test

Keep your first test boring. The meme in CRO is button colour testing, but one shade of blue against another teaches you almost nothing, because colour barely moves conversion on its own. Pick a real lever instead. Say research suggested people don’t understand how your delivery works, so you reassure them earlier, on the product page, and see if that lifts purchases. It’s mostly copy, easy to measure at more than one point in the funnel, and it’ll still make sense to talk about in four weeks.

You don’t need to be a developer. The tools have visual editors built so a marketer can iterate without an engineering squad. It helps to know roughly what’s going on underneath: a snippet loads early, asks the tool whether this visitor qualifies, and if so runs the JavaScript that changes the page. That can turn a preview bug from a lost afternoon into a five-minute fix.

Most first tests go wrong before they go live. Targeting fires too early on a page that loads in stages, so the split ends up uneven. Or the goal gets set on whatever’s easiest to track, like a click on the submit button, when it should be the page that only loads on success. Then a failed submission counts as a win. Be precise. Write down exactly what you’re counting as a conversion and how it’s measured, somewhere everyone can see it.

Then QA it. Try multiple browsers, multiple screen sizes, and a real phone rather than a simulator. Submit the form and check it arrives at the other end. And get whoever owns the page to use the variant before launch. They’ll rarely find bugs, but they’ll often spot a brand or tone objection you’d never have thought to flag.

Once it’s live, check the plumbing first: are logs firing for both control and variant, and does the split look roughly even? After that, leave it alone, and don’t read anything into the numbers until the test is powered. If someone asks how it’s doing three days in, tell them it’s healthy and roughly how many days it is from the sample size you’re aiming for.

Call it once it’s powered and has run long enough. Fourteen days is a decent floor, since it gives you two weekends. Then write it up: hypothesis, treatment, metric, confidence, and a plain conclusion of true, not true or inconclusive.

The moment that tests you most is the pull to call an early, underpowered result a win. The incentives all point that way, because businesses reward results more than the discipline behind them. Call wins that aren’t real and the gains never turn up in the sales figures, and people decide CRO doesn’t work. Holding the line keeps your later results believable.

Put it to work

Before you launch your next test, write down in one line exactly what you are counting as a conversion and how it is measured, then check that against what your testing tool is tracking.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.