The core terms of testing, in plain English

Most businesses already have a way to judge a change. They ship it on Monday, watch the numbers for a week, blame the weather if it dips and call it a win if it doesn’t. Nothing in that controls for anything else that moved, so the data will back up whatever you already wanted to believe. Testing vocabulary exists to replace that guesswork.

It borrows from drug trials, and the words are generic enough to describe a medicine as easily as a checkout. The old, unchanged experience is the control. The new thing is the variant. You run them at the same time, to the same kind of visitor, so the change itself is the only thing that differs.

Conversion started as the word for turning a visitor into a buyer. Now it means any action you’ve decided in advance counts: a sale, a signup, an upgrade. Decide what counts before the test runs and before you’ve seen any numbers.

Lift is the measured difference between control and variant. On its own it says nothing about whether to trust it, and that’s the part people skip, because lift is the headline number and headlines feel like the whole story.

Trust is what the rest of the words are for. Significance tells you how confident you can be that the change caused what you saw, rather than random chance and noise. A confidence level is the same idea from the other side: at 95% you’ll still get a false reading one time in twenty. Power is the one a lot of programmes get wrong, and it matters most. It’s your chance of catching a real effect if one exists. Run underpowered and good ideas get killed, because “not significant” gets read as “didn’t work” when there was never enough traffic to see it either way.

This is also where the vocabulary gets bent. Someone quotes a lift with no sense of the range behind it. Someone stops a test the moment it crosses a threshold. Others skip measurement entirely and claim every change as a win. Each borrows the authority of the words and skips the checks behind them.

The fix is mostly patience. Run to power before you say anything, and report the range alongside the headline. Then you can say something nobody else in the room can: it wasn’t the weather or payday, it was the change, and this is how much of it you can trust.

Put it to work

Next time someone reports a test result to you, ask two questions before you accept it. Was this run to full power, and what does the range behind the lift look like, as well as the headline number?

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.