What experimentation is, and what it can tell you

Most companies judge a change like this. They ship it on Monday, watch the numbers for a week, and if nothing looks broken they call it a win. If it dips, they blame the weather, or payday, or whatever’s convenient. Nobody’s lying exactly. There’s just nothing controlling for everything else that moved.

A test fixes that. You run two versions at once, split your audience down the middle, and read what your change did. Same traffic, same week, same weather, same payday, so all that noise lands on both sides equally and cancels out. What’s left is the difference you made. It’s the only reliable way to separate your change from the world it launched into.

It’s the same idea as a drug trial. Nobody gathers the cleverest people in a room and asks whether the medicine works. They give it to one group and not the other, and watch. A website test is the same method pointed at a different problem.

That’s why a test beats the loudest opinion in the room. A strongly held view, and the seniority of the person holding it, tell you nothing about whether it’s true. The highest paid person’s opinion is still an opinion, usually built on second-hand anecdote and a bit of rose tint around our own work. Evidence you can observe, and observe again, is worth more.

Be just as clear about what a test won’t do. It won’t tell you what to build, and it won’t explain why people behave the way they do. It gives you an accurate, tightly qualified read on one specific change. That’s narrower than the brochure promises, and more trustworthy for it.

It gets abused in two ways. The first is the lab coat: grubby nudges and dark patterns wrapped in testing language so they look legitimate, with a number to point at. The second is the universal truth, the idea that you can lift a result from someone else’s case study and it’ll hold for you. It won’t. Your market and product are different from theirs, which is why you test at all.

You’re going to lose more often than you win. The data says most ideas don’t move the metric, even at the best companies. But you’ve always lost more than you won, you just weren’t measuring it. Losing tests show you what was always happening, and if you learn something from each one, they’re worth having.

Put it to work

Take a recent change you shipped and made a call on. Ask yourself whether you could separate its effect from everything else happening that week. If you couldn't, that gap is what a test closes.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Stefan Thomke, Experimentation Works (HBR Press)
2. Ben Goldacre, Bad Science
3. Hierarchy of evidence, an overview
4. The HiPPO, and where it came from