You never have to blame the weather again

Every business already has a way to decide whether a change worked, and it usually goes like this. You launch it, wait a bit, and when someone asks how it’s doing, you reach for whatever variable is lying around. If revenue held, that’ll be the TV advert covering for a downturn. If it dipped, it was hot and people weren’t indoors shopping, so it’ll recover. Nobody’s lying exactly, but nobody can be proved wrong either, so it comes down to making an impassioned argument to whoever is most senior in the room and hoping they believe you. That’s the pain to name first with a stakeholder who has never run a test, because they’ve almost certainly lived it, even if they’ve never put it in those words.

The other version is success by consensus. You ship a redesign, everyone in the room agrees it looks nice, and that becomes the story. But the room liking it tells you nothing about whether it grew revenue, and a team that can’t show it’s contributing to growth, or protecting what already exists, can’t realistically claim to be performing well, whatever the room thinks. The less data you have, the more your performance comes down to how you’re seen, and that’s a shaky basis for pay rises, promotions and budget.

A controlled test gives you a precise measurement of one specific change. The weather, the advert, the marketing spend and everything else are present in both the control and the variant, so they’re held equal, and the difference left over is the change you made. That means you can talk with real confidence about your own contribution rather than the market’s, and argue from evidence instead of volume. You stop having to hope you’ll be believed.

None of that needs a stakeholder to learn statistics. Most people can skip power, p-values and the frequentist and Bayesian debate entirely. What helps is knowing it’s the same method used to find out whether a medicine works, a standardised randomised trial adapted for a website, and trusting that it’s being applied well. Beyond that the mechanic is simple: an old version and a new version, running at the same time, with the difference measured. The one discipline worth explaining properly is that you don’t read a test until it has finished gathering data. Your side of that is learning to answer “how’s it doing” without steering the outcome. Most of the time the answer is “it’s looking healthy, and we expect to conclude on this date”, and that’s enough.

Set expectations on pace and win rate early, and use real numbers rather than comfortable ones. One in ten comes from Stefan Thomke, drawn from a large independent dataset rather than companies reporting on themselves, which tends to flatter. One in five is the figure CRO agencies more often quote in their own marketing, and marketing tends to run a little warm. Give both as an upper and lower bound and say you expect to land somewhere between them. Being that open about it builds trust faster than a confident promise would. It also makes a useful point. The business is already shipping changes at roughly this hit rate without testing, it just can’t say which change worked, can’t roll back the ones that didn’t, and never learns from any of it. What a programme buys you is knowing which change was the hit.

“How long will this take” has a precise answer once you know the sample size needed to detect an effect in a given flow. Bigger samples move faster. Smaller ones need a longer minimum run, so you don’t read too much into a wild swing early on. The practical thing to ask a stakeholder is what they can tolerate. Would six weeks for an answer be acceptable here? You’re trading speed against how much false positive risk you’re willing to accept, and it’s better to say that plainly than pretend the trade-off isn’t there.

Before there’s a single result, trust comes from two habits. First, spread your bets across several treatments instead of going all in on one big idea and overselling it. You don’t control the outcome, so staking your reputation on one test turns the result into luck. Second, be precise about how confident you are. Say something like, I believe this might be true, this is the pattern that suggests it, and we’re going to test it. You’re not asking anyone to take your word for it. Over time that gets a stakeholder comfortable exploring an idea without having to be fully convinced of it first, and that comfort is most of what a healthy testing culture looks like.

Ownership is harder, because running the test is specialist work. It’s closer to having a cabinet made by a carpenter than co-writing a document, and the craft happens somewhere the stakeholder can’t follow. So bring the collaboration earlier. Share research and past learnings, then ideate on the problem together before there’s a treatment, so the eventual test has something of theirs in it. When it works, share the credit plainly, the business made more money and we learned this together, rather than taking a victory lap for one clever idea. That same generosity makes a loss easier to take later, because nobody feels it happened to them.

It also helps to be able to explain why you can’t just ask people what they want. With a handful of customers you can, phone them up and ask. Beyond that, what people say they want and what they do drift apart almost every time. They say they’d like it cheaper, they describe themselves in a flattering light, and they rarely volunteer the less generous truth about their own choices. An experiment is a scaled observation of what people do in a real environment. It’s asking people what they want, just more reliably than asking ever could.

This is also where trust gets abused, and the sales process is usually more to blame than the method. Some practitioners quote inflated win rates and play the researcher to close a deal, then deliver generic best practice once the contract is signed, a kind of cosplay of science that borrows the language of rigour without the discipline behind it. It works because it sounds appealing to someone trying to buy a commercial outcome. Over time the cost lands on the discipline as a whole, because the business on the other end rarely separates one bad practitioner from CRO itself.

The technical side of this work is the easier half. Keeping a room’s trust while you explain it, staying optimistic without over-promising, sharing credit and taking setbacks together, is harder, and it’s the part that builds over time. At some point you’ll be tempted to shade a number to keep someone happy. Hold the line. Clients will come to you with burnt fingers from somebody who didn’t, and they’ll trust you because you did.

Put it to work

Next time someone asks how a test is doing, answer with exactly two facts, healthy or not, and the date you expect to conclude, and see how much better that lands than a guess at the outcome.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.