What counts as a win, and when to ship it
A positive, significant result can still be the wrong thing to build, and a losing one can be exactly what the business needed.
A significant, positive result doesn’t automatically mean you should ship. Most of that decision gets made before the test launches, long before the number comes in and everyone gets excited.
Before you build anything, ask whether the change you’d make if it wins is viable long term. Does it prove something about what the product needs to become, or is it a hack you’d overwrite within a month? Set guardrails before launch too. Some are immediate, such as order cancellations, complaint calls or friction that spikes in the new experience. Others run longer, such as lifetime value, return rate and subscription continuity, and need checking a month or two after the primary metric has been read.
Weigh the cost of making the change against the revenue it’s likely to bring in. An expensive change has to be a very big win to justify shipping, and if it isn’t, the sensible call is often that it’s impractical right now. This matters more than people expect, because small effects need enormous samples, and most businesses don’t have the traffic.
The clearest example of a false win is a shortened lead form. The form says fill this in to get a price, and what arrives is an outbound sales call nobody was told to expect. Form fills go up and sales conversations get worse, because people wanted a low-pressure way to shop and got a phone call. Optimise only for the fill and you spend a lot to gain very little.
A flat or inconclusive result can still be worth shipping. Rebrands are the usual case. Once a business has committed to new branding, a website that doesn’t match costs more than shipping a flat result. Strategy counts too. Moving login earlier might not move purchases, but if it lets you gather persistent data that makes new products possible, a non-inferior result is exactly what you needed.
A real loss has value as well, mostly as money you didn’t spend on a change that would have cost more than it earned. That argument only lands if the business helped come up with the idea. Hand them only validated winners and every loss reads as your failure. Bring them into ideation properly and losses land as what they are, revenue protected.
A big, surprising win is worth getting curious about. A large effect on a properly powered test is usually more reliable. Chase the mechanism. Work out why it moved that much and there’s a good chance the same lever works elsewhere.
People will game whatever metric you give them, so expect it and design around it. Document the hypothesis, the metric and the guardrails before launch, and stick to that document throughout. Medicine handles this with a double blind, where the analyst calling the result doesn’t know which arm is which. Almost no business will pay for that, so most programmes accept more p-hacking and cherry-picking than they’d like to admit.
Hold to the plan you wrote before the test ran, even when the result makes bending it tempting. The business has to be able to trust the process, and once it’s been bent to suit one result, nobody can trust it even when it wins.
Put it to work
Before your next test launches, write down what would stop you shipping it even if the primary metric wins, and what would make you ship it even if the primary metric does not.
There is no right answer. Write yours first, then compare.
Shoshin is written by Another Web is Possible. Lessons are licensed CC BY-SA 4.0.Our promise.