Reporting wins and losses the same way

The report is where a testing programme keeps or spends its credibility, and most programmes don’t notice how much they’re spending. The win gets a deck and the loss gets a sentence at the bottom of an email. The headline number gets quoted without the range behind it. A small effect on a shaky sample gets the same confident language as a large one on a solid sample. Each of those is a small withdrawal, and they add up until nobody in the room quite believes the next result, even the good ones.

The fix is boring, and that’s fine. Use one template for every result, filled in the same way whether it won or lost: hypothesis, what you changed, the metric, the result, the range, the caveats, and what you’d do differently next time. The point of the template is that the win and the loss look alike on the page, so nobody has to read the tone to work out which one it was. People trust a report that reads the same either way, because it shows you’d have told them just as plainly if it had lost.

The range is the part everyone wants to leave out, and it’s the part that matters most. A result is a number with an interval around it, and a lift of eight per cent with a range of one to fifteen is a very different thing from a lift of eight per cent with a range of seven to nine. Report the headline alone and you’ve used the language of certainty for something that isn’t certain. Report the range and the stakeholder has what they need to make a decision.

Keep the result and the story apart, too. The test showed X. Why it showed X is a separate claim, and it’s usually a guess. A drop in signups might be the new form, or the season, or the test itself might be badly targeted. Say which part you measured and which part you believe. A report that blurs the two is asking the reader to believe the guess because the number was real, and the template is there to stop you doing that.

Spin is the thing to watch for, and most of it is small. The win reported with the range stripped out. The loss reported against a different metric from the one agreed. The inconclusive result written up as “we learned a lot” without saying what. None of it is quite a lie, which is why it’s dangerous, because everyone in the room gets to feel they’ve been straight while the truth slips away. So write the loss with the same care as the win, and treat the report as part of the product. A programme that reports straight keeps the trust it needs, and one that spends it on spin eventually finds there’s none left.

Put it to work

Take your last test result and write the one-paragraph version: hypothesis, result, range, and what you would do differently. Check whether a loss would fit the same template.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Stefan Thomke, Experimentation Works