Documenting tests so nobody has to run them twice

Someone in a results meeting suggests putting trust badges next to the checkout button. Someone with a long memory says we tried that a couple of years ago and it lost. Nobody can find it. The person who ran it has left, the agency that built it has moved on, and the deck that reported it is in an inbox nobody can get into. So you test it again. If it loses, you’ve paid twice for one answer. If it wins, you can’t tell whether the customer changed, the design was different, or the first test was broken all along.

That’s the ordinary cost of a programme without a memory, and it compounds. Kohavi, Tang and Xu give institutional memory its own chapter in their book on experimentation, describing a history of every experiment and change that becomes more valuable as a programme matures. Booking.com’s account of how it scaled testing lists a central repository of successes and failures as one of the ways it let knowledge spread across the company. Neither treats it as admin.

What to record for every test

Six fields cover it, and most programmes record two.

The hypothesis, written before launch in the if, then, because form from how to write a hypothesis. Writing it down first also stops anyone rewriting the story once the result is in.

The evidence behind it. Which research, analytics, recordings or previous tests made you think this was worth trying, with links. In two years this is what tells the next person whether the idea came from customers or from the most senior person in the room.

The design. The page, the audience, the split, the primary metric and its exact definition, the guardrail metrics, the minimum detectable effect, the planned duration, and a screenshot of every variant. The screenshots matter more than people expect. “Simplified the form” can mean almost anything once nobody remembers the form.

The result, with its interval. Record the observed difference, the range around it, the sample size, whether the test ran to plan, and any health problems such as an uneven split. A lift of 3% with a range from minus 1% to plus 7% is a very different finding from 3% with a range of 2% to 4%, and only the range tells the next reader which one they’re looking at. A library full of headline lifts with no ranges will mislead whoever reads it, because the small, shaky wins look exactly like the solid ones. Reporting wins and losses the same way covers how to give the range in plain words.

The decision. What you did with it, shipped, binned, iterated or retested, who made the call, and why. This is where you record the flat result you shipped anyway because the rebrand needed it, or the win you didn’t ship because it cost more to build than it would earn. Without the reason, a later reader sees a winner that never went live and assumes someone made a mistake.

The learning. One or two sentences about the customer, separate from the variant. “Variant B won” is a result. “Mobile visitors were put off when they had to create an account before seeing delivery costs” is a learning, and the next team can use it on a different page entirely. If you can’t write this sentence, that’s worth knowing too.

Keep the invalid and abandoned tests as well. A test that broke because the variant didn’t load in one browser tells the next team something about that page, and it explains a gap in the record that would otherwise look like nobody tried.

Tagging so patterns surface

A pile of good records can answer “did we test this?”. Tags let it answer “what tends to work for us?”, which is the more useful question.

Tag every test against a short, fixed vocabulary. Journey stage or page type, device, audience, the lever you were pulling (clarity, trust, price presentation, friction, reassurance and so on), where the evidence came from, and the outcome (win, loss, flat, invalid). Keep the list short and give one person ownership of it. Left alone, a tag list grows “checkout”, “Checkout”, “basket and checkout” and “CO” as four labels for one thing, and every search misses three quarters of the results.

Once the tags are consistent you can ask questions no single test can answer. How often do our trust tests win compared with our clarity tests? Do ideas from usability testing beat ideas from stakeholder requests? Which high-traffic pages haven’t been tested in a year? That kind of look across many experiments is what Kohavi’s chapter calls meta-analysis, and it’s where a library starts paying for itself.

Read those patterns carefully. Five flat tests on one lever may say more about how those five were built than about the lever, and a pattern across a handful of tests is a hypothesis for the next test rather than a rule. Feed it back into prioritisation as evidence, weighted accordingly.

Keeping it findable

The software matters less than three habits. Keep it in one place, rather than split between the testing tool, slide decks and a wiki. Write it in plain language that someone outside the team can search, so the pricing team can find what the checkout team learned, which is most of what scaling beyond one team depends on. And fill it in as part of closing the test, because a quarterly tidy-up never happens.

Your testing tool’s own archive is a poor home for it. It holds the setup and the numbers but rarely the reasoning or the decision, and if you change tools you can lose access to the lot. Export what you need into a place the business owns.

What to ask an agency

If an agency runs your programme, the library should belong to you. Ask where each write-up lives and whether you keep it if you part company. Ask whether every write-up includes the interval and the decision, or only the headline lift. Ask to search past tests by page and by lever. Ask to see the losses as easily as the wins. An agency that can only show you its winners is keeping a portfolio, and you need a record.

Some tests deserve a rerun. Audiences shift, products change, and a result from three years ago on a different site design may no longer apply. A good record lets you say we tried this, this is what we learned, and this is why it might come out differently now, which turns a repeat into a deliberate retest.

Put it to work

Take your last five finished tests and try to fill in all six fields for each one, using only what was written down at the time. Note which fields you couldn't fill.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Ron Kohavi, Diane Tang and Ya Xu, Institutional Memory and Meta-Analysis, in Trustworthy Online Controlled Experiments
2. Raphael Lopez Kaufman, Jegar Pitchforth and Lukas Vermeer, Democratizing online controlled experiments at Booking.com