How to brief a testing team so the work answers your question

The brief usually arrives as a single line. Can you test a video in the homepage hero, the managing director saw one on a competitor’s site. That’s a solution, and a team handed a solution will build it, run it and report on it. Odds are it loses or comes back flat, because most ideas do. Kohavi, Tang and Xu report that only a third of the ideas tested at Microsoft improved the metric they were designed to improve, and in well-optimised products like Bing and Google the success rate on some measures is nearer ten to twenty per cent. When the video loses, nobody has learned much, because nobody wrote down what problem the video was meant to solve.

A brief works better as a description of the problem and the rules of the game. You know the business, the politics and the constraints. The team you’re briefing knows how to design and run a test. The brief is where those two kinds of knowledge meet, and it covers seven things.

1. The business goal. What the business is trying to move, and what decision the test should help you make. “More first-time buyers coming back for a second order” gives a team something to work with. “Improve the checkout” leaves them guessing.
2. The evidence you already have. Research, analytics, support tickets, survey comments and previous tests, including the ones that lost. Without it, most teams start from a list of what worked for someone else. See where good test ideas come from.
3. Constraints. Legal and brand rules, pages nobody is allowed to touch, what the platform can’t do, release freezes and peak trading periods. Say them early, because a winning test you’re not allowed to ship is wasted traffic.
4. Metrics. The one primary metric you care about most, as close to real value as your traffic allows. Ask the team to tell you plainly if your traffic can’t support it, because a test sized to detect a lift you’ll never see is a test that can only come back flat. The minimum detectable effect is the number to ask about.
5. Guardrails. What must not get worse while you chase the primary metric. Refunds, cancellations, complaint calls, unsubscribes, page speed. Kohavi and his co-authors describe guardrail metrics as identifying what the organisation is not willing to change, which is a good test of whether you’ve listed the right ones. See guardrail metric.
6. Decision rights. Who signs off a test before launch, who can stop it early and on what grounds, who decides whether a winner ships, and who gets told the result. Write down that a test runs to its planned end unless a guardrail breaks. Most briefs skip this part, and it causes most of the arguments later.
7. Timeline. When you need an answer and why. A budget round or a relaunch date is useful context. Let the team tell you whether your traffic makes that date possible, and believe them if it doesn’t.

What to leave to the specialists

The treatments themselves, the sample size and run length, the statistical method, the number of variants, the QA and the build. Ask them to explain any of it, and you should, because you’ll be defending the result later. Specifying it yourself tends to get you the test you asked for, which may be different from the test the problem needed.

It helps to think of it like briefing an architect. You tell them how many people will live in the house, what the budget is, what the planners won’t allow and when you need to move in. You don’t tell them where the beams go. If you have an idea for a treatment, put it in the brief as one option with the problem you think it solves attached, so the team can weigh it against others and the result teaches you something either way.

Common briefing mistakes

Briefing a solution with no problem attached. “Test a sticky add to basket button” is fine as a starting idea. “People on mobile scroll past the button and we think they lose track of it” is a brief, because it can be tested several ways and a loss still tells you something about the problem. A hypothesis is the format for this.

Asking for a win. “We need a ten per cent uplift by Q3” tells the team the only acceptable outcome is one they don’t control. It also rewards anyone who reports generously. Ask for an answer by a date instead, and make it clear a well-run loss counts as delivery.

Hiding the politics. If a senior person loves the idea, say so in the brief. It changes how the result needs to be reported and how much care the setup needs, and a team that knows about it can plan for it.

Twenty metrics and no primary. Watch enough metrics and one of them will move by chance. Pick one to decide on, a few guardrails, and treat everything else as background. See multiple comparisons.

Leaving out what’s been tried. Agencies change and people leave, and old losers get rerun because nobody wrote them down. Include the archive, even if it’s a spreadsheet.

Forgetting the cost of shipping a winner. A test is often built as a quick front-end change, and making it permanent can take months of developer time. Say up front what development capacity exists after the test, so the team doesn’t design something that wins and then sits in a queue.

A useful check before you send it: imagine two different teams reading your brief and coming back with different treatments that would both answer the same question. If they could, you’ve described a problem. If they’d both come back with the video in the homepage hero, you’ve written an instruction, and you’ll get exactly what you asked for.

Put it to work

Take the next thing you plan to hand to an agency or internal team and write a one-page brief under the seven headings in this lesson. Mark every heading you couldn't fill in, because those are your first questions for the team.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments, chapter 1
2. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments