What to ask an agency or consultant before you hire them

Agency pitches for experimentation tend to look alike. A row of client logos, three case studies with large percentages on them, a slide about process with a circular diagram, and a confident estimate of what they could do for you. None of that tells you much about whether they’ll find out what’s wrong with your site, run the tests properly and tell you plainly when something lost. The questions below do a better job, and most of them can be sent in writing before the meeting so you can compare answers side by side.

Questions worth asking

How will you come up with ideas for a site you don’t know yet? A good answer starts with research on your customers and your data, and the ideas follow from what it finds. A weaker answer starts with a library of tactics that worked for other clients. Those tactics have no particular reason to work for you. Kohavi, Tang and Xu report that only a third of the ideas tested at Microsoft improved the metric they were aimed at, and the people proposing those ideas knew the product well. See where good test ideas come from.

What research methods do you use, and who does them? Look for a mix: analytics, user testing, interviews, surveys, recordings. Ask to see an anonymised research report and check that the findings lead to the test ideas, rather than sitting in an appendix nobody reads.

How do you decide how long a test runs, and when it stops? They should be able to explain sample size, the smallest effect the test can detect, and their stopping rule, in plain words. Ask what statistical method their tool uses and whether they check that the traffic split came out as planned. If the answer is “we run it until it’s significant”, that’s a problem. See the frequentist test and questions to ask your testing tool.

Can I see a report on a test that lost? This is the most revealing request on the list. A good team has plenty of losses, writes them up with the same care as wins, and can tell you what each one taught them. If all they can show you is wins, either they don’t run many tests or they don’t report the rest.

What share of your tests win, and how do you define a win? Published figures from large experimentation programmes are sobering. Kohavi, Tang and Xu cite one third at Microsoft, about ten to twenty per cent on some measures at Bing and Google, and a director at Slack putting monetisation experiments at about thirty per cent. An agency claiming far more than that may be defining wins loosely, stopping tests early, counting secondary metrics or segments, or testing only safe, small changes. Ask them to explain the gap. Some will have a good reason, and the explanation is worth hearing.

Who owns the learnings, the data and the code? The testing tool account, the analytics access, the research recordings, the test archive and the variant code should all be yours, or handed over in a usable form when the contract ends. If the tool account is in the agency’s name, you can lose years of history the day you part ways. Put this in the contract.

Who will do the work? The people in the pitch are not always the people on the account. Ask to meet whoever will run the research and the analysis.

Red flags

• A guaranteed uplift, or a projected revenue figure in the proposal. Nobody can promise a test result, and a projection made before any research is a sales number.
• Talk of “proven” tests or a library of winning ideas to roll out.
• Case studies with a percentage and no interval, sample size or run length.
• No losing test reports, or an unwillingness to share one.
• Results reported after a few days on modest traffic.
• The testing tool account set up in their name with no handover clause.
• Pressure to sign quickly, or a discount that expires at the end of the week.

Pricing models and their incentives

How an agency is paid shapes what it’s rewarded for, even when everyone involved means well. None of these models is wrong, but each needs a different safeguard.

Monthly retainer. You pay for a fixed amount of capacity each month. It gives the team time to do research properly, and it doesn’t reward any particular result. The risk is drift, months of activity without much learning. Agree what the retainer buys, in research, tests and reporting, and review it each quarter against decisions the work changed.

Per test or per project. You pay for each test or piece of work delivered. It’s easy to budget, but it rewards volume. Watch for tests that are small, quick and underpowered, because those are the cheapest to deliver. See how many tests should you be running.

Performance fee or revenue share. The agency takes a cut of the measured uplift. It sounds aligned with your interests, and the pitch usually says so. In practice the people measuring the result are paid by the result, which rewards generous calls on early stopping, favourable segments and large annualised projections. There’s a statistical problem too. Airbnb’s experimentation team showed that adding up the measured effects of the tests you chose to launch overstates their combined impact, so a fee based on that sum pays out on an inflated figure. If you do agree to this model, fix the measurement method in advance, base it on the lower end of the interval or a holdout group, and keep the analysis independent of the people being paid.

Day rate. Common for consultants. You pay for time, which suits advice, audits and training. It rewards hours, so agree a scope and a finish line.

It also helps to remember what the published big wins look like. Kohavi and Thomke describe a Bing headline change worth more than a hundred million dollars a year in the United States, an idea that sat unappreciated until somebody tested it. Wins like that are rare, and that one surprised the people who ran it. An agency that’s open about that, and still has a clear plan for finding out what works on your site, is usually a better bet than one that promises you a number.

Put it to work

Before your next agency pitch, pick five questions from this lesson and send them in writing ahead of the meeting. Ask for an anonymised example of a losing test report alongside their case studies.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments, chapter 1
2. Kohavi and Thomke, The Surprising Power of Online Experiments (Harvard Business Review)
3. Shen and Lee, Selection Bias in Online Experimentation (Airbnb Engineering)