Choosing what a test is judged on, before it launches

The readout opens with good news. The new hero banner lifted clicks on the call to action by 18%, it’s significant, and the agency recommends shipping. Three slides later, buried in a table, orders are flat and revenue per visitor is down a little. Which of those numbers decides the test depends on a choice that should have been made before anyone saw any results, and in a lot of programmes it never was.

A test needs three kinds of metric, and they do different jobs.

The primary metric is the one the decision rests on. There should be one of it, written down before launch, and it should be as close to the money as your traffic allows. Orders per visitor, revenue per visitor, completed applications, paid signups. The catch is that metrics closer to the money are usually noisier and rarer, so they need bigger samples. Revenue per visitor varies far more from person to person than whether someone ordered at all, which means it needs more traffic to show a difference. That’s a real trade-off, and the sample size lesson shows how fast the traffic bill grows. Settle it deliberately, in the plan, instead of letting the dashboard settle it for you afterwards.

Secondary metrics explain the result. If orders go up, did more people add to basket, or did the same number add and fewer drop out at checkout? Did average order value move? These are how you learn why something happened, so the next test is sharper. What they shouldn’t do is decide the test. Watch twenty metrics and one will look significant by luck, which is the multiple comparisons problem, and a secondary metric promoted to primary after the fact is almost always that problem in disguise.

Guardrail metrics are things you don’t expect to improve but can’t afford to break. Microsoft’s experimentation team list page load time, crash rate and abandonment as typical examples. For most commissioners the useful ones are page speed, error rates, cancellations, returns, complaints and unsubscribes. What counts as a win covers the longer-running ones such as lifetime value. This lesson is about the mechanics. Each guardrail needs a threshold agreed in advance, the size of drop that stops a ship, otherwise every breach becomes a debate about whether it matters this time.

The overall evaluation criterion

Kohavi, Tang and Xu use the term overall evaluation criterion, or OEC, for the metric a whole organisation agrees experiments are judged against. It’s often a composite, a weighted mix of a few measures. Kohavi and colleagues have long argued that it should not be focused on the short term, with clicks as the example to avoid, and that it should include things that predict long-term value, such as repeat visits.

A Microsoft paper on long-running experiments gives two examples of how a short-term metric can point the wrong way. Make search results worse and people search more, so query volume goes up in the short term while more of them give up and long-term value falls. Raise prices and short-term revenue usually rises while customers drift away.

You probably don’t need a formal composite. You do need a written answer to what the programme is optimising for, agreed above the level of any single test. If the business makes its money on margin, a test that lifts orders by pushing discounted lines may be a loss. If it lives on subscriptions, a signup that cancels in week two may not count. Put that in a page everyone signs, and the primary metric for each test follows from it.

Why clicks mislead

Clicks are cheap to move. Make a button bigger, brighter or more urgent and more people will click it, whether or not more people want what’s behind it. Confusion produces clicks too. Someone who can’t find what they need clicks around more, and a dashboard reads that as engagement.

A click measures interest at most, and plenty of interest never turns into a purchase. Some of it converts and some of it bounces off the next page. When a click metric wins and the downstream metric is flat, the usual explanation is that you moved people one step along and lost them at the next. That can still be useful learning, since the next step is now the place to test. It isn’t a reason to ship.

There’s a tracking version of the same trap. A goal set on a click of the submit button counts failed submissions as conversions, which your first A/B test covers. Count the page that only loads when the action succeeds.

Click metrics have a place, as secondary metrics that tell you whether people noticed the change at all. A redesigned delivery panel that nobody clicks or scrolls to probably didn’t get a fair hearing, and that’s worth knowing before you read the primary.

Decide before launch, and write it down

The metric plan only protects you if it exists before the data does. Once results are in, everyone has a reason to prefer the metric that flatters their idea, and nobody is lying when they do it. Writing the plan down removes the temptation instead of relying on willpower.

When an agency sends you a test plan, ask:

• Which single metric decides this test, and why that one?
• Exactly how is it counted? Per visitor or per session, once or every time, on which page or event?
• Is it sensitive enough to move with our traffic in the time we have?
• Which secondary metrics will you report, and what will each tell us?
• What are the guardrails, and what size of drop stops us shipping?

If the answer to the first question is a click, ask what happens two steps later and whether that can be the primary instead. Sometimes the traffic can’t support it and a closer metric is the right compromise. That’s fine, as long as it’s agreed in writing before launch and the readout says plainly which metric the decision rests on.

Put it to work

Pick a test in your queue and write its metric plan in four lines. The primary metric and exactly how it is counted, two or three secondary metrics, the guardrails, and the size of guardrail drop that would stop you shipping.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments
2. Dmitriev, Frasca, Gupta, Kohavi and Vaz, Pitfalls of Long-Term Online Controlled Experiments
3. Microsoft ExP, Patterns of Trustworthy Experimentation, Pre-Experiment Stage
4. Microsoft ExP, Patterns of Trustworthy Experimentation, During-Experiment Stage
5. Analytics-Toolkit glossary, Overall Evaluation Criterion