Scaling testing across teams without the chaos

Scaling experimentation across teams tends to go wrong in one of two directions. Either the centre holds on too tight, every test waits for the one person who approves things, and velocity dies. Or the centre lets go entirely, and two teams test the same page at the same time and learn the same thing twice, or worse, learn contradictory things and spend a quarter arguing about it. What matters is knowing what to centralise and what to leave alone, and the answer is the opposite of what most people expect.

Centralise the standards and let the teams make the decisions. Every team should agree on what makes results comparable: the same definition of a conversion, the same way of reporting a result, the same rules about when a test is finished, and the same guardrails for the things you won’t do. That’s the shared language that lets one team’s result mean something to another. Without it you have a hundred experiments that can’t talk to each other, rather than a programme.

Leave the ideas and the execution to the teams. The team closest to a page knows its customers and its traffic, and a centre that dictates what to test will keep getting it wrong, because it’s too far from the work. The centre’s job is to make sure a test run in Manchester means the same thing as a test run in London, so that when a pattern shows up in both, someone can see it.

The practical mechanics are dull and important. A shared backlog, so each team can see what the others are testing and two teams stop colliding on the same page. A shared library of learnings, so a finding about a form field isn’t rediscovered every six months. And a shared view of what’s live, because two changes to the same element at the same time won’t isolate, and the team that learns this the hard way wastes a quarter. None of it is glamorous, but it’s what lets the programme grow without scattering.

The other way scaling goes wrong is spreading the activity without spreading the thinking. Every team has the tool and runs tests, and nobody reads anybody else’s results, because the standards were never agreed and the learnings were never shared. It looks like scale and behaves like chaos. The tell is always the same: lots of tests, no shared language, and the same lesson learned over and over by teams that don’t know the others exist. Scaling well means more people testing in a way that adds up, and that only happens when the results can be compared.

Put it to work

List every live and planned test across your teams. Flag any two that touch the same page or element. That collision is the first thing to fix.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Ronny Kohavi, Diane Tang and Ya Xu, Trustworthy Online Controlled Experiments