Judging maturity by the decisions your tests change

Maturity is a word people use to avoid being specific, so let’s be specific. A mature experimentation programme is one where the business learns from its results and acts on what it learned, and the test count has little to do with it. You can run a hundred tests and learn nothing if every result is filed as a verdict instead of understood as a finding, and you can run a dozen and change a company if each one feeds the next. Most maturity models reward the number of tests anyway, even though it’s the wrong measure.

A useful way to think about it is a path from opinion to evidence. At the start, decisions are made by the most senior person in the room, and a test is something the marketing team does to a button colour. In the middle, tests get run, but only to confirm decisions that were already made, the test as a rubber stamp. Later, tests are run to find things out, and the business is willing to be wrong in public. At the end, the business wouldn’t dream of shipping a significant change without a test, and everyone from the founder down argues from evidence rather than from position.

A better question than “how many tests did we run this quarter” is “what did we change because of a result, and what did we stop doing because of one”. Those two answers tell you where you are faster than any framework. A programme that can’t name a decision it reversed because of a test is still early, whatever its test count says. One that can name several is further along than it thinks, because being changed by evidence is the point, and it’s rare in a company.

Getting the placement right matters because the next step is different at each stage, and the step most programmes skip is the one they need. Early on, the next step is building trust rather than running more tests, one straight result at a time, so a single loss doesn’t sink the whole thing. In the middle, it’s making the learning visible, so results stop being filed and start being used. Later, it’s governance, keeping a large programme in check, and spreading the discipline beyond the team that started it. Naming the step for your stage is the point of the exercise.

Watch out for the self-assessment that flatters. Every programme rates itself higher than it is, because the person doing the rating wants a higher score, and the framework that produces a flattering number is the one that gets adopted. The tell is activity standing in for learning, a maturity score built on how many tests were run rather than how many decisions were changed. Judge maturity by the decisions the evidence has changed and you’ll know where you are, which is where the next step has to start from.

Put it to work

Write down the last decision your programme changed because of a result. If you cannot name one, that is your actual maturity, and your next step.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. NN/g, The 6 Levels of UX Maturity