What sits behind the common objections to testing

Objections to testing come in a few familiar forms. “We already know what works.” “Just ship it, we don’t have time for a test.” “Tests take too long and we have targets to hit.” They sound like objections to the method, but underneath there’s usually something else: a fear of being wrong, a fear the answer will be uncomfortable, or someone who got burned by a badly run test and has generalised from it. If you argue with the surface you tend to lose. Answer what’s underneath and the objection often stops mattering.

“We already know” is usually a claim about authority, and the person saying it often can’t point to the number they already know. Arguing won’t help. Ask them, gently, what they’re basing it on, and offer the test as a way to find out whether the thing they know is true. You’re offering to check whether they’re right, which makes the test their ally rather than their critic. A senior person who believes something strongly is a good source of hypotheses, and treating them that way changes the conversation.

“Just ship it” is harder, because it’s often right. Some things aren’t worth a test, and insisting every change needs one is how testing programmes make themselves look ridiculous. Shipping and testing can happen together. Ship it, but to half the traffic first, or behind a flag, so the thing you were going to do anyway produces a result as well as an opinion. The test adds a small cost, and in return it tells you afterwards whether the shipping worked.

“Tests take too long” is a question about traffic and effect size, and it deserves a straight answer rather than reassurance. A test takes as long as the sample needs, and if your traffic is small, that’s a long time. Name the trade-off plainly. What could we detect in six weeks, and what would we have to accept to get an answer faster? People respect a clear limit far more than a promise that everything will be quick, because that promise is usually what burned them before.

Where this goes wrong is answering the objection by over-promising. You win the argument by saying the test will be fast and cheap and almost certainly a winner, then it’s slow and expensive and loses, and the next objection is harder to answer than this one was. It works better to treat objections as information about what the person is worried about. Someone who says tests take too long is usually worried about their targets. Someone who says we already know is worried about their authority. Address the worry and the objection tends to go with it.

Put it to work

Take the last objection you heard and write down, in one sentence, what the person was worried about underneath it. Then answer that.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. NN/g, Successful Stakeholder Relationships