Choosing a testing tool without being sold one

Most testing tools get chosen in a demo. Someone drags a headline around in a visual editor, a results screen lights up green, and a case study promises a big uplift for a business a bit like yours. The demo shows the easy part. The costs arrive later, in page speed, in consent banners, in a statistics engine nobody on your side can explain, and in a contract you find you can’t leave without losing your history.

This lesson won’t tell you which vendor to buy, and it’s worth being suspicious of any comparison that does without knowing your traffic, your team and your data. What follows are the trade-offs to understand and the questions to put to whoever is selling.

Client-side or server-side

A client-side tool runs a script in the visitor’s browser that changes the page after it arrives. Its strength is that marketers and agencies can build tests without waiting for developers. Its cost is performance and flicker, the moment where the original page shows before the variant replaces it. Vendors offer anti-flicker snippets that hide the page until the variant is ready, which trades the flicker for a blank screen.

REI’s engineering team wrote up what this looked like on their site. Their client-side testing script was blocking the page from rendering, and in lab tests deferring it cut the time to first content by around 60%. The vendor’s anti-flicker snippet added about four and a half seconds on mobile in their tests, and they found flicker was happening in production anyway, because of a race between the testing script and their own front-end code. Their team had assumed the snippet was doing its job until they measured it. Your numbers will differ, which is the reason to measure your own.

A server-side tool decides which version a visitor gets before the page is sent, often through feature flags in your own code. There’s no flicker and you can test things a browser script can’t reach, such as search ranking, pricing logic or an app. The cost is that every test is a code change, so it needs developers and a release process, and old flags need cleaning up once tests end. Many programmes end up with both: client-side for quick changes to marketing pages, server-side for anything structural.

Whichever you use, follow Google’s guidance on testing and search. Don’t show search engines a different version from visitors, use a canonical link on any alternate test URLs, use temporary redirects, and remove test code promptly when the test is done.

The statistics engine

Every tool makes statistical choices for you, and some of them change what a result means. Frequentist, Bayesian and sequential engines answer different questions and have different rules about when you can look. Don’t choose a tool for the name of its engine. Choose one that documents plainly what it does, warns you about broken splits, handles several metrics without inflating your wins, and lets you export the raw numbers. The eight questions in questions to ask your testing tool are the place to start, and if a vendor can’t answer them in writing, make a note of that too.

Warehouse-native tools

Some tools calculate results from data already held in your own data warehouse, rather than collecting their own events. The appeal is that the test can use the same definitions finance uses, such as revenue net of returns or repeat purchase, and there’s one version of the numbers instead of two that never quite agree. The cost is that you need a warehouse in good order and someone who owns the data models, and the results are only as good as that data. Ask where the metric definitions live, who maintains them, and who pays for the queries.

Privacy and performance

Every testing tool stores something on the visitor’s device and sends data somewhere. The ICO updated its guidance on cookies and similar storage and access technologies in April 2026, following the Data (Use and Access) Act 2025, and whether your testing setup needs consent depends on what it stores and what the data is used for. That’s a question for your data protection lead with the vendor’s answers in hand. If consent is needed, your tests only include people who accepted, so it’s worth knowing whether those people behave like everyone else.

On performance, web.dev lists A/B testing scripts among the third-party scripts that can slow a page down. Ask for the script’s size, whether it blocks rendering, and how the anti-flicker option works. Then measure your own pages with the tool on and off, because the vendor’s figures come from the vendor’s test pages.

What lock-in looks like

Lock-in usually looks like ordinary working practice: tests built in a visual editor that only exists inside that tool, years of results stored in the vendor’s database in a format nobody can export, audiences and goals configured in a way that has to be rebuilt elsewhere, and learnings that only live in the tool’s archive. Reduce it from the start. Keep your write-ups in a learning library the business owns, export the raw results for every test as it closes, keep variant code somewhere you control, and know your notice period before the renewal date arrives.

Build or buy

Building your own makes sense when you have plenty of traffic, engineers who already maintain a feature flag system, and tests that mostly happen server-side. The cost is ongoing. Someone has to own the assignment logic, the statistics, the health checks and the documentation for as long as it runs, and an in-house tool can carry a mistake for years because nobody outside ever checks it. Buying gets you something maintained and documented much faster, at the price of the trade-offs above. Plenty of businesses sit in between, using their own flags for assignment and their own warehouse for analysis.

Before you sign, get written answers on how the tool changes the page, what the statistics do, where the data lives and what it costs in speed and consent, and what you can take with you if you leave.

Put it to work

Ask your current vendor, or the one you're considering, for three things in writing. The script's effect on page load, what it stores on a visitor's device, and the format you would get your data in if you left.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Casey Carroll, Catching the Wheel, REI Co-op Engineering
2. web.dev, Loading third-party JavaScript
3. Google Search Central, Minimize A/B testing impact in Google Search
4. ICO, Guidance on the use of storage and access technologies
5. Pete Hodgson, Feature Toggles (aka Feature Flags), martinfowler.com