When personalisation pays, and how to prove it does

The pitch usually arrives from a vendor or a keen agency. One homepage for everyone is leaving money on the table, so show returning customers something different from new ones, show the people browsing tents more tents, and watch conversion climb. Sometimes it does. Often nobody checks, and two years later the business has a dozen experiences running that nobody can switch off, because nobody knows what any of them is doing.

Where it earns its place

Personalisation works when a group of customers needs something the default page doesn’t give them, and you can say what that is. Returning customers don’t need the explainer a first-time visitor does. Trade buyers want account pricing and bulk ordering while consumers want reassurance and delivery dates. Someone who clicked an ad promising free returns should land on a page that says free returns. A logged-in customer with a known order history can be spared a sizing guide they’ve already used.

The question to ask of any proposed segment is what this group needs that the default fails to give them, and what evidence says so. If the answer is that they’re different so they’d probably like something different, that’s a hunch, and it belongs in the backlog as one, scored like any other idea in prioritisation.

What every variant costs

Each segment experience costs traffic. A test aimed at a segment that makes up a tenth of your visitors needs roughly the same number of people as a site-wide test to detect the same effect, and it only gets a tenth of the traffic, so it takes around ten times as long. Many small segments can’t be tested at all in any useful timeframe, which is one of the cases in when a test is the wrong tool.

It also costs upkeep, for as long as it runs. Every experience needs its content refreshing, needs QA when the site changes, needs checking against every new test that touches the same page in case of an interaction effect, and needs someone who owns it. Ten segments across five templates is fifty experiences. A rebrand, a price change or new legal copy now has to be made fifty times, and the one that gets missed is the one a customer screenshots.

So ask the agency or vendor for the running cost along with the launch cost. Who maintains each experience, how will we know when one breaks, and what does it cost to retire?

Testing it properly

The comparison that matters is within the segment. Split the segment’s own visitors between the personalised experience and the default, and compare those two groups. The common mistake is comparing the personalised segment with everyone else. On most sites returning visitors convert better than new ones whatever you show them, so a report saying personalised visitors convert at twice the site average tells you about the segment and nothing about the personalisation. If a vendor’s case study reads like that, ask what the default experience did for the same segment over the same period.

After launch, keep a holdout. A holdout group is a small share of the segment that stays on the default for good, so you can see whether the personalised version still earns its place as the site around it changes. Spotify’s experimentation glossary describes holdouts of 1 to 5% of users, and is open about the drawbacks: a small holdout can be too insensitive to show much early on, the people in it miss improvements, and the default it holds drifts away from the live product, so it needs refreshing now and then.

This matters most for algorithmic personalisation, such as product recommendations. The only way to know whether the model beats a sensible default, such as bestsellers in that category, is to keep some customers on the default and compare. Ask the vendor what the lift is against a holdout within the segment, how it was measured and over what period. A lift against no recommendations at all is a low bar.

The shadow side

Personalisation goes wrong in two ways, and both get worse the more data you hold.

The first is creepiness. A 2015 study in the Journal of Retailing by Aguirre and colleagues found that more personalised ads raised click intent when people knew their information had been collected, and lost that advantage when it was collected covertly, because covert collection left people feeling vulnerable. Using what customers knowingly gave you, their saved size or their last order, reads as service. Using what they didn’t know you knew reads as being watched. A workable test is whether the customer could guess why they’re seeing this, and whether you’d be comfortable explaining it in one sentence on the page.

The second is discrimination, and pricing is where it bites. The CMA’s 2021 paper on algorithms lists personalised pricing among the ways algorithms can directly harm consumers. Section 29 of the Equality Act 2010 stops a service provider discriminating in the terms on which it provides a service, as well as in refusing it. A model fed postcode, device or browsing history can end up charging different prices along lines that track age, disability or ethnicity without anyone choosing that, and the business still owns the result. Any personalised price, discount or credit offer needs your legal team before it launches.

Personalisation is also profiling under UK GDPR, which the ICO describes as automated processing to evaluate things like a person’s preferences or behaviour. Where profiling leads to a solely automated decision with a legal or similarly significant effect, the rules in Articles 22A to 22D apply, along with safeguards such as a way to contest the decision. Those rules were rewritten by the Data (Use and Access) Act 2025, and the ICO consulted on updated guidance in 2026, so check the current version with your data protection lead. Most homepage banners won’t come near that bar. An individually set price or a credit decision might. The ICO also updated its guidance on cookies and similar technologies in April 2026, and whether your personalisation needs consent depends on how it’s built, which is another question for your data protection lead.

Write the segments you won’t target and the data you won’t use into the rules from governance and ethics at scale, before someone finds that a test using them wins.

Then start from one good default for everyone and personalise where you can name the need, afford the traffic, and are willing to keep a holdout running to prove it’s still worth it.

Put it to work

List every personalised experience live on your site or app. For each one, write down the segment, when it was last measured against a holdout, and who maintains it.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Spotify Confidence, What is a holdout group?
2. Elizabeth Aguirre and others, Unraveling the personalization paradox, Journal of Retailing (2015)
3. Competition and Markets Authority, Algorithms, how they can reduce competition and harm consumers (2021)
4. Equality Act 2010, section 29, provision of services
5. ICO, Automated decision-making, including profiling
6. ICO, Guidance on the use of storage and access technologies