Building funnels and segments that show where the problem is

Most funnels get built from the site map. Someone lists the pages in the order the designer intended, homepage, category, product, basket, checkout, confirmation, and the tool dutifully reports the drop between each. The trouble is that very few customers take that route. They land on a product page from a search ad, open three tabs, leave, come back from an email two days later and go straight to the basket. A funnel drawn from the intended path measures how many people followed your plan, and that’s a different number from how many people tried to buy and couldn’t.

So the first job is to build the funnel from the journey people take. Start by looking at where sessions that end in a purchase begin, and at what the common paths between entry and payment look like. If half your buyers enter on a product page, a funnel that starts at the homepage is describing a minority. Most tools let you choose between an open funnel, where people can join at any step, and a closed one, where they must start at the first. Google Analytics offers both in its funnel exploration. A closed funnel answers “of the people who started here, how many finished?” and an open one answers “how many people reached each step, from wherever they came?”. You usually want to see both, because the gap between them tells you how much of your traffic skips the front of the journey altogether.

Some steps have no page of their own. A single-page checkout can hide four distinct moments, entering an address, choosing delivery, entering payment, confirming, and if they’re all one URL, the funnel shows one step with a big drop and no detail. Those moments need tracking as events before the funnel can show you which one loses people. It’s worth asking early, because adding the tracking takes a sprint and the data takes weeks to build up.

Then split it. The analytics lesson covers why segmentation matters. The three splits worth asking for first are device, channel and new versus returning, because they’re the ones most likely to separate people who behave differently for reasons you can act on. Typing an address and card number on a phone is harder work than on a laptop, so forms and payment are where the two often part company. Paid social sends people who were scrolling something else, while email sends people who already know you, and the same page will perform very differently for each. New visitors are still deciding whether to trust you, and returning ones mostly want to get on with it. A step that looks fine on average can be broken for one of these groups and propped up by another.

When you read a split funnel, ask for the counts as well as the rates. A segment converting at 1% with 50,000 visitors matters far more than one converting at 0.2% with 300. Rates without counts are how teams end up spending a quarter on a segment that couldn’t move the total if they fixed it completely.

The count matters for a second reason too. Averages don’t only blur segments together, they can point the wrong way. Suppose last month 8,000 visitors came on mobile and converted at 2%, and 2,000 came on desktop and converted at 5%. That’s 260 orders from 10,000 visitors, 2.6%. This month mobile improves to 2.2% and desktop to 5.5%, so both groups are converting better. But the mix has shifted to 9,000 mobile and 1,000 desktop, which gives 198 plus 55, 253 orders, 2.53%. Every group improved and the total fell, because more of your traffic now sits in the group that converts lower.

That’s Simpson’s paradox: a relationship that holds inside every group can vanish or reverse when the groups are added together. The best-known case is graduate admissions at Berkeley in the early 1970s, where men were admitted at a higher overall rate, but department by department there was no bias against women, because women had applied in greater numbers to the departments that rejected most applicants. The arithmetic is the same as the mobile example. The blended number is driven by who ended up in which group, and you only find out by splitting.

In practice, this is why the monthly conversion report causes so many wasted meetings. Conversion dips, someone links it to a release, and a team spends a week investigating a change that made no difference, when the cause was a new paid social campaign bringing in lots of first-time mobile visitors. Baymard’s research suggests around seven in ten online baskets are abandoned, so a shift in who arrives can move the blended rate a long way on its own. Before anyone explains a movement in the total, check whether the mix moved first.

A properly randomised A/B test protects you from most of this, because chance puts the same mix of people in each arm. It stops protecting you if something changes the mix partway through, such as altering the traffic split after launch or a campaign that only reaches one arm. That’s one reason to leave the settings alone once a test is running.

Your part in this is knowing what to ask for, and noticing when it’s missing. The useful questions for an analyst or agency are fairly short.

• Where do the sessions that end in a purchase start, and does our funnel start there too?
• Is this an open or a closed funnel, and what does the other one look like?
• Which steps are hidden inside a single page, and are we tracking them?
• Can I see this by device, channel and new versus returning, with counts next to the rates?
• Has the mix of traffic changed over the period we’re comparing?
• Which segment carries most of the drop-off in absolute numbers, as opposed to the worst rate?

An analyst who’s already looked at these will answer quickly, and one who hasn’t will usually be glad of the prompt. What you’re after is the step and the segment where a fix would move the most customers, and the funnel’s job is to point the qualitative work at that spot. If it points somewhere different once you split it, trust the split.

Put it to work

Ask your analyst for your main conversion funnel split by device and by new versus returning visitors, with the visitor count for each segment alongside the rate. Find the segment whose share of traffic changed most in the last three months.

Your answer

Show the worked example

There is no right answer. Write yours first, then compare.

Sources

1. Google Analytics Help, Funnel exploration
2. Stanford Encyclopedia of Philosophy, Simpson's Paradox
3. Baymard Institute, Cart abandonment rate statistics