Mining support tickets, chat logs and reviews for test ideas
Your customers have already told you where the site lets them down, in writing, thousands of times, and it's sitting in a system marketing never logs into.
Every business that sells online has a pile of research it hasn’t paid for. Customers write to support when they’re confused, open a chat window when they’re stuck, tell a salesperson what nearly stopped them, leave a review when they’re delighted or furious, and tick a box explaining why the parcel went back. It’s all written by customers, mostly in their own words, at the moment something went wrong. And in most organisations it sits in a helpdesk tool, a CRM and a returns system that nobody on the testing side ever opens.
Each source tells you something slightly different. Support tickets are good at showing what people couldn’t find or didn’t understand, since they’re often written after the website failed to answer a question. Live chat logs are better still for the website, because the chat usually starts on a specific page, and many tools record which one, so you know where in the journey the question arose. Sales call notes, for businesses that have a sales team, record objections, the worries that come up before someone commits, which are often the anxieties the website could have answered earlier. Product reviews tell you what people expected compared with what arrived. Returns reasons tell you where the product page set the wrong expectation, which is a page problem you can test even though it shows up in the warehouse.
The most useful material is usually in the free text. A returns form with six dropdown options will tell you that most items came back because they didn’t fit. The comments box will tell you they came up small compared with the brand’s other ranges, which is something a size guide can address. Ticket categories chosen by the support agent are worth having, but they’re written in the business’s language for the business’s purposes, and they’ll lump together problems that need different fixes.
The method is the one from the qualitative analysis lesson, applied at a larger scale. Take a sample from a recent, defined period, a few hundred contacts is plenty to start. Read a batch first without tagging, to get a feel for it. Then write a short list of tags based on what customers are describing, such as “can’t find delivery date”, “promo code rejected” or “unsure which size”. Tag everything with that list, adding a tag only when nothing fits, and keep a note of what each one means so two people would apply it the same way. Then count.
Counting is what turns a pile of complaints into evidence. Without it, the backlog fills with whichever ticket made someone cross in a meeting last week. With it, you can say that one problem accounts for a fifth of all contacts and another for one in a hundred. Two further steps make the counts more useful. Tie each tag to a stage of the journey, so you can see where problems cluster. And where you can, express them as a rate, contacts per thousand orders for instance, so you can tell whether a problem is growing or the business has had a busier month.
Then the bias. The people who write in are a self-selected slice of your customers, the ones who hit a problem, cared enough to report it, and found a way to contact you. A customer who gave up at the payment page without a word and bought from a competitor leaves no ticket. So support data is good at telling you which problems exist and how they’re described, and poor at telling you how common they are across everyone who visits. Cross-check against the funnel before you size anything.
Reviews have a well-documented version of this. Research by Hu, Zhang and Pavlou found that online product reviews tend to follow a J-shaped pattern, lots of five stars, a cluster of one stars and relatively few in between. Part of the reason is that people who buy a product were already inclined to like it, and part is that people with middling experiences mostly don’t bother writing. Their conclusion was that the average rating is a poor guide to quality. For research, that means a review section overrepresents the delighted and the angry, and the people with a mild, fixable complaint are mostly missing.
None of this makes the data less useful, as long as you read it for what it is. Feedback like this is strong evidence that a problem exists and good material for how customers describe it, which is ideal for writing copy and hypotheses. It’s weaker evidence of how big the problem is. The same applies to research that asks shoppers for their reasons. Baymard’s study of why US shoppers abandon a basket, where extra costs such as delivery and fees top the list, is people’s own account of why they left. That’s valuable, and it’s still a self-report, best treated as a starting point and checked against what your own analytics shows.
So the sound way to use it is alongside the other sources. NN/g calls this triangulation: when the support data, the funnel and the recordings all point at the same step, you can be fairly confident you’ve found something. When support is full of a problem the funnel doesn’t show, it may be a small, vocal group, or a problem that happens after the visit, such as delivery, that your analytics can’t see.
A few practical questions help when you go and get the data. Ask customer service what their top contact reasons were last quarter and how they’re categorised, then ask for the raw text anyway. Ask whether chat logs record the page the chat started on. Ask the sales team for their notes on lost deals as well as won ones. Ask whether returns reasons include free text. If you’re working with an agency, ask whether this kind of data is part of their research, how they sample it, and whether they report counts or just pick out a few good quotes.
The support team is worth talking to as well as their data. They usually know the top five problems already, and they’ve often been raising them for years without anyone on the website side asking.
Put it to work
Ask customer service for an export of the last 300 contacts. Read the first 50 and write a short list of tags from what you see, then tag the remaining 250 with that list, adding new tags only when nothing fits. Count each tag and note which part of the journey it belongs to.
There is no right answer. Write yours first, then compare.
Sources
Shoshin is written by Another Web is Possible. Lessons are licensed CC BY-SA 4.0.Our promise.