Bonus content: <cc-checklist-3>42 Personalization Case Studies<cc-checklist-3>
Numbers and words — or quantitative and qualitative research — are two of the most important parts of thorough conversion research. The results are the basis for conversion rate optimization testing.
While locating conversion improvement opportunities through quantitative data is relatively easy and straightforward (simply compare your numerical results to your expected results!), implementing qualitative feedback for better conversions (CRO testing) just isn’t as simple.
Qualitative data isn’t easily compared, or analyzed, or even visualized! Which means there’s a lot of room for error.
It takes some serious interpretation skills to turn wordy answers from surveys and interviews into valuable, actionable insights.
But within these wordy answers lie the reasons your customers are or aren’t converting. It’s the most valuable kind of feedback you can collect.
And there are many, many ways to gather this data, each with its own benefits.
Understand your options for gathering qualitative feedback for proper CRO testing
The goal of qualitative research is to capture the “voice of the customer,” which you can do in several ways:
- Surveys, polls, and queries
- User testing
- Live chat logs
- Support call logs
- User reviews of products and services
- All other forms of direct feedback (social media posts, blogs, forums)
CRO Testing: Seven main types of qualitative research give you a well-rounded picture of your customer’s deepest pain points, desires, and priorities.
All of these methods are useful, but not all are appropriate to use every time. Here’s a rough guide for what method to use when.
1. Surveys, polls, and queries
As a method for getting the most possible responses from your visitors, you can’t beat surveys, polls, and queries. These methods enable you to pose identical questions to a large audience.
The results you get from surveys will be as relevant and actionable as the amount of thought you put into the questions. To get answers that truly reflect your customer’s perception of the website, your questions need to allow respondents to express their opinions freely and without being led.
Surveys come in several different varieties, including exit surveys, consumer surveys, pop-up surveys, and Net Promoter Score surveys. For a detailed discussion on how to conduct surveys, check out our guide.
Once you’ve conducted your survey and collected the results, you’ll have the challenge of interpreting them.
To successfully process this amount of text data, start by isolating the specific keywords that best indicate the issue you’re trying to diagnose.
Once you do that, count the number of times respondents mention that specific issue. This will give a good indication of the severity of the issue.
CRO Testing: Creating a word cloud is a popular technique for evaluating qualitative research.
You may find that you have to create several categories for the most commonly mentioned issues. If that’s the case, order issues by severity so you can tackle them in order when it comes time to A/B-test solutions.
Now it’s time to reread your results and try to hypothesize the best ways to eliminate the issues identified. Sometimes, if you’re lucky, the customers will point you in the right direction.
But don’t count on them to do the problem-solving work — that’s your job. As Henry Ford supposedly said:
“If I asked people what they wanted, they’d have said faster horses.”
Whatever hypothesis you create, be sure to test it first before implementing it permanently.
We know that surveys require a significant amount of effort. This is no excuse not to do them, though. Just be sure to plan and prepare your surveys to get valid results.
Otherwise, all that effort will be for nothing.
Polls serve a different function from surveys. They help to uncover the proportion of people who have specific issues with some aspect of your website.
You’ll use a poll once you have already uncovered an issue and need to decide how to approach the solution.
If your solution is difficult to implement, for example, polling your visitors may help eliminate less successful variants before you even design them, reducing the stress on your resources.
When running a poll, be very confident in the issue you’re addressing to avoid putting words in the mouths of the respondents. With open-ended answers, phrase the question in a way that leads to longer, more detailed responses.
Don’t set up a poll asking, “Are you leaving without purchasing because you don’t trust our site?”. Instead, phrase the question, “What kept you from purchasing today?”
Why polls rather than surveys?
Poll results often lend themselves more easily to comparison and interpretation. Multiple-choice poll responses can be given numerical values, so you can use those to instantly analyze your data.
The best way to use number-based polls is to set up questions with responses on a scale of 1 to 5 or 10, where 1 stands for “Strongly Disagree” and 5 or 10 stands for “Strongly Agree”.
2. Prospect or customer interviews
Interviews are by far the best way to gain insight into your customers.
During interviews, you have the opportunity to discuss their responses and get more detailed answers.
They can serve many useful purposes, from identifying and exploring issues in detail to finding fodder for future marketing copy and customer success materials.
They require a good amount of time and effort, however, when done well, they’re worth their cost and should be high on the list of priorities for qualitative research.
3. User testing
User testing involves recruiting a group of random users to navigate your website and identify common issues. They are usually given specific tasks to complete, and often, their session visit is recorded.
The results, from users’ verbal feedback to their session recordings, can be of great value on the heuristic side of CRO research. Users might just offer you a new view on your user interface, or reveal a problem you hadn’t noticed.
Here’s our detailed guide to user testing, including where to source users and how to interpret their feedback.
4. Live chat logs
If your website maintains a live chat option (and if it doesn’t, it should, then going through your live chat logs can be an enlightening experience.
Chat logs, support logs... Anything from visitors’ complaints to product information queries can indicate areas that need to be improved.
The best part is: you’ve already got these resources.
You do have them, right?
If your website does not have a live chat feature, make it the next priority for your developers. They’re relatively inexpensive and easy to implement, and can make a big difference in conversions.
In fact, research from Kissmetrics and eDigital Research indicates that conversion often doubles for visitors who use live chats while shopping online. Would you miss that chance?.
5. Support call logs & support staff interviews
Recordings of support calls from customers can also provide valuable information that you can use to improve calls to action, fix value propositions, and identify common problems with products or services.
In fact, interviewing support personnel is nearly as useful as conducting customer interviews!
Customer support workers are on the front lines with your customers, and they will have information you couldn't get any other way.
For example, in regular customer interviews, customers tend to be hesitant to express negative opinions because they don’t want to hurt your feelings.
However, they may not be as concerned about the feelings of your customer support staff!
6. User reviews of products & services
This form of direct user feedback is too often ignored, even by large companies. (Only click that link if you’re ready for a horror story featuring toilet paper and terrible support.)
It’s well worth dedicating time to this readily available user feedback. Customers volunteer so much information via their reviews, and you don’t need to organize a survey or conduct interviews to get it.
Ideally, you already have user reviews enabled on your website.
Sometimes stores allow reviews, but nobody writes them.
If this is the case, you may need to incentivize customers to review you, at least at the beginning. If participation is low, carefully check out the review process, and ask customers via email to review the product they purchased after a few days.
CRO Testing: Reviews are a source of invaluable (and often unfiltered) customer feedback.
Some companies use different incentives to compel their consumers to review, such as point-gathering schemes that offer rewards or coupons. Strategies like these don’t cost much to implement and can be very effective!
As with any other form of user feedback, this one is too valuable an opportunity to miss. Gathering consumer reviews is easy, and once the basic mechanism is set up, the only task remaining is to actually read the reviews.
The best way reviews can be used? To improve your copy.
Notice which words your customers use — then use those same words and phrases in your copy.
7. Other forms of feedback
Today, most web users maintain some sort of profile on at least one social network. While you can’t directly track the social network profiles of your customers, it’s possible to track their mentions of your website or the products/services you offer.
Sentiment analysis tools are the quickest way to get this information, and will additionally tell you whether these mentions are generally positive, or negative. You can use this information to keep tabs on how well you’re serving your target audience, to help inform your buyer personas, and to help your writing team “learn customers’ language” for future copy.
You’ll also want to monitor mentions of your brand or service on blogs and forums — particularly professional or niche forums for products/services that do not have a wide target audience.
It’s not the data-gathering that matters for CRO testing!
The most challenging part of conducting qualitative research isn’t gathering qualitative data (although that can be a labor and time-intensive process).
The issue is interpreting and quantifying that data accurately. Whenever words are involved, there is room for interpretation (, which can be flawed).
But when we begin to gather lots of voice-of-customer feedback, patterns emerge that can point us in valuable, conversion-linked directions.
When structuring your plan to gather qualitative data, remember to use more than one method.
No single method will give you a well-rounded perspective on what your customers are really thinking, feeling, and experiencing on your site.
And keep in mind: Your goal for conversion optimization is to find out what customers think about your website — not what they think of your products or services. That’s a whole different game.
Look for feedback that indicates areas of your website that cause friction, anxiety, confusion or distraction. These are the conversion-killers.
Then use that qualitative feedback alongside your quantitative data to form a clear picture of what areas to optimize first, and which hypotheses will stand the best chance of success.
<cc-checklist-3-download>Download 42 Personalization Case Studies NOW<cc-checklist-3-download>