Customer Research as an Enhancer for Ecommerce CRO Efforts

Customer research guide covers all aspects of research needed to understand your customers. Customer research plays a huge role in conversion optimization. Use customer research to find out information about your customers’ behavior that takes you beyond what they did and where they came from.

Chapter 1

How to Use Customer Research to Amplify Your CRO Efforts

Note: This is Article 1 in a series that will examine the process, tools, and analysis of performing customer research for conversion rate optimization. Stick around for following articles!

Because conversion optimization deals with improving website performance, and because websites are primarily technical constructs, it’s easy to lose sight of one simple fact: websites exist to draw customers, and customers are real people.

All of the traffic reports we see in Google Analytics (and similar tools) represent actual human beings visiting a site and trying to figure out whether to buy anything or not.

We can get overly absorbed in the technical details of tracked events, goals, conversions, and other more exact and numerically expressed indicators. All these quantified data points are easy to interpret — the data can’t lie, right? — and sometimes we can neatly, gratifyingly fit our findings to our theories.

However, we’re frequently surprised by the findings of the analytical tools that we’ve taken great care to configure. That’s because our visitors are human — so no amount of statistical data we collect will ever be enough to completely explain their behavior.

Using purely quantitative data, the “why” of our users’ observed behavior remains just outside our reach. Sometimes, users act in ways we cannot easily (or at all) explain.

For example, imagine you’re trying to solve the issue of high cart abandonment on your product landing page. While analytics and quantitative data will tell you how many of your visitors leave without completing their purchase, you will remain in the dark as why they abandoned their purchase.

The main objective of customer research, therefore is to answer the question of the “why,” and provide you with a central piece in the overall CRO puzzle: the actual face of your customer.

We go a lot more into how to set up effective A/B tests in some of our other posts, but it’s important to keep your tests scientific by only changing one element at a time.

For instance, if you’re trying to figure out if adding an interstitial helps or harms your sales page conversion rate, don’t also change the copy and button colors at the same time. You won’t be able to tell which change caused the uptick or drop in conversions.

It takes some patience, but isolate exactly what you want to improve and change one thing at a time.

Qualitative research plays a huge role in conversion optimization

We use qualitative research to find out information about our customers’ behavior that takes us beyond what they did and where they came from.

This information is important in more ways than one. Not only do you get to know what people think and feel about your website firsthand, you can also understand the story underlying the dry data.

While quantitative data often has its own “story” to tell, qualitative research has the advantage of being directly attributable to every business’ most important consideration: its customer. By citing the customer, you add more weight to any insight you may have.

Remember our landing page cart-abandonment example from before? When you perform qualitative research, you no longer have to say things like, “According to the traffic data estimates, navigation patterns, and places of exit, we think that the prospect left because they lacked trust in our payment processing provider.”

Instead, you can simply point to the number of real-life customers who say that they’d like to see PayPal as a payment option.

Qualitative research allows you to devise solutions to problems earmarked by quantitative research, without just guessing what your users want. It reveals a direct route to improvement. By just following your customers, you can match their expectations and meet their needs more closely — and close more sales.

The purpose of qualitative research is clear. So how do we go about doing the research itself?

Methods of qualitative customer research

The obvious answer to the question of how to gather qualitative data: ask your customers directly. There are multiple different ways to ask the right questions the right way.

First, though, let’s list the main sources of qualitative research:

  • Surveys and questionnaires
  • Direct interviews
  • User testing
  • Customer support interactions
  • Customer reviews of products offered on webpage
  • Social or forum posts

All these methods deliver insights obtained directly from customers or prospects. What combination of them to apply largely depends upon the purpose of your website. For example, ecommerce websites can generally glean the most valuable insight from sending surveys about their checkout process and placing polls at friction points. If you can identify why customers are unwilling to complete the purchase process, you’ll be near the solution.

The trick with any qualitative research is to conduct it in an unbiased way, collecting users’ genuine opinions without putting words in their mouths.

Consider the wording of these two questions, which both aim to understand why a customer failed to complete a purchase:

“What problem do you have with the website that is stopping you from completing purchase?”


“Do you find our website credible?”

In the first example, the customer is being asked to identify the problem in their own words. In the second example, we are already planting an idea in their head.

How customer research benefits your CRO efforts

Properly conducted customer research offers multiple benefits. The first and most tangible is the ability to get a clear idea of what bothers your visitors. By understanding your main friction points, you can hypothesize and test solutions, all the while knowing exactly what your visitors and customers expect.

Plus, once you get to know your customers and what they think of your website, you can create user personas with much more accuracy and even potentially predict the behavior of your existing and new visitors.

The advantages and possibilities of this deeper customer understanding are numerous:

1. You can better map out your marketing efforts
By knowing who your customers are, you can find out where they hang out, what social networks they use, and what message is likely to grab their attention and bring them to your ecommerce store with intent to buy a product. You can create campaigns and landing pages that will cater to their specific needs. You can also identify the search keywords and channels most likely to result in bringing your target audience to the site.

2. You can plan for future revenues
If you can accurately track what your regular customers buy, when they do it, and how much they spend, you can identify patterns and create predictions that relate to similar, unknown customers. This is one of the primary benefits of developing personas, which we’ll touch on in a bit. Once you can predict customer lifetime value, you can accurately plot spending and the resultant revenues.

3. You can predict costs
As a mirror image of revenue prediction, you’ll be able to predict and reduce your costs. Along with adjusting and perfecting your marketing strategy and ad spends, you can optimize your inventory to fulfill the needs of your customers.

4. You can start to personalize your user experience
The next logical step up from customer research? Personalization. The more you know about your customers, the more familiar the user experience you can offer them. It’s a well-recognized and thoroughly researched fact that personalization increases the likelihood of conversion and causes visitors to spend more on average. There is really no excuse to pass up this opportunity.

Speaking of which, what *is* personalization?

Personalization: The perfect child of CRO and customer research

Personalization means tailoring user experience to match users’ expectations and preferences. It’s not just limited to content — personalization can apply to design and interface elements, too, allowing the user to adjust website display and more to match their preferences.

The practice of personalization results first in increased brand recognition, as visitors naturally like your website more when it adapts to their preferences. And as we all know, if visitors like you, they’re more likely to convert.

Your visitor loyalty will increase, and you’ll be able to ask favors of your users — like answering surveys or polls to provide more information about themselves, which allows you to further increase your site’s personalization. This process is a positive feedback loop that eventually becomes self-perpetuating.

Plus, increased knowledge of your visitors (a natural byproduct of personalization) also allows you to present different content and offers that fit customers’ needs. You can tweak product offers, campaign messages, and even marketing channels to increase the number of visitors you bring in, while knowing they’re more likely to convert.

Personalization naturally leads to better engagement and an increased perception that your content is relevant and attractive.From there, you’ll see increased conversions, since visitors find your offer more attractive.

There are many examples of successful use of personalization in conversion research, but Amazon’s example stands out. They were able to reach $544 in average annual revenue per user by leveraging personalization according to Business Insider and their Prime members spend over 1.000 US$ a year..

What follows from more conversions? Well, more opportunities to cross-sell and upsell, of course — yet another example of a positive feedback loop started by increased personalization. And the more you know about your customer, the more relevant your offers will be, which increases the likelihood of additional purchases…

There’s just one major caveat.

Personas must be based on real data

Personalization will only work if you have actual information about your visitors. This is easy if your site requires user accounts, wherein customers register for a login that they can use to shop or access the site as a uniquely recognized customer. This is an ideal situation for personalization.

However, this ideal is often hard to achieve. Users may not be willing to give you all the personal information necessary to establish an account. Establishing this sort of trust takes time, and the trust process cannot be hurried.

In the meantime, you can start personalizing on a more general scale. Using analytics, the patterns of behavior or origin of individual users can be tracked, and they can be grouped into personas according to specific criteria. Group customers according to specific parameters, such as age, gender, geolocation, marital status, and other demographics that are interesting or relevant depending on the nature of your business.

Using tools like Tag Manager, it is possible to not only identify behavior patterns, but to assign specific attributes to individual users. When we gather sufficient data, we can sort new users into the right persona purely based on behavior.

These basic “personas” will allow you to identify a few distinct customer types that represent larger groups. Using this small sample, you can offer a more personalized experience to new visitors as they show up and begin exhibiting behavior that you’ve tied to a certain persona.

As a rule, you’ll need a relatively limited number of personas. Any more than 10 will most likely be too many. The point is to generalize visitors and reduce the number of different solutions for personalization.

Article recap & what’s coming next

Customer research is essential for ecommerce, as it creates unique opportunities to increase the relevance and attractiveness of your offers. Whether you deal with customers or other businesses, knowing more about who’s buying makes the process of selling easier. Customer research is mostly conducted through direct contact with customers or prospects.

Once you’re in the habit of conducting customer research, you’ll be able not only to make more relevant offers, but to further personalize the customer experience and create an environment where your customers enjoy themselves. This enjoyment will encourage customers to return more often, spend more money, and like your brand more.

Creating personas is the initial step of personalization, and you can do it with a relatively limited amount of customer research. Personas allow you to address the different needs of different groups of customers, recognize patterns of behavior, and emphasize the content or products in which a given group is interested.

Plus, using personas, you can extend personalization not only to the customers you know and recognize (AKA registered users) — but to new users and returning users who never registered.

In the other articles in this guide, we’ll go more in-depth on how customer research can be conducted, how to define personas, and how to use analytics to identify personas.

Chapter 2

User Testing: Why You Should Be Testing Your Website And How to Start

Usually when we talk about testing in conversion optimization, we’re talking about testing hypotheses: “If we change X, we think Y will happen.”

But there’s another type of testing involved in CRO that gets far less press: user testing.

What is user testing?

No matter the product or service you sell, the best way to see if your business is actually viable is to expose it to your target audience and see how they react. Often, this process can reveal unexpected insights or offer helpful guidance to the seller.

It’s why developers deploy “beta” editions of software before launching their product, so real users can catch the bugs before the product hits the marketplace.

Websites are no different. Designing a website always involves making assumptions about what visitors will expect to see and to do on the site. And no matter the amount of research and experience they’re drawing from, designers and web developers can’t predict 100% of site visitors’ actions and expectations.

Only a trial audience of real users can pinpoint potential issues, friction spots, and other problems with a website. They’ll come up with issues you didn’t even realized your site had!

By observing how visitors actually navigate your website, you can accomplish several important goals:

  • Validate or invalidate your initial assumptions
  • Uncover unforeseen benefits of your products or new ways to approach your content
  • Find flaws and issues that interfere with visitors’ use and enjoyment of the site

Remember the old engineering maxim: “You can make anything foolproof, but the fools are so ingenious.”

User testing can save you from rolling out costly mistakes

The best example of why user testing is crucial? Imagine your ecommerce site is poised to adopt new shopping cart software. Any hiccup in the shopping cart checkout. flow process will hurt conversions considerably, which makes any change well worth testing.

Before you roll out the new cart for your entire audience, you’ll want to make sure that it works on a small scale, and iron out any technical or other difficulties. This way, you can uncover bugs and glitches, remove or add features to reduce friction and improve usability, and generally improve the experience before it is deployed as complete – all without risking your entire income.

User testing also comes in especially handy when you’re designing a product category menu (a common feature of most ecommerce sites). Since website owners and designers are familiar with the products they’re selling, they often organize products into what *they* think are logical product categories.

Then visitors arrive, and get completely lost trying to find what they’re looking for. Conversion rates drop, and the owners are left wondering why!

To avoid poor outcomes in situations like these, submit your changes to tests by real users.

Pros and cons of different user testing methods

Websites can be user-tested in multiple ways. One way is to set up a few computers in an office and invite random people to navigate a given website or page, while you record their experiences. This is known as lab-based testing.

A researcher will directly observe users’ behavior, record it, and make their own notes. Often, participants will also be interviewed about their experiences, impressions, and opinions after the session ends. The results are then analyzed and used to craft hypotheses for improving the site’s UX (and thus its ability to convert).

However, there are a few problems with this technique. It’s expensive, complicated, and the simple fact is that people behave differently when they’re observed. When you’re looking over someone’s shoulder, they’ll act and react differently than they would if they were alone.

Another approach is remote user testing. This method records users’ actions as they navigate your website, ranging from simple mouse movement recordings to eye-tracking and heatmapping.

Remote user testing tends to be both cheaper and more effective than lab-based testing. It doesn’t involve the physical presence of another person, so testers can behave more freely.

There are numerous tools and web services that offer this type of user testing, including UserTesting.com, UserBob, TryMyUI, and more. Most services allow you to select or specify for users who match your target audience so your test results realistically align with what you’ll see on your live site.

How many users do you need?

Now, you might think more users = better. But when user testing, you’ll only need between five to 15 test participants for accurate results. In fact, just 10 testers will likely uncover 90% of your site’s issues.

Interestingly, for every additional user above 5, the number of additional issues identified drops off sharply. Plus, the more testers you recruit, the more data-intensive the analysis process will be and the longer it will take.

The best practice is to employ several batches of user testers, and test sequentially. In the first step, you’ll test with 5 users, then solve the issues they identify. Next, you’ll repeat the process with another 5 users. Do this until you get to the point where users can’t identify any issues.

Select your testers from the audience most likely to understand and appreciate your content – AKA your target audience. If your site offers products or services meant for a specific niche, recruiting a random web surfer as a tester won’t serve you well. You’ll get meaningless observations and “find” issues that don’t even exist for your target audience.

Run effective user tests with the process below

Truly informative tests don’t happen by accident. They begin with a plan.

Once you’ve chosen your testing method and testers, it’s time to form a plan for the most effective way to run your test. This is your test protocol, and it consists of the tasks you’ll assign your testers to do.

Prioritize testing the most important areas of your site – those that make you money!

This means your tasks should typically involve interacting with product pages, buying products, or signing up for your demo or newsletter (the big conversions you’re asking for). The most useful tests will always be those that require the tester to go all the way through the conversion funnel.

To create your protocol:

Design tasks around a specific objective, such as “Find and buy the most popular product in [Category Name].” Or, “Find a product that you like.” Or even, “Buy something from the site.”

Avoid asking abstract questions, like “Do you feel secure enough to buy something here?” The point of this exercise is to observe people as they behave, as they do (or don’t do) something.

Ask your testers to comment out loud as they go through the site, and make sure not to put words in their mouths. If they mention that something is an issue, take them at their word. If they don’t mention an issue you think is there, don’t suggest it to them! (It’s probably not as big an issue as you think.)

To test every part of the shopping experience, your test protocol must include payment. Most payment services allow the use of dummy credit card numbers for test purposes. Inserting dummy numbers enables testers to complete the purchase so you can see if and how everything works.

Watch out for these user testing pitfalls

While it can provide invaluable feedback, user testing is far from foolproof. Here are the four most common pitfalls to be aware of, plus how to avoid them:

  • Sample selection mismatch: Always select your testers to match your target audience.
  • Forgetting your mobile audience: Don’t forget to include mobile audience testing! Mobile visitors are a quickly growing segment for ecommerce stores. Ensuring that mobile users enjoy an optimal experience future-proofs your website (and will increase conversions).
  • Taking user results as gospel: The act of observation alters the behavior of the observed. It’s practically unavoidable. So take user testing with a grain of salt, and always confirm your qualitative results with quantitative data.
  • Inaccurate representation of risk: Testers will not behave exactly like real users do. The best way to see this is by user-testing a website that offers high-value goods or services. Real visitors will often have issues paying $500 for a product, while testers won’t, since they’re not risking their own money.

While these limitations are very real, they in no way make user testing pointless. Far from it. User testing is the only way you can directly observe and hear visitors’ opinions as they navigate your site.

Sure, heat mapping and surveys are helpful, too. But they also remove you from the people giving feedback, so you can’t be sure that your assumptions about the data match what was actually going on in those users’ minds while they were scrolling or answering survey questions.

The user-testing experience can be eye-opening for precisely that reason. What you may think of as the “best” user interface design may turn out to be clunky and difficult to navigate in the hands of real users. Without the raw input of actual humans, it’s too easy to assume that your “perfect” design isn’t the problem – it’s your users! (Hint: it’s probably your design.)

User testing allows you to spot usability issues and fix them easily, in a way no other type of testing can duplicate. The end result is a user-centric design that even brand-new users can easily understand and navigate.

Straight talk is invaluable

When you’re optimizing your ecommerce website for conversion, user testing isn’t optional – it’s a must. There’s nothing like insights from actual users to reveal and remove obstacles standing in the way of conversion!

While surveys can uncover issues that users are aware of and can verbalize, user testing helps optimizers spot issues that users cannot accurately recollect or explain well. Simply by providing real-time feedback as they navigate your site happens and complete assigned tasks, your users will provide you with countless insights.

You can find more useful guidelines for user testing here, developed by the web usability authorities at Nielsen Norman Group and based on their years of experience with user testing.

Chapter 3

User Survey Guide: If You’re Not Conducting Surveys, You’re Losing Conversions

Research is the foundation of the conversion optimization process. If we don’t know the facts behind what is happening on the site — and why — we can’t accurately pinpoint what to fix and how to fix it, no matter how much we test.

As we’ve talked about previously, research can be roughly divided into four main categories:

  1. Quantitative
  2. Qualitative
  3. Heuristic

Best Practices for Surveying

When you run a user survey, you’ll post direct questions to your visitors with the goal of discovering their perceptions of specific issues. For conversion optimization, you can also use user surveys to identify the most common sources of anxiety and friction.

To get the most useful results, pose open-ended questions, like “What do you look for when you shop for bath products?” This type of question allows users to voice concerns or opinions freely, and they may bring up points you hadn’t even considered.

How to structure user survey questions to get actionable results

How you structure your user survey depends largely on your goal. What do you want to know?

You can implement customer surveys, on-site surveys, exit surveys, in-app surveys, pop-up surveys, or email surveys. There are many different channels and times to offer a survey — so your first task is choosing the type of survey that appears at the right time, to the right audience, for the questions you want to ask.

Keep in mind that the timing of your survey will affect the types of responses you get.
For example, if you use an exit survey that pops up as someone is abandoning their cart before purchasing, you’ll be sourcing answers from a very different consumer segment than if the survey popped up after a successfully completed purchase.

Before deciding what questions to ask, first consider what you hope to achieve with your survey. Whatever you hope to achieve, the goal is always to obtain actionable insights.

With that in mind, consider these questions:

Do you have any problems with the site? Yes/No
Say you pose this question and offer two possible answers. After the survey runs, you find that 63% of people answered “Yes.” But what does that tell you? 63% of visitors DO have some sort of a problem — but what is the problem? You’ll have no clue.

The question above can be used only as an introduction to the next question, which will appear to people who answer with a “Yes”. It will probably be something like, “What problems are you experiencing with the site?”

Another not-so-useful survey question:

How would you rate this website?
Questions like a) and b) will not yield actionable insights because they’re not specific enough. Your users’ answers might tell you that something is wrong, but give you no place to begin finding out what, or how to fix it.

To improve your website, you need specific answers that identify problems. So don’t ask users to give you a grade. Ask them something like:

What nearly stopped you from buying from this site?
This is a great question to ask in our post-purchase exit survey example. It invites an open-ended response to a specific question, and will yield answers that point you in the direction of sources of friction and anxiety in the purchase process.

Good user survey design uncovers friction and anxiety

Every question you ask should directly relate to uncovering usability issues, existing friction, and sources of anxiety.

This way, you focus the survey on concepts that visitors can easily vocalize and express. However, don’t expect visitors to provide you with solutions — generally, all they can do is point out problematic spots in your website or sales funnel.

You can also use surveys to enhance your UX and improve conversions in another way: by finding out more about your audience.

When your aim is to learn about your prospects, rather than uncover what’s wrong with your website, you have to be even more careful with your questions. The questions you ask should be clear, short, and open-ended.

You must be very careful not to impose your point of view on your survey participants. In fact, you may want to repeat each question twice, in two different forms, to make the results more reliable.

To find out about your audience, you can ask things like:

“What can you tell us about yourself (your age, gender, and any other information you feel comfortable giving)?”


“What is the specific problem that our product solves for you?”

Both of these questions will likely result in answers that tell you more about your target audience and help you establish personas.

Naturally, your questions will differ depending on whether you’re surveying people who already bought your product (serve them a post-purchase survey) or people who opted out of the conversion funnel (serve them an exit survey), or visitors to your site who are just browsing your product catalog (serve them a traffic survey, asking how they heard of your site, what they expect to find, etc).

Similarly to the timing of your surveys, asking prospects and customers to provide feedback at different stages can provide multiple useful views of your website.

But if you have to choose just one type of user survey to run, you’ll probably get the most useful information from surveying customers who did not buy or who dropped out of the conversion funnel.

How to time user surveys effectively

When surveying your customers or prospects, timing is critical. You must ask your questions as soon as possible so the experience is still fresh in their mind, and the feedback they give is closer to reality. The responses should enable you to minimize any friction that either almost stopped customers from buying, or successfully stopped them from buying.

Keep in mind that you should restrict your questions to the experience of shopping on the website. You are not conducting a customer satisfaction survey, so do not ask questions about the product itself — that comes after a customer has bought. Try to figure out what obstacles or friction the customer has overcome to buy the product.

As a rule, you want to serve your surveys in the least disruptive way possible, which is why so many surveys are sent by email. Email allows a user to choose the best time to respond.

But email surveys also run a high risk of being ignored, which is why we recommend including both a deadline and an incentive for completion. (When you use an incentive, you do run the risk of people answering your survey in a perfunctory manner just to get the incentive. Therefore, structure this type of survey with care so that you get the best possible answers while requiring as little of the user’s time as possible.)

You can also use exit user surveys to find out why individual potential customers did not complete the conversion process. The survey should pop up when the customer displays intent to exit the page (often this is triggered by mouse velocity or location). Exit-intent questions should aim to identify the reason why the person is leaving the site.

A few exit user survey questions we recommend:

  1. What information would you need about the product to purchase it?
  2. What concerns did you have about the product that prevented you from purchasing?
  3. What most influenced your decision not to buy?

Sometimes the answers will identify business decisions as the source of the problem, such as product cost, shipping cost, or similar. In these cases, if the number of similar answers is significant enough, you may want to reconsider the offending policy.

Another option for survey timing is to enable user surveys to pop up for visitors as they go about your website. Use these sparingly, as they can create UX issues if they’re annoying and disruptive! Make it easy to opt out or close the survey to limit the negative influence that the interruption may have on your visitors.

In pop-up surveys, frame questions so that they help you identify user experience problems. These surveys may be triggered to fire when the user has spent some time on the website, checking out the products but neither leaving nor converting. Ask if they’re finding the information they need, for starters.

For any survey, the rule is to keep them short. Anything over 10 questions, and you’ll see a drastic drop in the quality of answers and the number of completed surveys. People just won’t stick with you for that long.

How to conduct a user survey & encourage responses

Once you have prepared your questions, you must decide how to incentivize visitors to actually answer them.

The most common incentive is to offer a discount or give access to gated, premium content. In any case, make sure you attract enough responders.

Don’t run your survey for very long. Each survey needs at least 200 to 300 responses to be statistically significant. Once you have more than 200 responses, you can safely stop the survey, as it’s unlikely that more responses will result in more insights or shed light on more issues.

Limiting survey duration also increases relevance, cuts costs, and reduces negative effects on user experience.

Once the responses are in, the hard part of the survey process starts.

How to analyze and interpret user survey responses

After your survey, you’ll have a massive spreadsheet of text responses. The unenviable task you then face is to go through these and isolate useful insights.

That’s a tall order when you have more than a thousand responses (assuming at least 200 survey takers answering 5-6 questions each).

The best response-filtering method to find the most important insights faster is to isolate certain keywords that represent the most-mentioned concerns or concepts.

Isolating keywords will allow you to create categories of issues and sort responses accordingly. For example, you may have categories like shipping cost, trust, price, ease of use, and so on. By tallying the number of responses that belong in each category, you can rate the severity of each issue.

Once you have this tally, try to make brief summaries of the issues, using the exact words of the responders wherever possible. That way, you’ll have framed the issue, and you’ll be ready to prepare hypotheses for experiments.

Don’t skimp on survey analysis! Use however much time it takes to analyze the answers thoroughly and comprehensively. Try having a team divide and conquer the survey by question, analyze results individually, and then crosscheck the results.

Methods like word clouds or cluster analysis offer an efficient, effective way of structuring survey results so you can quickly spot the most serious issues.

User survey – An example of Gaussian distribution used in cluster analysis of survey results. The red word is the issue most frequently mentioned by responders.

User survey – Word clouds are another method of discovering patterns and commonalities across large amounts of data. The largest words represent the most severe issues mentioned in the most responses.

Avoid these common user survey mistakes

  • Too abstract/vague: If you ask abstract questions that do not tell you anything about your visitors, you’ll invalidate your results. Answers are only as useful as they are specific — and they have to point out real issues on the website.
  • Closed-end questions: Yes/No questions, or even “On a range of 1-10” questions can overlook problems that the test creator may not think of, but that customers notice. Always give survey respondents a chance to voice their opinions in their own words.
  • “Leading the witness”: Mentioning a possible problem in a survey question can “lead” the customer and bias their answer.

Another common blunder is targeting the wrong audience.

For example, if you conduct an exit survey, make sure you leave out people who have completed their purchase. Ideally, you should target the people who have visited a product page and checked the price, or added a product to the cart but did not complete a purchase.

Or, if you are conducting a customer survey, target only the people who have actually completed the purchase.

Every user survey tool allows for audience targeting. Do it!

The analysis stage of surveying is also fraught with potential for misunderstanding and miscategorization. Survey responses should be analyzed comprehensively — because if you miss a potential issue, you are wasting the effort that went into making and running the survey.

You may also be tempted to pay attention to vocal outliers, when in fact solving more statistically significant issues can do more to improve your conversions. Knowing the exact count of responders who mention each issue is vital. Not collecting enough answers is an easily avoidable (yet all too common) mistake.

Always remember the significance table included above, and make sure you have enough responses for statistical significance. If the margin of error is too large, you can’t trust the results. For small samples, even a 10% margin of error may be too much.

And finally, making your survey too long is a mistake we see too often. It’s the easiest mistake to make, and the easiest one not to. Our rule of thumb is: If the onsite survey has more than three questions, it’s too long. Email surveys should have no more than 10 questions.Surveys longer than 10 questions often fall prey to “the error of central tendency”. This refers to the phenomenon of respondents becoming fatigued, and starting to respond with nonsensical answers or answers too short to be meaningful.

Keep your user surveys short and engaging. If you need to have more than 6-10 questions, make separate, shorter surveys, rather than creating one long one.

Getting to know your customers benefits you in the long run

User surveys are a great method to get to know your visitors and customers. You can learn about their motivations, thoughts, and perceptions, and access insights that quantitative research just can’t give you, AKA the “why?” behind the “what”.

Knowing why your visitors do the things they do will help you create faster, easier, more enjoyable ways for them to do it. Also, the “why” can help you improve your website and marketing copy to boost conversions, and give your users a more personalized, enjoyable experience. In the long term, you’ll establish a genuine bond with your customers.

We hope this short guide will make user surveys seem more approachable and useful. Conducting them properly will provide you with a treasure trove of insights — and we mean that literally. These insights will make you money.

Chapter 4

Conversion Research: 6 Treasure Troves of Qualitative Research You Can Access Right Now

Conversion research is an important part of the CRO process. Qualitative research isn’t just about visitor surveys and user testing. While both of these methods give us the opportunity to directly observe visitors’ behavior and get a glimpse of the thought process behind it, surveys and user testing suffer from a serious limitation: visitors and testers are aware that they are being observed.

And that changes things.

Read any quantum physics lately?

One of the fundamental premises of quantum theory (known as Heisenberg’s principle) is that by the very act of watching, the observer affects the behavior of the observed.

And this is where it gets weird: The same thing is true of electrons.

Conversion Research – Needless to say, finding your way around Heisenberg’s department can be confounding

Support calls, live chat recordings, transcriptions, or logs are easy to get — they should already be among your resources (just ask your customer relations department).

The first advantage of these qualitative data sources is that they contain the opinions of customers who have questions about, or issues with, your service or products.

Their second advantage is that people tend to talk more freely in these channels. They specifically point out the problems they’ve observed, which may not be the case with surveys.

The cherry on top? You don’t have to offer any incentive to compel these users to provide their opinions.

If you don’t have these resources already available, put their implementation on your priority list.

First, establish a support call line and a live chat on your website. Consistent research shows that the presence of live chat on ecommerce sites increases conversions. The live chat must be manned and available at all times, or at least during periods of normal operating hours.

Live chat enables visitors to pose their issues in real-time to a live person and get answers immediately. The fact that such prompt service is available offers many benefits, in addition to resolving potential hurdles that stop visitors from converting:

  • It increases visitor trust
  • It creates a bond between the visitor and the website
  • And, in the context of qualitative research, it creates a permanent record of the issue so that it can be analyzed.

Analyzing chat logs is very similar to interpreting open-ended survey results. A good first step is to look for repetitions: Repeated words, phrases, question types and concerns. Finding common keywords allows you to do a keyword search to get an idea of how frequently each type of question is raised. From there, you can resolve the issues that come up most frequently (an indication that they are the most significant).

Support call transcripts or recordings also require the same approach. It’s all about determining categories and calculating frequency to find which fixes will have the most impact.

Apart from live chat’s conversion benefits, qualitative data is why most ecommerce sites nowadays offer live chat. Having consumer data on record saves a lot of effort (and money, too) when doing research for conversion optimization, marketing, and copywriting.

Conversion research #2: Mine useful messaging from product reviews

Most e-commerce websites allow visitors to review products they’ve bought. This feedback is highly valuable in several ways:

  1. It works as “social proof,” which builds trust with would-be purchasers and improves conversions
  2. Voice-of-customer data can be used to improve your marketing messaging and copy. Let your users tell you the most important benefits!
  3. You can use this feedback for your qualitative research

A common mistake businesses make with reviews is to limit the feedback to ratings only (usually using stars), without the option for written feedback. This type of rating is useless.

If your product or service receives many 1-star ratings, it will have a hugely negative impact on conversions — and to boot, you won’t have any understanding of why customers are unhappy. Even if you receive 5-star ratings, you still won’t know why (and neither will your prospective customers)!

Conversion Research – Sony, for example, has made user reviews extremely useful at first glance

Written feedback lets you know what to improve and what customers like about the product. This is why every review system should allow written feedback.

Bear in mind that not many visitors will trouble themselves with leaving elaborate feedback. This is why we recommend that some sort of incentive should be offered.

The most obvious incentive is a monetary one, like offering discounts, coupons, or other shopping benefits to customers who write a review. It’s certainly effective, and can be seen on many sites that reward reviewers with points or tokens that lower the price of a product. If it can be executed without significant impact on the bottom line and protected from manipulation, this type of system works well.

The second way to incentivize customers to leave feedback is to create some sense of achievement, but without offering any direct monetary benefits. Instead, the reviewers might be shown the number of views their reviews had and how helpful they were. They might even reach “Elite Reviewer” status.

This way, a person’s individual sense of accomplishment effectively drives them to leave detailed, helpful reviews. This is the system that Amazon.com uses with great success.

Finally, there is the option of offering access to premium features of the website or service. This is usually used by SaaS companies, which provide reviewers with features not otherwise found in the package they purchased — but Zappos also uses a similar system. Earn enough points by purchasing products and leaving reviews, and you gain access to their “elite”-level perks (like faster shipping).

Conversion research #3: Tackle research head-on with customer interviews

Customer interviews offer the significant benefit of being able to ask participants to elaborate on their answers. You can conduct these interviews either in person or over the phone.

Don’t be too quick to jump on the phone rather than do the footwork to interview in person. Remote interviews may be more cost-effective, but they remove an essential ingredient that is only available via direct contact with the interviewee.

If you’re in the same room with the person you interview, you can observe body language, facial expressions, and other subtle non-verbal communication that can reveal a great deal more about the subject than only verbal answers.

Conversion Research – Never lose sight of this fact, and take it into account when interpreting your interview responses

It must be noted that face-to-face interviews are time-consuming and difficult to arrange, especially if the business is online. Customers may be spread across the globe!

The best alternative in this case is the video interview. Skype, Zoom, and other videoconferencing platforms provide a satisfying middle ground for interviews, and also offer the option of recording the interview for later analysis.

To be useful, interviews must be structured and well-planned. Interview questions can be the same as in those you use in an online survey, but prepare additional questions that complement or expand upon the primary ones.

You should also plan selection criteria for your interviewees ahead of time. It’s very important to only choose respondents from your existing or potential customer base (people who have already bought something from your website).

The fact that they’ve already purchased something means that they already find value in your products or services, which makes them your target customers. You want their feedback. You don’t want feedback from people who are uninterested in what you offer, or who are chronically supportive friends or family members.

To get the best results from in-person or video interviews, they shouldn’t feel like interviews at all. There is an art to the “un-interview,” and establishing a rapport is vital.

A good interview flows like a regular conversation. It feels more natural than awkward. Interviewers need strong interpersonal communication skills to pull this off.

Conversion Research – customer service

Use interviews to find out how customers use your product or service, what their anxieties and motivations are, and what their deeper emotional backgrounds might be. Interviews are all about creating a depth of understanding — going past the surface to find the emotional, psychological, and purely pragmatic underpinnings behind user behavior.

These findings will enable you to improve your copy, even to the point of using the responders’ answers verbatim as statements of your product’s benefits; in headlines or ads; and even as part of your value proposition. Successful interviews can also become testimonials, provided the respondent agrees to it.

Another use for the interview is to find out about your competitors. Questions like, “Did you consider any other product/service?” can reveal the ways your product is better than other available solutions, so you can update you value proposition with those insights.

After the fact, you can analyze interview findings the same way you interpret survey results: Comb through responses and identify the most frequently mentioned issues.

For further reading on conducting interviews, check out: Start Talking! How To Do Customer Interviews That Reveal Priceless Insights

Conversion research #4: “Eavesdrop” on social media posts

Social media channels, including Facebook and Twitter, and other places where customers spend time, like blogs and forums, yield an even greater opportunity to glean customer opinions.

Every day on social media, your customers are freely sharing their opinions of products they bought from you. This can be a source of free promotion, if those opinions are positive. And it can also be a valuable source of user feedback and voice-of-customer data (even if opinions are negative).

When people post about products voluntarily, they’re much less inhibited. They’ll tell you how they really feel in unvarnished language. It’s raw, authentic, and can provide deeper insights than even surveys or interviews (where, again, people are aware they’re being observed).

This is why we believe it’s vital to track relevant social media posts and blog mentions.

A few handy tools to monitor when your brand or product is mentioned:

  1. Meltwater: Use it to instantly search blogs, Twitter, and Facebook for specific terms
  2. Social Mention: Use it to monitor multiple websites in one place and find out which keywords people use when talking about your company
  3. IFTTT (If This Then That): Use it to automate simple online tasks using “recipes”. For example, “If [website] mentions [company], then send me an email alert”

There are many other tools out there. When selecting those you prefer, choose tools that show you not only where you were mentioned, but exactly what was said. That will give you the voice-of-customer data you need.

Conversion Research – A lot of presents can come from the social media box… if you just open it

On your website, you can also provide social media links for visitors to use to share their opinions directly on Facebook or Twitter.

Quantitative analysis can then help you identify which social media platforms are most useful to your website, since your analytics tool will track the number of visitors coming from each individual channel.

Conversion research #5: Got a niche product? Learn to love blogs and forum posts

Even more than social media, blog posts and forum posts can offer deep, comprehensive opinions and critiques of products and services. Tracking these mentions should be a matter of course — you need to know what people are saying about you, because these reviews directly affect conversions.

And, for the purpose of conversion optimization, these reviews are very useful as additional sources of insight of what people expect, what they find, and where you might be losing them.

For niche products especially, blog posts and forum posts are the best and most readily available sources of voice-of-customer data. Influencer opinions and reviews reach your target audience, giving the most valuable form of social proof you can have. They can also do serious damage to your conversions if their impressions are less than favorable.

Since conversion optimization is primarily interested in the performance of the website as the main source of conversions, issues with products themselves may be outside of your purview. But blogs and forums still contain useful feedback that you can use to improve the copy on your website and address usability issues.

Conversion research #6: How to stay focused & avoid data overwhelm

Too often, the most readily available sources of user feedback are ignored in favor of more complicated and expensive methods of acquiring qualitative feedback. Don’t let that happen to you.

We recommend beginning your qualitative data gathering process by making a comprehensive review of existing available sources. Just be careful not to get distracted by issues that are not directly related to conversion optimization (which is all too easy to do!).

Conversion Research – This maxim is just as applicable for conducting customer research as it is for leading Apple

Your task as an optimizer is to improve the website, not the products or services themselves. To reduce the risk of distraction, approach your data gathering with a plan. Use your survey questions as a guideline for what types of answers to look at more closely.

And if product issues continue to arise, pass on that information to the appropriate people — so you can stay focused on your job.

Chapter 5

How to Find All the Qualitative Data You Need for CRO Testing (In Unexpected Places)

Numbers and words — or quantitative and qualitative research, as optimizers think of them — are two of the most important parts of thorough conversion research. The results are the basis for CRO testing.

But while locating conversion improvement opportunities through quantitative data is relatively easy and straightforward (simply compare your numerical results to your expected results!), implementing qualitative feedback for better conversions (CRO testing) just isn’t as simple.

Qualitative data isn’t easily compared, or analyzed, or even visualized, which means there’s a lot of room for error. It takes some serious interpretation skills to turn wordy answers from surveys and interviews into valuable, actionable insights.

But within these wordy answers lie the reasons your customers are or aren’t converting. It’s the most valuable kind of feedback you can collect.

And there are many, many ways to gather this data, each with their own benefits.

Understand your options for gathering qualitative feedback for proper CRO testing

The goal of qualitative research is to capture the “voice of the customer,” which you can do in several ways:

  1. Surveys, polls, and queries
  2. Interviews
  3. User testing
  4. Live chat logs
  5. Support call logs
  6. User reviews of products and services
  7. All other forms of direct feedback (social media posts, blogs, forums)

CRO Testing: Seven main types of qualitative research give you a well-rounded picture of your customer’s deepest pain points, desires, and priorities.

All of these methods are useful, but not all are appropriate to use every time. Here’s a rough guide for what method to use when.

1. Surveys, polls, and queries

As a method for getting the most possible responses from your visitors, you can’t beat surveys or their cousins: polls and queries. These methods enable you to pose identical questions to as large an audience as you want.

Depending on which tool you use, you may be able to segment your audience and target only the people who bought something, the people who logged in a specific number of times to your website without buying, and so on.

The results you get from surveys will be as relevant and actionable as the amount of thought you put into the questions. To get answers that truly reflect your customer’s perception of the website, your questions need to allow respondents to express their opinions freely and without being led.

Surveys come in several different varieties, including exit surveys, consumer surveys, pop-up surveys, and Net Promoter Score surveys. For a detailed discussion on how to conduct surveys, check out our guide.

Once you’ve conducted your survey and collected the results, you’ll have the challenge of interpreting them. Ever try to make sense of 200 or more answers? It’s not easy!

To successfully process this amount of text data, start by isolating the specific keywords that best indicate the issue you’re trying to diagnose. Once you do that, count the number of times respondents mention that specific issue. This will give a good indication of the severity of the issue.

CRO Testing: Creating a word cloud is a popular technique for evaluating qualitative research.

You may find that you have to create several categories for the most commonly mentioned issues. If that’s the case, order issues by severity so you can tackle them in order when it comes time to A/B-test solutions.

Now it’s time to reread your results and try to hypothesize the best ways to eliminate the issues identified. Sometimes, if you’re lucky, the customers will point you in the right direction. But don’t count on them do the problem-solving work — that’s your job. As Henry Ford supposedly said:

“If I asked people what they wanted, they’d have said faster horses.”

Whatever hypothesis you create, be sure to test it first before implementing it permanently.

Now, there are some drawbacks to conducting surveys. For starters, they require a significant amount of effort. This is no excuse not to do them, however — just be sure to plan and prepare your surveys to get valid results. Otherwise, all that effort will be for nothing.

Eliminating the issues your customers help you identify through survey responses is one surefire way to increase your conversions… as long as you interpret the answers correctly.

On-site polls
Polls serve a different function from surveys in that they help to uncover the proportion of people who have specific issues with some aspect of your website.

You’ll use a poll once you have already uncovered an issue and need to decide how to approach the solution. If your solution is difficult to implement, for example, polling your visitors may help eliminate less successful variants before you even design them, reducing the stress on your resources.

When running a poll, you should be very confident in the issue you’re addressing in order to avoid putting words in the mouths of the respondents. When running a poll that calls for open-ended answers, phrase the question in a way that leads to longer, more detailed responses.

For instance, don’t set up a poll asking, “Are you leaving without purchasing because you don’t trust our site?” Instead, phrase the question, “What kept you from purchasing today?”

Why poll rather than survey?
Poll results often lend themselves more easily to comparison and interpretation than survey results, which can include multiple answers that may all relate to each other.

Multiple-choice poll responses can be given numerical values, so you can use those to instantly analyze your data. The best way to use number-based polls is to set up questions with responses on a scale of 1 to 5 or 10, where 1 stands for “Strongly Disagree” and 5 or 10 stands for “Strongly Agree”.

2. Prospect or customer interviews

Interviews are an expanded version of surveys. During interviews, you actually talk with your prospects or customers, so you have the opportunity to discuss their responses and get more detailed answers.

Interviews are by far the best way to gain insight into your customers. They can serve many useful purposes, from identifying and exploring issues in detail to finding fodder for future marketing copy and customer success materials.

The only disadvantage to conducting interviews is the amount of time and effort they require. However, when done well, they’re worth their cost, and should be high on the list of priorities for qualitative research.

3. User testing

User testing involves recruiting a group of random users to navigate your website and identify common issues. They are usually given specific tasks to complete, and often, their session visit is recorded.

The results, from users’ verbal feedback to their session recordings, can be of great value on the heuristic side of CRO research. Users might just offer you a new view on your UI, or reveal a problem you hadn’t noticed.

Here’s our detailed guide to user testing, including where to source users and how to interpret their feedback.

4. Live chat logs

If your website maintains a live chat option (and if it doesn’t, it should — but more on that later!), then going through your live chat logs can be an enlightening experience.

Chat logs, especially support logs, can contain a treasure trove of insights. Anything from visitors’ complaints to their product information queries can indicate areas that need to be improved.

The best part is: you’ve already got these resources. Unless you don’t.

If your website does not have a live chat feature, make it the next priority for your developers. They’re relatively inexpensive and easy to implement, and can make a big difference in conversions. In fact, research from Kissmetrics and eDigital Research indicates that conversion often doubles for visitors who use live chats while shopping online.

5. Support call logs & support staff interviews

Recordings of support calls from customers can also provide valuable information that you can use to improve calls to action, fix value propositions, and identify common problems with products or services.

In fact, interviewing support personnel is nearly as useful as conducting customer interviews. Customer support workers are on the front lines with your customers, and interviews with them will frequently yield insights you can’t get any other way.

For example, in regular customer interviews, customers tend to be hesitant to express negative opinions because they don’t want to hurt your feelings. However, they may not be as concerned about the feelings of your customer support staff!

6. User reviews of products & services

This form of direct user feedback is too often ignored, even by large companies. (Only click that link if you’re ready for a horror story featuring toilet paper and terrible support.)

It’s well worth dedicating time to this readily available user feedback. Customers volunteer so much information via their reviews, and you don’t need to organize a survey or conduct interviews to get it.

Ideally, you already have user reviews enabled on your website.

A slight snag: sometimes stores allow reviews, but nobody writes them. If this is the case, you may need to incentivize customers to review you, at least at the beginning. If participation is low, carefully check out the review process and try to make it more prominent, and ask customers via email to review the product they purchased after a few days.

CRO Testing: Reviews are a source of invaluable (and often unfiltered) customer feedback.

Some companies use different incentives to compel their consumers to review, such as point-gathering schemes that offer rewards or coupons. Strategies like these don’t cost much to implement and can be very effective.

As with any other form of user feedback, this one is too valuable an opportunity to miss. Gathering consumer reviews is effortless, and once the basic mechanism is set up, the only task remaining is to actually read the reviews.

The best way reviews can be used? To improve your copy.

Notice which words your customers use to describe the biggest benefits you offer and the biggest pain points your product solves — then use those same words and phrases in your copy.

7. Other forms of feedback

Today, most (if not all) web users maintain some sort of profile on at least one social network. While you can’t directly track the social network profiles of your customers, it’s possible to track their mentions of your website or the products/services you offer.

Sentiment analysis tools are the quickest way to get this information, and will additionally tell you whether these mentions are generally positive, or negative. You can use this information to keep tabs on how well you’re serving your target audience, to help inform your buyer personas, and to help your writing team “learn customers’ language” for future copy.

You’ll also want to monitor mentions of your brand or service on blogs and forums — particularly professional or niche forums for products/services that do not have a wide target audience.

It’s not the data-gathering that matters for CRO testing

The most challenging part of conducting qualitative research isn’t gathering qualitative data, though that can be a labor and time-intensive process.

It’s interpreting and quantifying that data accurately. Whenever words are involved, there is room for interpretation, which can be flawed. But when we begin to gather lots of voice-of-customer feedback, patterns emerge that can point us in valuable, conversion-linked directions.

When structuring your plan to gather qualitative data, remember to use more than one method, because no single method will give you a well-rounded perspective on what your customers are really thinking, feeling, and experiencing on your site.

And keep in mind: Your goal for conversion optimization is to find out what customers think about your website — not what they think of your products or services. That’s a whole different ballgame.

Look for feedback that indicates areas of your website that cause friction, anxiety, confusion or distraction. These are the worst conversion-killers. Then use that qualitative feedback alongside your quantitative data to form a clear picture of what areas to optimize first, and which hypotheses will stand the best chance of success.