I first started playing around with the world of Conversion Optimisation back in 2007, when Google first released their Google Website Optimiser tool (which they later renamed Content Experiments and merged with Google Analytics).
For data nerds marketers like myself, back then it looked like a dream tool to play with. No more second guessing or endless discussions on which UX elements to modify on the website on the basis of someone’s opinion. I would now have a neutral and objective way to assess what was working and what not.
Combining this with my growing awareness on the importance of Conversion Rate Optimisation, pushed me into a period of irrational exuberance, whereI started running countless A/B tests on a wide range of different websites.
At the time, most of my tests were driven by the desire to emulate what other best-in-class players were doing on their own sites.
You would think that by replicating the user flows and UX elements of the top performers, you could easily improve your own conversion rates, right? WRONG.
One after the other, each A/B test failed miserably. Delivering not even the slightest improvement in any of the conversion funnels I was aiming to improve.
The most frustrating part was that, at the time, I was already quite familiar with the whole digital marketing space, having co-founded a quite successful and fast growing digital marketing agency, which was helping clients on their user acquisition efforts.
How come then that I couldn’t nail Conversion Rate Optimisation?
Qualitative (conversion) Research to the rescue
After devouring all the existing reading material from the top CRO experts in the market and putting each new learning into practice, I realised what the original sin was.
I had been spending all the time focusing on the conversion metrics, without putting any context around them(and the people generating such metrics).
More specifically:
- relying almost exclusively on quantifiable data to the detriment of qualitative research
- (as a consequence to the above) starting the A/B testing phase way too early, instead of spending more time gathering insights directly from the users
Put it differently, if quantitative data allows you to know what your users are doing on your site, qualitative research will tell you why they are doing it.
Without one of the two components, your chances of A/B testing anything which can have a substantial impact on your conversion rate will be close to 0.
How to get started with Qualitative Research
Here is a step-by-step approach on how to perform Qualitative Research, with what I’ve found to be the 3 most valuable source of insights over the years:
- Customers surveys
- Onsite surveys
- Usability testing
1- Customer surveys
Sending online surveys to your existing customer database is the quickest way to know what your customers are thinking of your product.
1.1 — Choose your survey tool
Pooldaddy or SurveyMonkey are the tools I’ve used the most and they are both great at what they do. They both offer free plans , although I suggest you to go for their entry paid plan (50$/month for SM, 200$/year for PD) to take advantage of their more advanced features (from white labelling your questionnaire, to export functionality, tracking specific email responses, etc). The ROI will be many times the initial subscription cost, so don’t worry too much the small initial investment
For a more exhaustive list of survey apps to choose from, take a look at Zapier’s handy guide:
1.2 — Segment your recipients
You should segment your audience according to the type of users you are going to send the survey to, so to be able to ask more specific questions to each segment.
The degree of segmentation depends on many factors but one thing you need to keep in mind is that each of your segment needs to have a minimum number of recipients for it to make sense. There is no point in being ultra granular if you end up with segments of <100 people.
For eCommerce players, a simple segmentation you might want to start with, can be the following:
- Existing customers — users that have already made an order
- Abandoned cart — users that have added at least 1 product to cart in the past X months but haven’t made their first purchase yet
- Newsletter subscribers — users that have signed up for your newsletter but don’t fall into the two above audiences
1.3— Choose the questions to ask
The most critical activity as the quality of the answers you will collect will only be as good as the questions asked.
The folks at Qualaroo have a fantastic guide to surveying users, where you can find inspirations on what questions to ask, how to ask them and the rationale behind asking them.
These are the key aspects to keep in mind before drafting your survey:
- ask only actionable questions. For every question you ask, you need to think before hand what could be the ‘so what’ and the action following the answers collected. If you can’t act on it, you shouldn’t ask the question in the first place.
- don’t bias your questions. As much as you would like your users to answer in a way which can validate your initial hypothesis, avoid at all costs to do so. Answers to questions such as “We recently revamped our website to look more modern and easier to navigate — what do think about it?” won’t be of much value.
- combine a mix of close-ended and open-ended questions. Open-ended questions will require more work to be digested and analysed (given their unstructured format) but the insights you could get from them can be of incredible value. With too many open-ended questions, on the other hand, you risk having a higher percentage of users skipping the survey altogether. A good compromise is to add an open-ended “other” answer so to leave it up to the respondent whether he wants to gives you a specific feedback.
For more valuable tips on the type of questions to ask, head over to the Qualaroo guide.
1.4 —Offer an incentive
In order to increase the response rate of your surveys, so to get more value out of it, think about the incentives you could give to anyone completing the survey.
Two simple incentives which work quite well are to offer a voucher for the next purchase on your site or the chance to win a prize.
1.5— Write the covering email
The last step before sending the survey is to craft the covering email to ensure the highest possible open and click-rate.
These are three killer tips from Conversion Rate Experts to follow:
In the subject line, emphasize the benefit to customers. That’s why we used “Get your holiday free” instead of “Can you fill out our survey?”
Sell the survey as you would your products or services. We emphasized that the survey was easy to fill out (“it only takes about 3 minutes”) and added the incentive (“win back the cost of your last holiday”).
Emails should be personal. The more personal the email, the more likely people are to respond. That’s why we included the customer’s name in the email and signed it from a person, rather than from an impersonal marketing department. We emphasized this further by including a photo of sunshine.co.uk’s managing director, Chris Brown.
Keep in mind that the overall response rate of your survey will be primarily driven by the type of users you will send the survey to.
With a catchy cover email, balanced mix of open-ended and close-ended questions, and the right incentive in place you could obtain a 10–20% response rate from your existing customer base and anywhere between 1–5% for email leads you have captured on your site (such as newsletter subscribers in the case of eCommerce).
2) Onsite surveys
Onsite surveys, or traffic surveys, allow you to ask questions to your visitors while they are navigating your website / app.
2.1 — Choose the onsite survey tool to use
The best tool around is probably Qualaroo (former KISSinsights), which unfortunately doesn’t come cheap. They no longer offer monthly plans and their most affordable plan starts at $63/monthly (billed on an annual basis).
A cheaper alternative is Qeryz’s survey tool, with plans starting at 15$/monthly. Definitely not as powerful as Qualaroo’s, plus it can be quite buggy at times, but enough to do the job if you can’t afford the more expensive solution.
2.2 — Target your questions
The main advantage of onsite surveys is that they allow you to ask targeted questions at the right time to the right visitors. Your survey widget can be prompted on the basis of variables you can decide such as the page URL, time on page, number of visits, referring channel or any other custom variable based on your internal data.
2.3 — Choose the questions to ask
Compared to your external survey your onsite questions should be much shorter and mostly close-ended. It’s very hard to collect open-ended answers from your visitors while browsing your website / app.
Examples of targeted questions for eCommerce:
- (on product pages AND with time on page > 30sec) Is there any other information you would like to see on this page?
You want to know if there are any important piece of information your visitors are actively looking for which are not currently displayed on the page (or which are not visible enough).
- (to visitors who haven’t made their first order AND with abandoned carts) Is there anything holding you back from proceeding with the purchase?
You should provide different possible answers based on your specific offering (free shipping, more product pictures, etc) and an “other” option to collect insights you haven’t thought about before.
- (on successful transaction page for first time customers) What convinced you to make the first purchase on [YOURBRAND.com]?
You want to know directly from your first time customers which elements of your site convinced them to shop (vs. any of your competitors). Once you have a clear understanding of your most persuasive assets, you should try to to further highlight them on your site.
3 — Usability testing
Let’s start by defining what usability testing is:
Usability testing is a method of evaluating a website or app’s readiness for release by testing it with real users who are part of the target audience. Usability tests measure the relative ease with which users can accomplish a set of tasks that a typical user of the app or website would need to accomplish. (from Optimizely’s glossary)
If you are just getting started with usability testing, before anything else I suggest you to read these two very short books by User Experience’s guru Steve Krug:
- Don’t make me think: A Common Sense Approach to Web Usability
- Rocket Surgery made easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems
Another must read are Jakob Nielsen’s (aka the Godfather of web usability) articles on user testing.
3.1— Recruit your testers
You need to recruit participants who fall into your target audience. Also, if you have very diverse user groups, it’s advisable to recruit different users for each groups and test them separately.
Try avoid using your internal staff as you should test only users that had no direct involvement with the development of your product.
How many users should you test?
A big misconception is the need to recruit a sample big enough to be statistical significant. Truth is, you need no more than 5 users to discover on average 85% of your usability problems.
To quote Jacob Nielsen:
The most striking truth of the curve is that zero users give zero insights.
As soon as you collect data from a single test user, your insights shoot up and you have already learned almost a third of all there is to know about the usability of the design. The difference between zero and even a little bit of data is astounding.
When you test the second user, you will discover that this person does some of the same things as the first user, so there is some overlap in what you learn. People are definitely different, so there will also be something new that the second user does that you did not observe with the first user. So the second user adds some amount of new insight, but not nearly as much as the first user did.
The third user will do many things that you already observed with the first user or with the second user and even some things that you have already seen twice. Plus, of course, the third user will generate a small amount of new data, even if not as much as the first and the second user did.
As you add more and more users, you learn less and less because you will keep seeing the same things again and again. There is no real need to keep observing the same thing multiple times, and you will be very motivated to go back to the drawing board and redesign the site to eliminate the usability problems.
After the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new.
How to recruit your users?
Start by distinguishing between people already familiar with your service and the ones who are not.
People not familiar with your service
a) Friends
PROs
- Easiest people to recruit
- You probably don’t need to offer them any incentive to participate to your user tests
CONs
- Limited in number
b) Offline strangers
You can recruit people that live/work in your same building, in any cafés or literally in the street.
PROs
- Infinite pool of people to recruit from
CONs
- You need to overcome the rejection fear as the majority people won’t be interested in participate to your user testing
- If your user base is very targeted, it will be hard to find users with similar characteristics among a random pool of people
- Time intensive
c) UserTesting.com
UserTesting.com allows you to tap into their gigantic panel of users (+1M) for $49/user ($99 after the first 10 users).
You can choose the target audience to recruit from on the basis of criteria such as age, income, gender, country, web expertise (average or advanced) or operating system.
PROs
- Most scalable solution if you need to recruit a lot of people for different tests
- Quickest way to get started (you can get the first test performed in under 1 hour)
CONs
- The targeting criteria are suitable only if your target audience is pretty generic
- You won’t be able to interact with your users while they perform the tasks assigned
- Only available for english websites
People who are already familiar with your service
a) Customers
Your existing customers are an obvious option to leverage, especially if you are launching a new product.
PROs
- You can choose your own (advanced) targeting criteria
- Relatively quick and easy to recruit users
CONs
- You don’t want to use send them too many email for recruiting purpose, so to avoid spamming them
b) Visitors
You can intercept people directly while they are navigating your website / app through a recruiting screener. I’ve found ethnio to do a terrific job at this (for $49/monthly). Can’t recommend it highly enough.
To get started, you need to place the ethnio javaScript on your site and create the screener with their built-in editor. You can choose to show the recruiting screener for specific geographical area, to certain devices, or setup your own custom filters based on your customers database.
Where the magic happens, is that you can integrate ethnio with your own calendar so to schedule automatically the user testing sessions without any time consuming back and forth communication between you and the users.
Don’t forget to provide an incentive for people to participate, as it will drive up quite substantially the number of participants you will be able to recruit.
PROs
- Highly targeted recruiting: you can for example recruit people who have browsed a section of your website but haven’t complete a specific action
- Once the initial set up done, you should be able to recruit quite a large number of targeted users without any additional time investment
CONs
- The initial set up take a bit of time, especially if it’s your first time using the tool
3.2— Write down the test script
Once you have recruited your pool of users, it’s time to think about what to ask them an the task you will want them to perform. If you have read Steve Krug’s Rocket Surgery made easy, you should be already good to go.
If you haven’t, you can refer to Jakob Nielsen’s guidelines for usability testing:
- Make the task realistic: realistic and typical for how people actually use the system, when they are on their own time, doing their own activities
- Make the task actionable: encourage the users to interact with the interface
- Avoid clues: don’t give away the answer
If you are looking for a template to start with, you can take Steve Krug’s usability test script
3.3— Conduct the usability test
Whether you invited your testers in your office or you are meeting them in a noisy cafe, you should always remember to record the screen session while taking notes. Not only it will help you recall anything you might have missed while interviewing the user, but you will also be able to show the recording to any stakeholder within your company, so to further convince them on why something needs to be changed.
No need to look for any fancy software here, the screen recording feature of QuickTime Player will do the job perfectly.
Conclusion
Qualitative research enables you to look inside the mind of your users, so to avoid speculating or copying whatever best practices you see on other best-in-class websites.
If the whole process looks to you like a lot of work, you are right, it is a lot of work.
However, the initial amount of time invested in qualitative research will allow you to drastically reduce the chances of running unsuccessful A/B tests later on..and giving up on Conversion Optimisation.
Happy Researching!
-Matteo
