Preference Testing: A Step-By-Step Guide

What’s preference testing? What are its benefits and drawbacks? How to conduct a preference test and collect feedback?

These are some of the questions that this article tackles, so if you’re after the answers, you’re in the right place!

Let’s get right to it, shall we?

TL;DR

• Preference testing is a research method used by UX and UI designers to decide which designs users prefer and why.
• This kind of testing normally happens in the early stages of design but you can also conduct them iteratively to improve existing designs.
• Preference testing uses both quantitative and qualitative data to evaluate how users perceive different designs. A/B testing, on the other hand, requires a working product and quantitatively measures actual user behavior.
• Preference testing helps teams design user-friendly and intuitive UIs, websites, and apps that satisfy user needs.
• The main downside of preference tests is that it doesn’t involve a fully functional product and there could be big discrepancies between how users perceive a design and how it actually works.
• Always start the preference testing process by setting a goal, like creating a design that’s easy to navigate.
• To recruit participants that are representative of the target audience, use dedicated services or tools. If recruiting from your existing user base, leverage the segmentation features of your tool and engage them in-app.
• Avoid general questions, like ‘Which design do you prefer?’ and make sure they are aligned with your goals. Always include follow-up questions to gather qualitative insights.
• Before making any decisions, ensure that the differences in quantitative data are statistically significant. If possible, use AI tools to organize and analyze the qualitative insights.
• UsabilityHub enables teams to quickly create and run a preference test and it analyzes the quantitative data automatically.
• Hotjar offers 1:1 interview functionality that automates the process of recruiting test participants, scheduling the test, as well as recording and transcribing the interviews.
• Userpilot is a product adoption platform that you can use to recruit test participants and run simple preference tests. Book the demo to see how it can help your team.

What is a preference test?

A preference test is a research tool used in UX and UI to assess users’ preferences for a particular design.

During the test, you present potential customers with two or more options and ask which one they like the best.

For example, you could present your users with two menu designs, a collapsible sidebar menu and a horizontal dropdown menu, and then solicit feedback on which of them is easier to navigate and better fits the overall website design.

Is preference testing the same as A/B testing?

Preference testing and A/B testing are two different types of research.

Preference testing is used at the early stages of the development lifecycle. It assesses how users perceive the design to inform the design process.

A/B tests, on the other hand, measure actual user interactions with a working product later on in the lifecycle.

Why is conducting preference tests beneficial?

There are a number of reasons why you should consider conducting preference tests.

First, they are flexible and fairly quick to arrange and administer when compared to other research methods. They are a source of both quantitative and qualitative data, and the results are simple to interpret.

Most importantly, however, they offer teams data that helps them create interfaces, websites, or applications that are intuitive, user-friendly, and aligned with user preferences. This leads to improved user experience and usability.

What are the drawbacks of preference testing?

Preference tests have a number of limitations you need to be aware of.

For starters, the obtained data doesn’t always reveal the whole picture.

For example, it doesn’t tell how much users like each of the preference test design options, and the differences might be marginal.

Even if you follow up with open-ended questions, the qualitative user feedback could be too vague to be of any use because users aren’t always able to explain their own preferences.

However, the main downside of preference testing is that you actually don’t test a functional product, only the design. Users may choose one that is visually more appealing but that doesn’t mean it will be user-friendly and easy to navigate.

When should you conduct preference testing?

Preference testing is usually done early on in the product development cycle when you’re still working on the design.

However, you can also leverage them during later iterations, after the launch, or when you’re redesigning/optimizing the UI and UX.

For example, by testing your own design against those of your competitors, you can identify areas for improvement and gain insights into what makes a good user experience in the eyes of potential customers.

Step-by-step process for performing preference tests

Is preference testing the right research method in your circumstances? If so, great!

Here’s how you conduct preference tests, step by step.

1. Identify objectives for carrying out the preference test

To give it the necessary focus, start preference testing by setting your goals.

For example, your goal could be designing a user interface that is easy to navigate and helps users achieve their goals in less time.

To help you set your goals, use a framework like OKR, SMART – or a mix of them.

2. Recruit test participants representing your target audience

Next, it’s time to recruit test participants.

If you’re designing a new product, you could recruit testers via social media or online communities, or you can hire professional testing services to do the job for you. Some usability testing tools also give you access to a pool of potential testers.

If you already have a product in place, the easiest way is to leverage your existing user base. In this way, you ensure that the test participants are representatives of your target audience.

How can you do that?

Use the segmentation functionality in your analytics tool to choose users with the right properties. While it may be tempting to reach out to your power users, they are not always a representative sample of your user population.

Next, target the chosen users with an in-app message, like the modal below, and invite them to take part in the test. To get enough participants, offer them some incentive, like a voucher.

Speaking of which, what sample size do you need to ensure the statistical significance of your results? For quantitative studies, you need at least 20-40 participants.

3. Prepare the preference tests and the questions to ask

Start preparing the preference test by creating the design options you will be testing and the questions.

To get actionable insights, avoid general questions like ‘Which option do you prefer?’ Instead, make sure they reflect your objectives and isolate the points of comparison.

For example, if you want to simplify the design to reduce the time users need to complete their tasks, you could ask, ‘Which design is the easiest to use?’ or ‘Which design is the easiest to understand?’

Once you settle on the main question, select relevant follow-up questions that will enable you to gain a deeper understanding of user preferences.

You could start with a general open-ended question like ‘What makes this design easy to understand?’ To get more controlled results, you could provide a list of adjectives that best describe the design, like ‘clean’ or ‘simple’, and ask users to choose 3-5.

For even more detailed insights, ask users to rate specific aspects of the design on a Likert scale. For example, ‘On a scale of 1-5, how intuitive is the menu design?’

4. Run the preference test to collect user feedback

Once you set up the test, it’s time to run it.

There are different ways to do it. You could arrange interviews with participants, either face-to-face or over video calls, or use dedicated testing tools like UXtweak.

The latter is much easier and quicker but also less personal and doesn’t allow the same level of flexibility as face-to-face interviews.

However, if you choose interviews, make sure you don’t lead your participants with your questions, as this will corrupt the results. To avoid the recency bias, alternate the order in which you present the different options.

5. Analyze the results from the preference test

Let’s look at how you can analyze the results to get the best out of the preference test.

Quantitative feedback

Analyzing the quantitative user feedback is fairly straightforward.

All you have to do is compare how many users picked each of the test options.

The only issue to look out for is the statistical significance of the results. To determine if there’s a statistically significant difference, you may need to carry out statistical tests. Unless your tool does it for you!

If there’s no difference, consider running a follow-up test with a larger sample.

Qualitative feedback

How you analyze qualitative feedback depends on the questions as well as the tools you’ve used.

If you’ve collected the data via interviews, start by transcribing the answers or at least use the recordings to verify the accuracy of your notes. Next, organize the responses and look for patterns.

If you’ve used a dedicated market research or feedback tool, the odds are it will do a lot of the work for you. For example, many tools allow you to tag and categorize the responses or leverage AI to analyze qualitative user feedback.

Even if your tool doesn’t support such functionality, you should be able to export the data and prompt generative AI tools like ChatGPT to help you with the analysis.

Preference testing examples to consider

What aspects of product design can you test with user preference testing?

Here are a few ideas to consider:

• Logo and branding – for example, color palettes
• Website or product dashboard layout – to choose the most effective one
• Product prototypes – to help inform product development and design decisions
• Copy and messaging (including microcopy of in-app messages) – to pick ones that resonate best with target users
• Reviews and testimonials for the website – to choose the most impactful ones
• Videos, images, and sound files

The best preference testing tools of 2023

Having the right tools in the stack can make preference testing easier and enable you to extract valuable insights in less time. Let’s look at a few platforms that are worth considering in 2023.

Userpilot – for recruiting participants and collecting feedback in-app

Userpilot is a product adoption platform with robust analytics functionality. It also allows you to collect customer feedback and design personalized onboarding experiences.

How can you leverage Userpilot for preference testing?

Its segmentation features allow you to select the right participants for your test. You can then trigger in-app messages to invite them to take part in the experiment.

In addition, you can run simple preference tests inside your product with Userpilot’s in-app surveys. Just add your questions, both closed- and open-ended, and launch the survey for the chosen user segment.

UsabilityHub – for conducting user research and analyzing participant responses

UsabilityHub is a remote product research platform that helps designers make data-driven design decisions.

What preference testing features does UsabilityHub offer?

It allows for the testing of up to six design options at a time, and you can ask follow-up questions to gather qualitative data. The quantitative data is automatically tested for statistical significance so you don’t have to worry about running complex tests.

You can also export the qualitative feedback for detailed analysis.

Hotjar – for carrying out interviews to collect qualitative user feedback

Hotjar is best known for its session recordings and heatmaps, but the tool also allows you to schedule and run 1:1 interviews, so it’s the perfect solution if you want to conduct the preference test in this way.

What are the benefits of using Hotjar for user interviews?

For starters, Hotjar gives you access to a pool of 200,000+ testers from 130+ countries and various industries. This is an excellent solution in the early design phase when you don’t have a user base yet.

Apart from recruiting participants, the platform allows you to automate the process of scheduling interviews. It also records and transcribes them for you so that you can easily share the findings with your team.

Conclusion

Preference testing is a quick and cost-effective way to determine which of your design options is most appealing to your target audience. In this way, it can help you design a product that offers great usability and accessibility the first time around. You can also use them to refine existing designs.

If you want to see how Userpilot can help you arrange and administer preference tests, book the demo!