As product teams move to a more customer-centric approach, user testing is becoming increasingly important. After all, how can you know whether you’re building something that will delight your customers if you never put it in their hands? Your team should provide end users with the opportunity to test the product thoroughly and engage with the product throughout its life cycle.
Over the next few paragraphs, I’ll go over the key challenges product leaders face when conducting user testing and some techniques for navigating each of them.
Finding the right user
Identifying your customer is one of the fundamentals of product management. However, product managers often consider very generic user personas rather than the specific ones actually interacting with the product. For example, let’s consider an e-commerce portal like eBay. The majority of you will agree if I say that sellers and buyers are the most appropriate personas for eBay. However, this might not be true in every scenario. If you are a product manager for eBay and are building an advertising analytics feature, the customer persona would be neither a buyer nor a seller. It’d be the advertising agency’s analytics manager. If you tested the product feature with a bunch of buyers and sellers rather than the analytics manager, you wouldn’t be able to gather any useful insights.
The simplest way to avoid this mistake is to work backward from the customer. Instead of first building the product and then finding users to test it, a product manager should start with identifying the customer need and defining the problem to be solved. One technique that can help you do this is the “working backward” process pioneered by Amazon. The idea is to write an internal press release for the product before you start building it. Essentially, the press release describes the end product with the specific customer problem in mind.
Selecting the right number of users
Once a product manager identifies the right group of users, they must then define the correct number of users for testing. This step is critical for gaining insights that can help improve the product, validate hypotheses, and avoid selection bias. However, there is no one-size-fits-all solution to this challenge. The right answer depends on the product and the target customer. For example, an enterprise product such as a database service might need significantly fewer users than a consumer-based one like an e-commerce website. Furthermore, complex user studies require a greater number of participants to cover as many user scenarios as possible.
Irrespective of the actual number, a product manager should ensure that the users participating in the testing are thoroughly trained on the product and its features. A well-trained user base makes testing more accurate. In situations with a smaller number of participants, a well-trained user can help cover multiple test cases. In those with a large participant count, the PM might need to get a bit creative. Conducting tests via video conferencing or sending out surveys via Qualtrics can be extremely useful for boosting the number of participants. Further, attracting users with gift cards and discount coupons can be another way to increase the participant count.
Gathering the right insights
The final challenge is gathering the right insights. After a PM finds the right users and determines the correct sample size, they then need to focus on designing the tests so that they produce the most useful and relevant insights. If a product manager for an e-commerce website is focused on ensuring that the checkout workflow is working well, he or she should avoid distractions such as the shipping workflow. While these other features might indeed have issues, the tests should focus on what is most relevant to the roadmap for the product or feature at hand.
Your team must also ensure that the process of gathering insights is metrics-driven. Identify relevant metrics such as NPS and click-through rate beforehand. Then ensure that the tests will effectively capture these numbers. Gathering data will help make the whole process of user testing actionable. Techniques such as A/B testing, cohort analysis, and funnel analysis allow you to tell whether the user is able to complete the target workflows.
To summarize, user testing is not easy. Your team’s success depends on many moving parts. Every product manager and their managers should understand that failure is very common in this process. The only way to increase the probability of success is through experimentation and making the right choices with respect to user identification, sample size, and insights analysis.
