Getting Better At Product Testing
I’m working on some new Q&A Videos about topics I’ve discussed frequently. I’m starting a short series on testing over the next few months. The first few questions are some of the key questions that I’ve been asked regularly around best practices for testing: https://www.youtube.com/watch?v=EGDM7PZdC5k
What’s the most effective method for testing products?
I think what I’ve seen as the most effective thing is thinking about the goals that you’re trying to kind of get out of your testing. I think there are a variety of ways to think about how to test new products. For me, what has been the most valuable way to think about it is, who is the customer and what problem am I trying to solve for that customer? And whether that’s for developing a brand new product or iterating on existing product, it really just comes down to that.
So start from the basis of who is my customer? Then lead into, what is the hypothesis I have about that customer? And then that leads to what sort of tests should I run? Is the hypothesis something simple, that can be easily A/B tested about a user, or is it something more complicated where you actually need to put screenshots or mock-ups or UX in front of a customer before you’re actually going to build it? And I think the key is to figure that out first. Don’t just decide, I only A/B test, I only do mock-ups, I only do customer research. The key is really figuring out what problem you’re trying to solve, and then that helps you understand what sort of testing you should be doing.
What sort of research should companies do in advance of testing their products?
I think it’s really key when you’re doing research to look at two different key things. The first type of research is to understand who your users are. When you’re thinking about what types of testing you’re going to do, understanding who the user is, what the profile or the persona of the user is, is really critical. And then when you have concepts that you want to test with customers, taking those concepts and doing research with those users is also really, really important.
The second really big type of research that is really critical, is actually looking at what your competitors and others are doing in the space, or companies that are similar. So it’s not that you necessarily need to track every single thing or measure yourself to your competitor. But if you see that your competitors are doing something that’s working really incredibly well, there’s no reason that you shouldn’t consider fast copying or quickly figuring out how they built those parts of their product. Then you should consider is it something that would also benefit your customers, and can you improve your user experience by testing or implementing the same idea.
What factors should companies consider when testing their products, and why should they test at a minimum?
The key factors to consider are what are the KPIs, or the metrics, that you’re trying to move. Whenever I think about why I’m testing a product, I’m usually thinking about what are the goals that I have in mind for running the test? In the case of Showmax, it would have been things like, are we trying to improve our conversion of new users? Are we trying to recruit the retention of existing users? In the cases of other products, it might be things like how do we increase our email open rate? How do we increase our email click-through rate? How do we get more engagement with our app, with our product?
But I think the biggest thing that you need to understand up front before you start testing is, what are the goals? What are the metrics that you’re trying to move, and then formulate the test based around what’s most likely to move those metrics. And then have really good tracking on those metrics with each test that you’re running, so you can get a sense for is the test actually impacting the key business metrics or is it not? And if it’s not, then you can quickly say to yourself, “Okay, this test didn’t work. We need to iterate on it and come up with other things that we’re going to run in future tests.”
