In this ProductTank San Francisco talk Jon Noronha, then Senior Director of Product Management at Optimizely, what he’s learned about the benefits of product experimentation and how to avoid the common pitfalls that may lead you astray.
Watch the video to see the talk in full, or read on for an overview of the key points:
- Product experimentation can support profitability
- Many things that can go wrong when applying an experimentation mindset to product
- There’s much to consider to ensure that your experimentation is implemented correctly – iterate on your metrics, know your numbers, and test
Product experimentation delivers value
In 2011 Jon Noronha joined the product team for Microsoft’s Bing search engine at a time when the product, as he puts it, was “in a big hole”. As the years went by, however, that turned around, and in late 2015, Microsoft reported that Bing had become profitable. Jon and his colleagues at Microsoft attribute that growth to a change towards experimentation. As the then Microsoft GM of Analytics and Experimentation, Ronny Kohavi commented: “The growth of experimentation is the major reason Bing is profitable and its share of US desktop searches nearly tripled.”
Product experimentation Pitfalls
Seeing the value of product experimentation first-hand, Jon moved from Microsoft to the experimentation platform Optimizely, where he is now Director of Product. Jon learned that though powerful, there are many things that can go wrong when applying this mindset to product. He shares three common pitfalls:
1. Testing with the wrong metrics
At first, Jon and team focused their product experiments at Bing on driving more searches per user. The more time a user spends on your page the better, right? Not necessarily when you’re a search engine. After some tests around this metric, they figured out this was actually counter to what users really want and what the market leader (i.e. Google) was focusing on. Instead, they decided to try decreasing searches per session but increasing sessions per user. These metrics in tandem got the user to what they were looking for faster and meant they turned to Bing to get them what they needed.
To avoid this pitfall, Jon recommends:
- Put yourself in the users’ shoes. What if they knew what your goal was? Would they be aligned with that experience?
- Think about your goal as part of the big picture. If this metric went up and everything else remained flat, would this be good for your business?
- Constantly reevaluate.
2. Being mislead by statistics
Peeking at results too soon or getting fooled by false positives can mislead your direction. If you’re testing 20 variations with a 5% chance of a false positive, you need to consider that at least one result may be incorrect.
To avoid this pitfall, Jon recommends two things:
- Involve a professional who knows the numbers — have a data scientist or statistician join the team
- Use a testing platform that’s built to consider nuances in the data
3. Testing too few variations
We tend to focus on A/B testing, but at Optimizely, Jon and team have found that testing many variations is more likely to lead you to a significant winner. On average, two variations lead to a significant winner 14% of the time while five variations lead to a winner 27% of the time.
To avoid this pitfall, Jon recommends that, if you have the resources, test more than just A and B. This helps the team think out of the box around other possible solutions.
As Jon shared, experimentation is a valuable tool in the product managers’ tool kit, but there’s a lot to consider to make sure it’s implemented correctly. Iterate on your metrics, know your numbers and test! Check out Jon’s talk to hear more stories about product experimentation. You can also read Jon’s article, How do you Implement Effective Product Experimentation?
Enjoy more from ProductTank
ProductTanks are informal meetups, created by Mind the Product, to bring local product people together and to enable speakers to share amazing product insights. Today we have ProductTanks in more than 200 cities across the globe and there’s probably one near you.
Explore more ProductTank talks
Comments 1