How To Create A Powerful Product Experimentation System?
Part 3 of series on experimentation
This Article will cover:
- How to create an experimentation idea backlog?
- How to prioritize the ideas for maximum business value?
- Cross functional alignment with the rest of the business.
- The right data and experimentation tools.
A well designed experimentation system allows a company to accelerate growth by creating faster feedback loops and enabling progressive delivery. It enables a diverse set of ideas from across the organization to be tested systematically and learnings to be internalized by everyone. In addition, it reduces risks associated with large product releases by allowing the team to start with smaller releases and validate the opportunity and risk before extending to the whole user base.
To have a successful experimentation program, the product managers need 3 things: 1. A powerful experiment roadmap with a large idea backlog prioritized for value. 2. Cross functional alignment with the rest of the business. 3.The right data and experimentation tools.
1. A powerful experiment roadmap with a large idea backlog prioritized for value.
First, let’s make sure you have a large idea backlog to draw from.
- Allocate time regularly- The only way to get enough ideas to run a high velocity testing system is to set aside time for it. Whether it is a formalized group brainstorming session with a pre-set agenda or allowing each team member to reflect on their own experiences or do independent research.
- Make lists- One trick that has helped our team a lot is to always keep lists; lists of things that others do that we like, lists of things we wish we could do and lists of things we need to improve. Even though the lists get very long, reviewing them regularly enables us to evolve or amalgamate ideas overtime to create some very successful experiments.
- Ask everyone- The most unique ideas often come from team members who are least informed of the existing biases and limitations. Cross functional brainstorming and sessions with the wider business contributed to more diverse test ideas and even brought up some strategic questions that forced us to re-evaluate our assumptions. To make these sessions most valuable we mixed in some blue-sky brainstorming with some themed discussions and group exercises to generate ideas in a certain area.
- Have a process- It’s one of those things that is easier said than done, but cross functional ideation doesn’t just happen, and it needs a process! Soliciting ideas from a wider audience takes a lot of constant communication, reminders and even incentives. We used to run a weekly contest for the best idea in a given theme.
- Make it accessible- The easier it is for your team to record their ideas as they go, the more ideas you will have. Make the idea repository available to everyone, they might even get inspired when they read ideas others have come up with.
- Give credit- There is nothing worse than having an amazing idea that generates a tonne of value and not getting recognition for it. Even if the experiment is based on an evolution of a few ideas, make sure the contributors for those ideas get the credit they deserve.
- Begin with a real problem- In order to get high value ideas, we always try to start with a problem that our users are facing. It is otherwise very easy to fall into a trap of coming up with ideas that are based on personal preferences and likely not relevant to our customers.
- Test everything! Or close to everything- Sometimes teams disregard ideas because they think “well, I would never do that!” so our customers probably will not react either. This is called the False Consensus effect; we basically believe that everyone else thinks and acts like we do, and our preferences will apply to our users. As smart as we all like to think we are, we do not always know how users will behave and the most non-obvious ideas are often those that set businesses apart.
- Focus on the route cause- Don’t try to fix retention, instead look at the small steps that improve retention and come up with ideas for those. For example, improving sign in experience will ensure customers can comeback with less friction and retain longer.
- Listen to your users- Whether it’s through direct customer research or using the reviews of your product, you can get a tonne of ideas that customers actually care about. Your Customer Support team is also a great conduit of customer pain points and suggestions.
- Get inspiration from others- Talk to other teams that are facing similar questions/ challenges and ask them what ideas they are testing and why. We also regularly look at other companies that are outside our market but face similar challenges and have a similar customer base. We then try to figure out why they are doing certain things and what parallels we can draw in our business to test.
- Be exhaustive with each idea- Whether it is the number of ways to test a hypothesis or the number of variants of each hypothesis, ensure that you are covering as much as possible to maximize outcome and learning. A study by Optimizely has shown that experiments with more exhaustive sets of variants were more likely to have a higher overall insight rate (a 4 variant test has 50% higher rate than a 2 variant test!)
Now you need to meticulously prioritize all these ideas:
- Create a process for prioritization that is easily understood by everyone but leave space for ambiguity and change. Once stakeholders agree with the process of prioritization, it will be easier to agree to the ongoing prioritization.
- Create consistent criteria to measure impact and cost of experiments so they can be easily compared to one another. A lot of teams use the ICE or PIE Frameworks which basically estimate the Impact, Certainty of results and Effort for each experiment and create a combined score. This combined score is then ranked to create the foundation of the priority list
- Remember prioritization is an art and science. It is important to strike a balance between simple, low risk experiments and those that are uncertain but have the potential to substantially move the dial. Some of your experiments may need to run in a certain sequence or at a certain time, so the calculated ICE score should not be the only consideration in your prioritization.
- Your estimates of impact and certainty by definition will be inaccurate, especially when you first start working in any given area. However, this shouldn’t deter PMs from doing their best to create a relative scale of impact for their prioritization score.
- Start the prioritization and alignment discussions as early as possible but don’t commit to a set priority too far into the future. As you learn from the tests and the business evolves, you should strive to be agile and reconsider your priorities continuously.
- Involve all stakeholders in the prioritization, let them see the process, compromises that are made and how changes to the plan are handled.
- Have a living document with the priority but don’t expect the stakeholders to constantly refer to the document. Instead have a cadence for communicating upcoming priorities and any changes in a constant and predictable manner.
- Find a way to make the opportunity cost of each experiment tangible for your team and stakeholders. This can be as simple as talking about the number of hours of development or the user traffic that will have to be dedicated to this experiment. It’s good to have a medium term expectation of the resources required but still keep the prioritization ongoing to adopt to new changes.
- Don’t forget Tech debt and bugs. Even when you are focused on high velocity testing you have to maintain your systems and platform to keep testing in the long run
- Keep a backlog for ideas that don’t make it to the priority list. It might not be the right time for the idea now, but later it may become your top priority.
2. Cross functional alignment with the rest of the business.
Once you have involved your stakeholders in the idea generation and prioritization steps, all that is left is to keep them informed on how the experiments are going and what is next. For us, holding showcases and sending regular updates has created alignment and excitement across other teams. Having a strong executive sponsor who can further represent the experimentation program in leadership meetings and communicate the process and outcomes across the organization can also be immensely powerful in creating a culture of experimentation.
In order to increase trust and transparency, we try to give visibility to the rest of the company on why we are doing certain experiments. It is ok if the reason we are doing a certain experiment is to simply learn about a segment or behavior. In addition, we try to use consistent metrics and methodologies to analyze and communicate the experiment results so everyone can easily follow and compare outcomes.
3. The right data and experimentation tools.
This is where this gets a bit tricky because the type of data you collect and how you organize it will fully depend on your business. As for the experimentation tool, what you use will depend on your resources and priorities. Many large organizations create bespoke experimentation platforms for their own use, however tools such as Optimizely, Google Optimize and Split.io are often all you need to set up a simple experimentation program.
This is the part 3 of a series on Product experimentation and Growth. In the next article I will cover “How to create a culture of experimentation?”
Part 1- The ultimate guide to experimentation for product teams
