How To Create A Culture of Product Experimentation?
Part 4 of series on experimentation.
Companies such as Google, Facebook, Amazon and Netflix have famously harnessed the power of an experimentation culture to continuously drive innovation and create a sustainable competitive advantage in their respective markets.
When you adopt an experimentation culture you no longer have to guess what customers want and your insights are even more effective than surveys because you get to see the real behaviour of customers. This means you can save time and money developing features customers really care for and reduce risks of undesirable effects on your customers. Regularly reporting on experiments also helps the whole organization better understand what customers want and to use the behavioural insights as a guiding light when making decisions.
There are a few key fundamentals that need to be put in place before a culture of experimentation can be established in an organization. Here are some of the most important pillars:
1. An engaged executive sponsor- This is an executive leader within the company who is well-versed in experimentation and acts as a champion for the importance of experimentation within the company. The executive sponsor communicates the vision, strategy and process of experimentation through all levels of the organization, creating a culture of transparency and trust. In addition, this person works closely with the experimentation teams to ensure that they are aligned with the overall business objectives, measure results objectively and follow best practices. Since the executive sponsor is responsible for the results of the team, they can also ensure the team has the right roles, skill set and even instrumentation, or create a case for additional investments where required.
2. A well designed Experimentation System- The system includes a strategy, roadmap and prioritization methodology. The strategy enables the team to know what are the main company objectives that they are optimizing towards and how to gauge the success of the tests. The roadmap will allow the team to connect the very large idea backlog and high level business goals, to tests that can influence metrics in the short term and move systematically through a prioritised list of initiatives. Finally the prioritization methodology creates alignment across the organization for why certain experiments were selected to run. Here is a more detailed view of how these aspects of an experimentation system work:
3. Embracing failure and investing to learn- Every business, by nature, encourages an increase in outcome as well as a reduction in risk. To create a culture of experimentation you will need your leadership to be comfortable with taking small risks to either validate that an idea is worth pursuing or learn from the behaviour. Doing Minimum Viable tests allows development teams to invest time and resources only on validated initiatives and reduces risk of unforeseen circumstances.
4. Data driven decision making-Impactful and valid experiments will require accessible data sources to quantify the benchmark performance to measure any experiment’s results against. This data allows the teams to make objective observations from the experiments and correctly project the value of the wins as well as the impact of experimentation as a whole. This also becomes crucial when talking to executives with lots of experience who have an intuitive understanding of customers, as it allows qualitative and quantitative insights to be compared and contrasted.
5. A trained experimentation team — the skills required for proper experimentation is quite broad and team members from the product manager to the designers, analysts and developers will need to be trained in using a singular framework and language. This will ensure the team can engage in ideation and high quality collaborative test design while each specialist contributes their expertise to the experiments.
6. Creative time and psychological safety- People come up with their most innovative ideas when they feel secure and supported in their day to day environment. It is crucial for organizations to create a culture that celebrates employees taking time to think and work on a side project without fear of judgement. Google famously encourages employees to spend 20% of their time on a side project which has brought about some of the most famous Google innovations such as Gmail and AdSense.
7. Company wide excitement and engagement- From idea generation to prioritization and execution, experimentation is a collaborative effort. To increase stakeholder buy in and alignment, it is best practice to hold open brainstorming sessions for interested parties to raise new concepts, vote on priorities and raise any early concerns. Teams often also hold showcases to give everyone a chance to review early designs of experiments as well as final results and learnings. In order to maximize alignment, it is important that all stakeholders are clear on the consequences of experiments and any learnings are shared widely. In addition, it is crucial to let customer facing groups know the exact treatments of variants and their targets so they can quickly alleviate customer concerns or raise issues with the team.
8. Tools & technology- To target segments of customers, gather data from variants and analyze results you will need the right tools. The exact tools will depend on your business, tech stack and even your team’s preference. What we have found useful is to experiment with tools the same way we experiment various ideas and involve all teams who will use the tool to give feedback and input to the final decision of what gets integrated into the system.
It is worth noting that creating a culture of experimentation is complex, requires support from all levels of a business and it can take years!
How does an experimentation culture manifest in an organization?
Do you think you may already have an experimentation culture? Here are some quick tests:
- The leaders are invested in learning by experimentation. They understand that the value of experimentation is in identifying the best approach to a problem and de-risking a solution, so they celebrate wins and learnings (losses). They know the relevant metrics will improve once the team has identified the best levers to impact.
- The teams are empowered to come up with novel new ideas and know that they can earn executive sponsorship by validating the premise using experiments.
- Companies with a highly ingrained experimentation culture usually run bolder experiments and learn faster. Ideas selected for experimentation are at the intersection between a widely accepted good idea and what seems like a really bad idea. These types of experiments yield positive results less frequently however when they do, their impact is considerably larger and create a sustainable advantage for the company.
- Teams know that the secret to their success is receptivity to new ideas, flexibility and adaptability so when they are looking for new team members, they look for people who have shown that they can try new ideas, learn new things and change direction with ease.
One final note is that experimenting and optimizing at scale is only valuable once you have achieved Product Market Fit. If you are not sure that you can maintain Product Market Fit your efforts will be much better invested in getting there first.
This is the part 4of a series on Product experimentation and Growth. Here are the last 3 parts:
Part 1- The ultimate guide to experimentation for product teams
