Growth Experimentation: What Is It and How to Conduct One?

Growth is fundamental to every SaaS business: growth experimentation is a way to turbocharge your growth prospects and drive customer success.

In this article, we’re going to explain what a growth experiment is, how to create one, unpack how they’re implemented, and how they can help you create sustainable growth over time.

Let’s get right into it!

TL;DR

• The growth experimentation process is a systematic approach to testing new ideas. It involves developing hypotheses about what will drive growth, designing and running experiments, and analyzing the results.
• Why is growth experimentation valuable? Firstly, it boosts the velocity of data and research-driven decision-making. A culture of experimentation also helps drive an approach focused on your target audience. Finally, utilizing the right growth experimentation framework can help improve conversion (alongside other metrics) at key touchpoints in your product.
• Let’s break down the process for running a growth experiment. Start with brainstorming ideas for problems to solve.
• Next, prioritize your list of experiments: you won’t be able to tackle everything at once, so optimize based on value, feasibility, and potential impact.
• Once you’ve chosen your top priority experiments, it’s time to design your growth experiment template or framework. Think about your hypothesis, the type of testing to conduct, sample size, experiment duration, and which metrics you’ll track.
• Finally, all you have left to do is actually run your experiment. Don’t forget to carefully analyze your experiment results: it’s that insight that’ll show you tweaks and changes you can make across your product to optimize growth.
• So that’s what it takes to get a growth experiment set up. But how do you ensure your experiments are effective and ultimately successful?
• Dedicated growth teams solely responsible for designing, optimizing, and launching experiments can help turbocharge your growth (rather than expecting busy product managers to do it ‘side of desk’).
• Another principle of success is remembering experimentation isn’t a ‘once and done’ activity. Continuous experiments help shorten the feedback loop and drive success.
• It’s also good practice to clearly segment users, rather than treating them as one homogenous group. It’ll enable you to launch and track more targeted experiments optimized for distinct user groups’ needs.
• To make sure you and your team have confidence in the results of your carefully designed and time-consuming experiments, you should ensure you draw from a statistically significant sample of your user base.
• Finally, make sure you’re documenting and reflecting on each example of a growth experiment you run. By adding a detailed writeup into a tracker document, you’ll be incrementally adding to a valuable source of knowledge – one you can refer back to for years to come.
• Choosing the right tool for the job can help make launching growth experiments far simpler: Userpilot is a powerful, multifaceted product growth tool that can help.

What is growth experimentation?

Growth experimentation is a systematic approach to testing new ideas and strategies to grow a business. It involves developing hypotheses about what will drive growth, designing and running experiments to test those hypotheses, and analyzing the results to make informed decisions about how to move forward.

This careful, data-driven approach helps you make effective decisions – rather than fully committing off the back of an educated guess.

Growth experiments are a key component of any successful growth strategy. By taking a systematic approach to testing and learning, businesses can increase their chances of success in the long term.

Why carry out growth experiments in your company?

First things first, let’s take an in-depth look into why running multiple experiments on growth makes sense.

Enables decision-making backed by a scientific method

Rather than making decisions based on hunches and intuition, you make decisions based on experiment results. Ultimately, it’s a way to give structure to your decisions. You also reduce the inherent risk of making a decision and it failing.

Each experiment represents progress and greater understanding. Over time, new experiments can help you build valuable insights that give you a strong foundation of knowledge to draw from: user behavior, expectations, preferences, what makes them buy-in, engagement with new features, and more.

All of that can help inform future strategies for you and your engineering team to consider when pulling together your product roadmap.

Ensures companies follow a customer-centric approach

Growth experiments are based on resolving customer problems and pain points. This puts the customer in the front seat, helping you form strategies that satisfy their needs.

The better your product meets user needs (i.e., the more customer-centric it is), the more successful it will be.

Experiments help keep thinking fresh by encouraging an ideation mentality. You will generate concepts you might plan, discuss, and implement before you ultimately determine the right course of action.

Optimizes conversion rates at important touchpoints

As a product manager, the analysis of key metrics is crucial.

Growth experiments can focus you and your team relentlessly on rapid learning and problem-solving.

By consistently solving customer problems, you’ll inevitably boost your conversion rate. Let’s say your onboarding needs improvement. A growth experiment might help run several tests on an onboarding process that’ll improve the free-to-paid conversion rate.

What is the growth experimentation process?

Next up, we’re going to make sure you have a complete understanding of the growth experimentation process – one step at a time.

1. Brainstorm the customer problems you want to explore

First things first, you need to start with a series of ideas.

You and your team should brainstorm potential tweaks, changes, and ideas for experiments you want to run. It’s good practice to make sure each idea is focused on impacting a key part of the growth process (i.e., things like acquisition or retention).

Where do those ideas come from?

You can gather them directly from your customers. Feedback will shine a light on where to focus your efforts (although there’s no harm in widening your research too).

2. Prioritize the many growth experiments you plan to conduct

If everything is a priority, nothing is a priority.

You’ve got to choose which growth experiments to conduct. How you do that is up to you, but typically you’ll want to choose hypotheses based on factors like reach, effort, feasibility, and potential impact.

Prioritization frameworks like value and effort or RICE can help you come to structured decisions on experiments that map to your goals.

3. Create the experiments to test

Now, we go from idea to action: you’ll create detailed experiment plans outlining exactly what you want to test and how you want to test it.

A plan should include:

• Type of testing (i.e., A/B testing or multivariate testing)
• Success criteria (i.e., metrics to measure)
• Duration
• Sample size

Once you’ve fleshed out these details, you’ll have a much more accurate view of which tests you want to run.

Userpilot’s advanced A/B testing functionality will be available in Q4 of 2023 – one powerful hub to design, build and launch tests.

4. Run experiments on your target audience

Now it’s time to execute the experiments you’ve been carefully planning for your target audience.

Whether that’s making changes to your website, marketing campaign, pricing models, or other areas of your product itself, the key here is to track your results.

Without gathering that data, you’ll struggle to understand how the tweaks you’ve made impact your customers’ behavior.

5. Analyze growth experiments to gather valuable insights

You’ve come up with an insightful idea. You’ve prioritized the experiment you want to run, and your teams have fleshed out the plan. Your experiment’s gone well. What next?

The final step is to analyze your experiment results.

This step will show you which variations and changes you’ve made have had the most tangible impact on your chosen success metrics. The data will show you patterns, trends in behavior, and useful insights that’ll help inform product decisions in the future.

What is an example of a growth experiment for a SaaS company?

We’ve covered the key steps in putting a growth experiment together. Now let’s look at an example you can use as a growth experiment template:

• Experiment: An A/B test looking at introducing a checklist to the onboarding process.
• Step of the pirate funnel: Activation.
• Hypothesis: Visitors are more likely to get activated when provided with a checklist compared to the traditional onboarding process. The assumption is that a checklist will increase the activation rate.
• Running the experiment: Use Userpilot to create an onboarding checklist and then trigger it for 50% of your new users. The other 50% of users will see the existing onboarding flow.
• Duration: Run the experiment for at least two weeks to gather enough data.
• Analysis: The hypothesis is validated if the activation rate is 30% higher using the checklist versus the traditional flow.

Tips for creating a successful growth experimentation framework

You’ve now got a solid idea of what it takes to set up a growth experiment. But how do you make it successful? Let’s find out!

Create a growth team responsible for conducting experiments

For any business that’s achieved lasting success, there’s a common attribute: the leadership usually organizes resources to form a focused growth team.

These might take different forms: in a smaller company, you might just have a specialist product manager or analyst rather than a specialized team.

But the point is that dedicated growth teams (and a growth lead) make sure growth experiments don’t fizzle out. One growth experiment should lead to the next, and the next, to achieve growth and drive success.

Carry out growth experiments continuously

Growth experiments should trigger the continuous generation of ideas and execution.

In a nutshell, if your experimentation business strategy relies on you launching just one growth experiment and expecting that to make the difference, it’ll fail.

You should conduct experiments continuously: a structured plan setting out when experiments should be carried out can help keep the momentum up for you and your business.

Segment your customers before you test them

Your customers aren’t one homogenous group. Most SaaS companies cater to a diverse user base with different needs, preferences, and behaviors. Segmentation allows you to tailor experiments to specific user groups, making it far simpler to uncover insights that are relevant to each segment.

Remember, segmentation also maximizes the potential impact of your experiments!

Choose a sample size with statistical significance

Not all experiments are equal. There’s a difference between a biased experiment chatting to a handful of users and a statistically significant experiment that gives you rock-solid insight.

Statistical significance helps you answer whether the observed differences in outcomes are likely due to the changes you made (in the treatment group) or if they could have occurred randomly.

To choose a statistically significant sample, you need to consider the following factors:

• Population size: The larger the population size, the larger the sample size you need.
• Confidence level: The higher the confidence level, the larger the sample size you need.
• Margin of error: The smaller the margin of error, the larger the sample size you need.

They’ll help you determine an appropriate sample for your experiment.

Develop a growth experimentation tracker document

You don’t want to let knowledge slip by the wayside. Each growth experiment you run should build a knowledge repository.

Create a tracker that helps you to document experiment details. Make sure to include your hypotheses, variations, success criteria, metrics, and perhaps most importantly results: what have you learned?

Conclusion

Hopefully, you now have a comprehensive understanding of exactly what a growth experiment is, the process for putting one together, a handy growth experiment template you can use, and tips to make your experiments a success.

Want to conduct growth experiments code-free? Book a demo call with our team and get started! Check out the banner below for more information.