How to select the next best test from the Experiment Library

David J. Bland
August 27, 2020
#
 min read
topics
Testing Business Ideas
Innovation Metrics

We are excited about the popularity of Testing Business Ideas. One of the main reasons we wrote the book was to get beyond just a list of experiments. Simply running a bunch of experiments to see what happens will create an illusion of progress. This can actually hinder your efforts in de-risking the business idea. Matching an experiment to the type of risk in your business strategy is still difficult for teams to grasp at the very beginning, but with some help, you can pick it up in no time.

What are the principles of selecting an experiment?

Before jumping to an experiment, it is important to consider the key principles of rapid experimentation when testing a new business idea.                  

  1. Go cheap and fast early on in your journey - Don’t spend a lot of money if possible early on. You are only beginning to understand the problem space, so you don’t want to spend money when you can learn for free. Also try to move quickly to learn fast, instead of slow and perfect.

    Quite often, I am asked “how many experiments should I run”? This is a very hard question to answer as it depends on so many variables. But a general rule of thumb is 12 experiments in 12 weeks. 1 experiment a week is a good pace to keep up momentum and 12 data points is typically a good checkpoint to come back and reassess your business model.

  2. Increase the strength of evidence with multiple experiments for the same hypothesis - Don’t hesitate to run multiple experiments for the same hypothesis. Rarely do we witness a team that only runs one experiment and uncovers a multimillion dollar opportunity. The idea here is to not get too excited or too depressed after running one experiment. Give yourself permission to run multiple to understand if you have genuinely validated your hypothesis.

    Remember, a typical business model may have several critical hypotheses you need to test and for each hypothesis, you may need to run multiple experiments to validate it. Refer to our previous blog post on Assumptions Mapping to learn how to define your hypotheses and determine which ones to run first.

  3. Always pick the experiment that produces the strongest evidence, given your constraints - Not every experiment applies to every business. B2B differs from B2C which differs from B2G. A 100 year old corporation’s brand is much more important than a 100 hour old startup’s brand. Pick an experiment that produces evidence, but don’t risk it all. Make small bets that are safe to fail.

    When working with corporate innovators, the phrase I most often hear is “we can’t do that”. Innovators often feel hamstrung when working in heavily regulated industries. Many may choose to bypass testing certain hypotheses because of the constraints. This is not something I would recommend. If you really cannot test a hypothesis, the last resort would be to consider pivoting your business model. Remember, a testable idea is always better than a good idea.

  4. Reduce uncertainty as much as you can before you build anything - In this day and age you can learn quite a bit without building anything at all. Deferring your build as long as possible, because it is often the most expensive way to learn.

    In the Experiment Library we’ve compiled a list of creative ways to demonstrate a prototype without building out your final product. Experiments such as Wizard of Oz, Clickable Prototype and Single Feature MVP.
Experiment Library PNG

Discovery or validation, what’s the difference?

Pulling inspiration from Steve Blank’s 4 Steps to the Epiphany, it is quite obvious that we framed the experiment library in a similar fashion. One of the biggest differences however is that instead of “customer discovery” and “customer validation” we decided to test the entire business model. Discovery experiments are often open ended and directional, whereas the Validation experiments have more of a true value exchange. Most teams start with discovery since the validation experiments cost more and take longer.

Here is our one page cheat sheet to the Experiment Library.

You will learn about templates and more by signing up for our master workshop on Testing Business Ideas here

related reads

No items found.

About the speakers

David J. Bland
Founder & CEO of Precoil, author and advisor

David J Bland is an author and founder based in Silicon Valley. In 2015, he founded Precoil to help companies find product market fit using lean startup, design thinking and business model innovation. David has helped validate new products and services at companies such as GE, Toyota, Adobe, HP and Behr. David is the co-author of Testing Business Ideas, a Wiley business book with Alexander Osterwalder.

by 
David J. Bland
August 27, 2020
Share

Download your free copy of this whitepaper now

  • 1
  • 2
  • 3
Team member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatar
Let's talk
Whether you’re looking for more information or you’re ready to start a project, we’re ready to help.
Thanks for your interest in our solutions. We will be in touch with you soon.
How to select the next best test from the Experiment Library
Methods

How to select the next best test from the Experiment Library

How to select the next best test from the Experiment Library
Methods

How to select the next best test from the Experiment Library

August 27, 2020
#
 min read
topics
Testing Business Ideas
Innovation Metrics

We are excited about the popularity of Testing Business Ideas. One of the main reasons we wrote the book was to get beyond just a list of experiments. Simply running a bunch of experiments to see what happens will create an illusion of progress. This can actually hinder your efforts in de-risking the business idea. Matching an experiment to the type of risk in your business strategy is still difficult for teams to grasp at the very beginning, but with some help, you can pick it up in no time.

What are the principles of selecting an experiment?

Before jumping to an experiment, it is important to consider the key principles of rapid experimentation when testing a new business idea.                  

  1. Go cheap and fast early on in your journey - Don’t spend a lot of money if possible early on. You are only beginning to understand the problem space, so you don’t want to spend money when you can learn for free. Also try to move quickly to learn fast, instead of slow and perfect.

    Quite often, I am asked “how many experiments should I run”? This is a very hard question to answer as it depends on so many variables. But a general rule of thumb is 12 experiments in 12 weeks. 1 experiment a week is a good pace to keep up momentum and 12 data points is typically a good checkpoint to come back and reassess your business model.

  2. Increase the strength of evidence with multiple experiments for the same hypothesis - Don’t hesitate to run multiple experiments for the same hypothesis. Rarely do we witness a team that only runs one experiment and uncovers a multimillion dollar opportunity. The idea here is to not get too excited or too depressed after running one experiment. Give yourself permission to run multiple to understand if you have genuinely validated your hypothesis.

    Remember, a typical business model may have several critical hypotheses you need to test and for each hypothesis, you may need to run multiple experiments to validate it. Refer to our previous blog post on Assumptions Mapping to learn how to define your hypotheses and determine which ones to run first.

  3. Always pick the experiment that produces the strongest evidence, given your constraints - Not every experiment applies to every business. B2B differs from B2C which differs from B2G. A 100 year old corporation’s brand is much more important than a 100 hour old startup’s brand. Pick an experiment that produces evidence, but don’t risk it all. Make small bets that are safe to fail.

    When working with corporate innovators, the phrase I most often hear is “we can’t do that”. Innovators often feel hamstrung when working in heavily regulated industries. Many may choose to bypass testing certain hypotheses because of the constraints. This is not something I would recommend. If you really cannot test a hypothesis, the last resort would be to consider pivoting your business model. Remember, a testable idea is always better than a good idea.

  4. Reduce uncertainty as much as you can before you build anything - In this day and age you can learn quite a bit without building anything at all. Deferring your build as long as possible, because it is often the most expensive way to learn.

    In the Experiment Library we’ve compiled a list of creative ways to demonstrate a prototype without building out your final product. Experiments such as Wizard of Oz, Clickable Prototype and Single Feature MVP.
Experiment Library PNG

Discovery or validation, what’s the difference?

Pulling inspiration from Steve Blank’s 4 Steps to the Epiphany, it is quite obvious that we framed the experiment library in a similar fashion. One of the biggest differences however is that instead of “customer discovery” and “customer validation” we decided to test the entire business model. Discovery experiments are often open ended and directional, whereas the Validation experiments have more of a true value exchange. Most teams start with discovery since the validation experiments cost more and take longer.

Here is our one page cheat sheet to the Experiment Library.

You will learn about templates and more by signing up for our master workshop on Testing Business Ideas here

related reads
Tools
The Experiment Library
Books
Testing Business Ideas
How to select the next best test from the Experiment Library

We are excited about the popularity of Testing Business Ideas. One of the main reasons we wrote the book was to get beyond just a list of experiments. Simply running a bunch of experiments to see what happens will create an illusion of progress. This can actually hinder your efforts in de-risking the business idea. Matching an experiment to the type of risk in your business strategy is still difficult for teams to grasp at the very beginning, but with some help, you can pick it up in no time.

Thanks for your interest in 
How to select the next best test from the Experiment Library
How to select the next best test from the Experiment Library
ONLINE COURSe

Read more
Team member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatarTeam member avatar
Let's talk
Whether you’re looking for more information or you’re ready to start a project, we’re ready to help.
Thanks for your interest in our solutions. We will be in touch with you soon.