Stepping up Your A/B Tests

Mind the Product

This is the first of two posts that deep dive on A/B testing, expanding on a talk I gave at Google Playtime 2016 in London. In this post I share some of the learnings we’ve had after running 60+ A/B tests at Peak , looking at each step of the A/B testing cycle in turn.

Tracking your A/B tests

Mind the Product

This is the second of two posts that deep dive on A/B tests, expanding on a talk I gave at Google Playtime 2016 in London. As mentioned in the first post, if you plan to run a lot of A/B tests then you should think about using a document to store test information, which I call the tracker.

Trending Sources

How to Hook Users in 3 Steps: An Intro to Habit Testing

Nir Eyal

Changing user habits isn’t easy — but understanding how to conduct Habit Testing will increase your odds of success. In this video, I provide a brief introduction to the three steps of Habit Testing.

3 Tests for Great Product Messaging

280 Group

Fortunately Bill is here to provide us with three tests that.[continue The post 3 Tests for Great Product Messaging appeared first on 280 Group Product Management. In this new video and blog post we discuss something that is exceedingly important in the product world: messaging.

Why an MVP Needs to Be a Product and Not Just a Test

280 Group

The post Why an MVP Needs to Be a Product and Not Just a Test appeared first on 280 Group Product Management. Language is dynamic, and terms in product management are subject to the same forces. Meanings evolve, and this can lead to ambiguity.

Daniella Patrick – Innovation Lab Product Development

Mind the Product

At Accenture, Daniella is a part of a design team whose primary goal is to research, ideate, prototype, and test solutions to improve the experience of their candidates, employees, and alumni. Daniella’s team operates with a process of ideating, prototyping, testing, and learning. Testing means they select their population, define their success and failure, and carry out the experiment. As a Product Manager, Daniella’s first lesson she learned is to user test continuously.

Using Experimentation to Drive Product – Stephen Pavlovich (CEO of

Mind the Product ran a test version of a product where they created the levels by usage rather than functionality. The participants don’t even realise they’re in a test and so biases that invalidate the outcomes can’t be established.

10 Ways to Use Feature Flags

Mind the Product

By offering higher risk features to certain groups of power-users or opt-in users only, you can test your new feature on the groups who know and love you best. ProductTank San Francisco Video developers Product Design Product Development Software user testing

Tactics for Redesigning and Relaunching Legacy Products

Mind the Product

Test if you have time: Some people put out beta versions in advance or launch new versions to partial audience segments. Two examples of forgetting missing pieces are not stress-testing code, since new code is often not battle-tested, and not redirecting old links properly.

Level Up Your Product Game Using Empathy

Mind the Product

Jeremy takes some time out from our often highly data-driven testing culture, to show how employing empathy drives you to build products people love , not just stuff they use. A/B Tests and MVPs. Product Design ProductTank Video A/B testing empathy mvp personas user research

Q&A on User Testing

Under10 Playbook

Most product managers recognize the value of user testing. So it is often hard to fit in testing and design iteration before the sprint deadline. Steven Cohn , the Founder of Validately and a user testing expert, gives some practical advice on how to balance the time pressures of product management and conducting proper user testing. (SJ Why should a product team allocate its limited time to user testing? SJ: What are the different testing options?

UX 2

Your survey data is flawed, and what to do about it

Product Beautiful

“…Among 1008 surveys, their test flagged 17% as likely to contain a significant portion of fabricated data. Fortunately, the researchers, Michael Robbins and Noble Kuriakose, point to the likely culprit: “The basis of the test is the likelihood, by chance alone, that two respondents will give highly similar answers to questions on a survey.

Sprint: Solve problems and test ideas in five days

Product Bookshelf

The process moves through identifying questions, creating solutions, storyboarding, prototyping, and testing with customers. Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days by Jake Knapp. Over the course of a week: ideate and flesh out, prototype, and test it.

How to Improve Your Experiment Design (And Build Trust in Your Product Experiments)

Product Talk

Experiment design, on the other hand, is the plan that a product team puts in place to test a specific hypothesis. My concerns are that the format encourages teams to test the wrong things and it doesn’t require that teams get specific enough to lead to sound experiment design. Test Specific Assumptions, Not Ideas. When we test an idea, we get stuck asking, “Will this feature work or not?” The best way to answer that question is to build it and test it.

The Ladder of Evidence: Get More Value From Your Customer Interviews and Product Experiments

Product Talk

But no matter how good you get at collecting specific stories, you still have to test those insights. Too often, when we are ready to test a solution, we put it in front of people and ask, “What do you think?” In this case, you can use existing solutions to test your idea.

Managing Product Teams for Success

Product Talk

The most important habits I like to see are a regular cadence of customer interviews , prototype tests, and assumption tests. Good product teams have a regular cadence of customer interviews, prototype tests, and assumption tests.

When should Product Backlog Grooming Take Place?

Roman Pichler

Note that this approach still assumes that you can collect the relevant feedback in the sprint review meeting, for instance, by carrying out a product demo, usability test, or solution interview. Option 1: In the Sprint Review Meeting.

How Two Companies Hooked Customers On Products They Rarely Use

Nir Eyal

It’s called “The Toothbrush Test.” Larry Page, CEO of Alphabet (the company formerly known as Google), has a quirky way of deciding which companies he likes. According to the New York Times, when Page looks at a potential company to acquire, he wants to know if the product is, like a toothbrush, “something you will […]. The post How Two Companies Hooked Customers On Products They Rarely Use appeared first on Nir and Far.

Focus on the Metrics that Matter: Identifying Your Product’s Key Metrics and KPIs


It can become a sickness when there are too many metrics flying around and some are going up, others going down, and actions are paralyzed because there’s too much discussion about whether some test has illustrated an improvement or not.

How to Be Customer-Centric at Scale


Your world is all about interviews, a/b tests, and nailing a value proposition and user experience that resonates with the market. When you start building a product, you are obsessed with “the customer.”

The T-Shaped Product Manager

Roman Pichler

Taking my product management test helps you with this. Test competing offerings. Balancing Specific and Generic Skills. To do a great job as a product manager or product owner, you require two skills sets: product-specific and generic ones.

Getting the Band Together: How to Get More Development Resources as a Product Manager


A common challenge that Product Managers face is trying to move their product and company forward with limited development and testing resources.

Product Manager vs. Product Owner

Roman Pichler

They would then hand off a requirements specification to a project manager who would work with development and test to deliver the product. As you may know, the product owner originated from Scrum, where the role is responsible for maximising the value the product creates.

3 Best Practices for Adopting Continuous Product Discovery

Product Talk

They interview customers , run usability tests, and conduct A/B tests. Hypothesis Testing. And you can use the other half to test a couple prototypes. Because Frank’s team already had designs, they just started by prototype testing their concept designs.

Is Scrum Right for Your Product?

Roman Pichler

Scrum is a great fit at this stage: Your product will benefit from an iterative, cyclic process that allows you to quickly test an assumption, address a risk, generate new insights, and come up with new ideas. When is Scrum Most Helpful?

UX 92

10 Product Backlog Tips

Roman Pichler

Addressing uncertain items early on allows you to test your ideas, to fail fast, and to learn how to continue. Tip #1: Complement your Product Backlog with a Product Roadmap. Use a roadmap to sketch the overall journey you want to take your product on.

Coping with Product Manager Burnout

280 Group

I could actually run tests to see immediately if.[continue Vision Built on Uncertainty When I started my career as a young programmer I was tasked with bug fixing as a way to learn the code base….I I hated it. It wasn’t challenging enough for me.

Scaling the Product Owner Role

Roman Pichler

The feature and component owners manage their assets: they capture new ideas and requirements, test them using feedback and data, and collaborate with the teams developing the features and components. Scaling and the Product Life Cycle.

How to Prioritize Product Backlogs in Three Easy Steps

280 Group

Prioritize Product Backlogs in Three Easy Steps Using effective, time-tested techniques to prioritize product backlogs is an important step in getting a product to market rapidly that meets customer needs.

How Compare and Contrast Decisions Lead to Better Product Outcomes

Product Talk

The easiest way to do that is to identify the key assumption that needs to be true for each idea to work and then run experiments to test each of those assumptions. To test this: We can prototype the user interface and see how people respond.

The PM’s Toolbox: Product Management Tools and the Problems they Solve


When it comes to studying user-experience and their interactions with your product, UserTesting comes highly recommended due to its robust set of usability-testing functions. It’s no secret that product managers wear a lot of hats.

How To Fix A Product Management Mistake: Diet Pepsi

The Accidental Product Manager

They conducted countless taste tests and tinkered with their Diet Pepsi product development definition all in a quest to give their customers what they wanted. Pepsi’s product managers screwed up Diet Pepsi, can they fix it? Image Credit: Mike Mozart.

The Evolution of Modern Product Discovery

Product Talk

Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days. If we build in usability testing earlier in the development cycle, we are able to answer, “Can customers use it?” It’s helping us test if our solutions deliver on our desired outcomes.

Choosing the Right Beta Sites

Under10 Playbook

The key word in the phrase “beta testing” is not “beta,” it’s “testing.” Beta testing is defined as testing of a release candidate by customers on customer systems.

Andrew Harder – The Seven Best Ways to Screw Up Your User Research

Mind the Product

Andrew Harder, Senior Research Manager at Ebay, spoke to Product Tank San Francisco about how to avoid screwing up your user testing. Andrew spoke about his experience at Nokia: throughout user testing their customers said that text input was very important, leading Nokia to dismiss the first iPhone. Testing to get Blessed. Testing must be an integrated part of the whole product process. Test to get coached: use testing to experiment, learn, and iterate.

Why This Opportunity Solution Tree is Changing the Way Product Teams Work

Product Talk

It’s easy for teams to work along both dimensions of product discovery—working to understand their customer’s context and iteratively testing their ideas—but that doesn’t mean they know how to connect the two sets of activities. We don’t want to test every idea we have.

Agile at Scale – Outcome Driven (or Broken)

Tyner Blain

It was more than just incremental delivery, we really did incorporate agile principles like pairing and demoing, and engineering practices like automated build and test processes, code-reads, design for testing.

Agile 56

Why You Are Asking the Wrong Customer Interview Questions

Product Talk

Your first list probably includes things like interview customers, run usability tests, look at metrics, get feedback from stakeholders, a/b test it, release it only if it beats out the control. I recently asked a woman what factors she considered when buying a new pair of jeans.

Ways to Define Success and Measure Value for your Product Features

bpma ProductHub

Erin McHugh Saif showed us how she used analytics and customer insights to help her tackle a large-scale business problem at TIBCO Spotfire and Sooyoung Kim shared how Rue La La is using Riskiest Assumption Testing to rapidly test ideas with zero engineering effort.

Mindfulness Tips for Product Managers and Product Owners

Roman Pichler

Here is a simple test: Close your eyes and gently observe your breath. And we will struggle to run effective experiments and test new ideas, as we will be tempted to either reject any data that doesn’t support our view, or interpret the test results so that they confirm our opinion.