Six Common KPI Mistakes to Avoid

Key performance indicators (KPIs) are metrics that measure how well your product is doing. As useful as they are to proactively manage a product, they are not always effectively applied. In this article, I discuss six common KPI mistakes. I explain how you can overcome them and leverage key performance indicators to maximise the value your product creates.

Listen to this article:

1 No Product KPIs

Imagine that you have trained hard to run a half-marathon. Shortly after the start of the race, your smart watch stops working, and you discover that you didn’t bring your phone. Consequently, you don’t know for sure how fast you are running and if you are on track to achieve your target finish time. The same is true when you don’t use any key performance indicators. You’ll end up guessing how well the product is doing and if it is creating the desired value.

While common sense suggests that managing a product without the right measurements is not a sensible approach, I’ve seen product teams who did not use any KPIs. This can be caused by an intense focus on execution and delivery—being so concerned with adding features and running sprints that tracking the product’s overall performance is neglected. Consequently, these teams relied on:

• Anecdotal feedback: “Customers love our product, they told me so.”
• Gut feeling: “Trust me, I’ve seen this before, and I’m sure we’re on the right track.”
• Solution-centric data: “We’re making great progress; we’ve implemented 50 more user stories, and velocity is up by eight points!”

Sadly, the data above is not helpful to see clearly how much value the product is creating. Instead, it can lead to making wrong product decisions and ultimately mismanaging the product. 

2 Wrong Product KPIs

When you reflect on the key performance indicators you use, how confident are you that you have chosen the right metrics and that you collect the right data? My experience suggests that it’s rather common that not all indicators used are helpful. There are four common reasons for this:

1. The analytics tool decides: The measurements are largely determined by the analytics tool employed—you trust the tool to collect the right data for you. This often leads to too much data being gathered. You consequently spend too much time analysing the data, and you may struggle to determine the relevant data. This, in turn, can cause you to draw the wrong conclusions and make the wrong decisions.
2. The value the product should create is not clearly understood: A validated product strategy and an actionable product roadmap are missing. You therefore end up guessing which indicators you should use rather than being able to systematically derive the right ones.
3. A powerful stakeholder or line manager determines the KPIs—not the person in charge of the product. This usually leads to using metrics that are valuable for the individual but not necessarily for the product. In the worst case, you collect irrelevant data that unduly influences product decisions.
4. Vanity metrics are used. These are indicators that make the product look good rather than paint a realistic picture of its performance, as I’ll discuss in more detail in one of the following sections.

Using wrong or unhelpful indicators means that you will collect wrong or irrelevant data. If this data is actioned, bad product decisions will be made. Therefore, ensure that all measurements you use are truly helpful. To achieve this, refer to the needs and business goals stated in the product strategy and the product goals on the product roadmap. Then ask yourself how you can tell that these goals have been met. Additionally, include health indicators, metrics that measure how healthy your product and team are, as I explain in more detail in the article How to Choose the Right KPIs for Your Product. Don’t forget to regularly review and adjust your KPIs. Do this at least once per quarter, as a rule of thumb, ideally as part of the product strategy reviews.

3 Stakeholder or Big Boss Dictates KPIs

In theory, the key performance indicators should be systematically derived along the lines just mentioned. But in practice, that’s not always the case. I have worked with product people who were told to use certain KPIs by a powerful stakeholder or their boss.

If that’s the case for you, then you may not be fully empowered. As the person in charge of the product, you should have full-stack ownership of the product. You should possess the authority to ultimately determine which KPIs are used, and which ones aren’t—even though I recommend involving key stakeholders and development team members in the decision-making process. If you feel that you lack empowerment, consider how you can increase your authority. My article Boost Your Product Leadership Power will help you with this.

Additionally, have the courage to take ownership of the KPIs. Learn to effectively say no to stakeholders and line managers without losing their support, as I explain in the article 5 Tips for Saying No to Stakeholders. It would be a mistake to use KPIs only to please powerful individuals. Your job as the person in charge of the product is not to make the stakeholders happy but to maximise the value your product creates for the user, the customers, and the entire business.

4 Vanity Metrics

Everybody who is committed to their product wants it to be successful and do well. Nobody wants their product to fail and be retired early. It can therefore be tempting to choose measurements that paint a rosy picture of the product performance rather than a realistic one. These KPIs are also referred to as vanity metrics, a term coined by Eric Ries.

Examples of vanity metrics might be number of downloads and page views. Both measurements might look reasonable, but they usually don’t allow you to generate new insights and make the right product decisions. If the number of downloads is increasing, then this does not necessarily mean that people actually use the product. It would therefore be better to measure activations and daily active users. Similarly, if the page views are up, then this does not imply that people find the content offered valuable and act on it. It may therefore be better to track conversion rate.

To mitigate the risk of using vanity metrics, ensure that your KPIs are actionable, accessible, and audible:

• Actionable: The metrics help you understand if the product is creating the desired value for the users and customers and for your business. The data they collect helps you make concrete decisions about how to best progress your product; it allows you to inspect and adapt the product strategy and product roadmap.
• Accessible: The data can be collected in a timely manner without too much effort. Ideally, most of the data is automatically gathered by an analytics tool.
• Auditable: The data can be clearly traced back to its source. It is transparent where the data came from. This allows you to assess the quality of the data and determine if it is relevant and should be used to drive product decisions.

5 Biassed Data Analysis

KPIs help you make better decisions by using empirical evidence. But data alone is not useful; we have to analyse and interpret it and draw the right conclusions from it. That’s not always easy, though. As humans, we are affected by cognitive biases. These impact our ability to objectively work with data and cause us to make wrong product decisions. Here are three biases, which I find especially prevalent in product management:

• Confirmation bias causes you to prefer data that confirms preconceived ideas and views. In the worst case, you ignore data that challenges your opinions, and you miss out on the opportunity to gain new insights, make the right decisions, and improve your product.
• Anchoring bias is the mistake of relying too heavily on one piece of information when making decisions. Often that’s the first piece of information you acquire: You don’t wait for all relevant data to become available and draw your conclusions too early. This can cause you to make wrong and suboptimal decisions.
• Authority bias means giving too much weight to the opinion of an authority, for example, an important customer, an influential stakeholder, or a senior manager. This can cause you to be swayed by the individual’s views instead of critically assessing their relevance. Consequently, you might make a bad product decision.

To mitigate cognitive biases, I recommend the following two measures:

1. Be mindful of the way you work with and interpret data. Don’t cling to your ideas and views. See difficult feedback and critical data as something positive, as an opportunity to learn and improve. Remember: You cannot advance your product if you don’t discover any issues and shortcomings.
2. Involve key stakeholders and development team members in the data analysis. This allows you to leverage the individuals’ expertise, and it helps counteract individual biases.

6 Data Worship

“In God we trust. All others must bring data,” is an adage commonly attributed to Edwards Deming. While I hope that you’ll agree with the intended meaning of the phrase, it would be a mistake to expect that data will tell you what to do. As I pointed out above, data alone does not say anything. It’s your analysis and the conclusions you draw from it that make it valuable.

Therefore, be data informed, not data driven. Collect the relevant data using the right KPIs and do your best to objectively analyse it. But make sure that you clean the data and check that it is relevant. Have the courage to discard data that is poor quality and cannot be used. The data you base your decisions on should be:

• Representative: The data is valid for your product’s entire target group, that is, all users and customers, as well as the entire business.
• Reliable: It originated from reliable sources that can be identified.
• Relevant: The data is up to date and useful to determine how much value the product is creating for its target group and the business.

Additionally, don’t work exclusively with quantitative data. Make sure to collect qualitative data by directly interacting with (selected) users and customers, for example, by interviewing the individuals and by directly observing them. This allows you to empathise with the beneficiaries of your product, and it helps you develop a deep understanding of their needs—which will improve your ability to draw the right conclusions from the quantitative data you gather.