My Top 5 challenges of Product Management and Development — Part I
Measuring Product Development
Note: this is Part I of an (I hope) 5 Part series
Product Management and Development are quite hard.
For the last few years, I have consolidated a few of the things that I struggle the most (and see others struggling as well). So I thought I might as well share it and see if anyone can share tips :)
1. How to measure Product Development
Even when there are tons of things to measure in product development, there are three big grouping topics that I believe to be the most important ones:
- How much value you added — and whether it was the most value it could be added (discovery efficiency and opportunity cost)
- How much it took / cost to produce — and whether it was efficient or not
- How much it costs to operate it with good SLAs — and how that costs and SLAs compare to other competitors in the market
Those are not straightforward things to measure, and each has its own challenges.
How to measure value added
We do have concrete ways to measure economic value and even some other customer value metrics such as conversion, engagement, satisfaction, etcetera.
The problem is those are affected mostly by what everyone in the company does. The most obvious players affecting those may be sales, marketing, and customer service. How do you isolate the value that the product team has added versus how other teams affected those metrics?
We do have A/B testing, so we can measure the impact that a new deployment had. But A/B tests have disadvantages, like needing a lot of traffic to detect lower impact changes. Furthermore, we are not going to be able to use it for everything or in every product.
And even more difficult, there are things that we should do to have a great product but are hard to measure in terms of value -like moving a few pixels of a misaligned field on a web page-.
How to measure efficiency
On the other hand, if you give a feature requirement to N different teams you will end up with N different implementations and N different costs and times. Just to give a quick example, teams that work with different technology stacks will certainly face different challenges and thus have different times and costs to implement your feature.
But there are many subtler things to consider:
- A team may be faster but may produce software of lower quality or harder to maintain.
- A team adding that feature in a legacy system may have to work around a lot of “regression” stuff that a team with a newer product may not face.
- A team may be “fast” but due to some organization issues may face dependencies or other handoffs that slow their delivery.
It is really hard to measure if a team was fast or efficient. And if it’s hard to measure by experts, consider how hard it will be to numerically tell stakeholders whether you are fast or not.
How to measure optimal operation
Finally, optimal operation does have measuring challenges as well. There are tons of metrics and KPIs, starting in costs and SLAs or SLOs, but also including average response times, the number of incidents per period of time, mean time to resolution (MTTR) of incidents, and many more.
Those do have some complexity (for instance in a complex system with many moving parts how do you quantify the degradation of one component in terms of overall system SLO?). But the most challenging aspect is comparing to others.
Are your infrastructure costs adequate for your business, size, and systems? Do other competitors or players in the industry achieve better results? Can you compare cost as the percentage of revenue with other companies? The same can be said for the other set of metrics. Response time may be the easiest to measure, but MTTR or number of errors it usually isn’t public information to be compared.
That’s it for today! I would love to hear feedback (or claps!) and your tips on some of these challenges.
Update: PART II is ready, read more here!
If you enjoyed it and want to receive more tools & tips to improve your product, you can subscribe here and join hundreds of readers of my Lean Experimentation blog!
