Beyond Vanity Metrics
In defense of unmeasurable success criteria
I just returned from Wisdom 2.0. The conference that brings together the technology community with the mindfulness community. As part of this experience, I had the pleasure of seeing Tristan Harris of the Center for Humane Technology speak. And I was inspired to write this post that has long been on my list.
We are told as PMs to be data-driven. To focus on metrics. To have KPIs. And I agree those are all “good things”. But I worry this need to have measurable success criteria causes us to forget the bigger picture. The “why” of our products.
Vision
This may not be universally true, but most companies are in business to make the world a better place, not just to make money.
One step down from that is why your particular company/product exists. Your vision statement. Hopefully you have one of those. If not, go get one and come back. I’ll wait (cue Jeopardy music).
What does success look like?
But even if you do have a clear vision, the trend seems to be toward establishing KPIs that have little to do with it. The vanity metrics that measure your usage as opposed to how well you are achieving your make the world a better place goal.
That, plus the trend (fast-approaching religion) of being “data-driven”, is resulting in less value creation. What are we creating instead? Mildly addictive features that drive up KPIs without adding actual value.
Just because your product is being used more, does not mean your users are extracting more value from it.
The Like Button
The Like button on Facebook is fast becoming the canonical example. The company’s new mission is to increase meaningful social interactions. Who can argue with that?
The question is, does the Like button aid this mission? Maybe the sad button does because I’m communicating I’m feeling sympathy? Is that “meaningful”?
The value Facebook users talk about is seeing friends’ updates from around the country, keeping up to date on what family members are up to, etc. We don’t talk about how much value we get out of checking our posts to see how many Likes we got while we are waiting in line at the grocery store.
Meaningful Interactions
I don’t know whether Facebook is actually able to measure meaningful social interactions. The PM in me is kind of dying to know if there is some combination of metrics that they have up on a dashboard in the halls of Facebook that track this seemingly unmeasurable thing. But I 100% applaud them for having it either way.
I believe that having this intermediate step between vision and KPIs will change the product over time. In fact, if they are measuring it, I think they should stop. It’s not really about it being measurable. It’s about keeping it top of mind, day in and day out, while you make all those hundreds of decisions you need to make to deliver products.
Anything you actively and regularly strive for will improve, even if you can’t necessarily prove it.
While we are on the topic of Facebook, it’s also interesting to note that this mission they have is relatively new. If Facebook had started with that mission, would the current product have been the result? I think this means that it is going to take longer for the company to pivot to this new definition of success. But again, so glad they are trying.
Apple Maps
While I was at Apple Maps I pushed hard to stop people from focusing on app usage. Our vision as an org was to enable Apple users to navigate the world around them. Not Apple Maps users. Apple users.
This lead us to actively create features that obviated the need to launch the app. Features like alerting you when a major accident happened and you should take a different route home. Or swiping left and seeing quickly what traffic is like right now on your commute, saving you from launching the app to check traffic.
There is no way to measure how much value these features added. And if we were solely focused on app usage, we never would have done them.
It’s easier to do this at Apple, granted. Apple is well known for using intuiton more so than being data-driven (which aren’t that different, but I digress…). They are an example of a company that is focused on doing the right thing for its own sake. And assuming the money will follow. It’s probably as close as you can get to PM heaven. But even at Apple the pull of what’s measurable can be felt.
Call to Action
I believe that the answer to Tristan’s call for more humane technology lies in the hands of PMs. We are at the center of the products that are being delivered today, which is a powerful place to be. And with power comes responsibility.
So instead of focusing on driving up new users or active users or time spent in app (or any other “vanity” metric), what is the unmeasurable KPI that your company should be talking about?
What would happen if you looked at every feature you are planning and measured it against its ability to deliver on this fuzzier definition of success?
You might end up with (gasp) less usage. Perhaps though, you will have succeeded in making the world a better place.
Good luck and happy innovating!
