Data-driven Product Management — Setting up User Analytics
A quick start guide based on my experience in setting up a tracking and analytics framework for an early-stage consumer tech company.
The first step towards a Data-driven mindset is to start collecting data. In this post, I write about the basics of “What to track” along with insights on selecting the right tools and metrics to analyze user data.
What is User Analytics?
User Analytics refers to the tracking of user interactions on digital products and the analysis of the resultant data-sets. The term is used interchangeably with other loosely defined terms such as ‘customer analytics’, ‘product analytics’ or even ‘clickstream data’.
Setting up and leveraging Product Analytics is the most impactful way to become data-driven. While a PM can (and should) access other forms of data such as customer care queries, reviews on public digital spaces, surveys, internal data, benchmarking and user research — Product Analytics has the widest reach and the largest number of data points (more relevance with more scale)
What should be tracked?
The first step towards collecting actionable data from users’ interactions on your product is to define the ‘Tracking Implementation Spec’ which is a carefully named list of user interactions and associated data points to be tracked. As a PM, this first step is the most critical part of your entire Product Analytics universe. Mistakes or omissions at this stage effectively mean you collect data in a format that may make it incredibly difficult to derive meaningful insights from. You would also miss out on collecting data you may need later. Take extra care in defining each name (including upper/lower case considerations) since the names form the basis of analytics on the tracked data.
This ‘Spec sheet’ primarily consists of 3 components — Events, Event Properties, and User Properties. (Examples used hereon refer to a generic E-commerce website with nomenclature based on Mixpanel’s)
1- Events: Every single user interaction on your digital product can be tracked — including clicks, hovers, scrolls, touches, app-opens, and mouse-cursor positions. Each of these interactions is called an Event.
A few insights on defining Events;
- Events should be aptly named to convey the interactions. Eg- ‘Opened Search Window’, ‘Performed Search’, ‘Viewed Product Page’, ‘Added To Cart’, ‘Initiated Payment’, and ‘Order Successful’ are all valid event names that intuitively communicate the user’s interaction. Usually, it helps to name events as a [Verb+Noun].
- Events should be generic and they should describe the interaction independent of added context or history. Eg- If the Product details page can be accessed from a home page banner, or a search result, or a category-page — it should still be a single event called ‘Viewed Product Page’ instead of 3 separately named events such as “Clicked Product on Home page”… and so on. The path to the product page can be captured with Properties instead (explained below). Once you make an initial list of events, try to merge events that effectively mean the same thing to a single event.
- Do not track ‘Page load’ or ‘PageView’ as a distinct event — Many SaaS tools define ‘Bounce rate’ as ‘No event being triggered once a page loads’. Adding a ‘Page load’ event will result in an almost zero reported bounce rate. Also, most tools charge per event, so tracking page views is expensive and redundant (even if you build your own tools internally). However, tracking only the first page viewed on a web platform as an event called ‘Session Start’ may help analysis.
2- Event Properties: In addition to these interactions, there are additional data points such as the page a user interacts on, the item they interact with, or the browser/device they are using — which give further context to the interactions. Eg- When a user triggers an ‘Added to Cart’ event, there could be a long list of relevant Event Properties you may want to track such as ‘Product Name’, ‘Price’, ‘Product’, or ‘User Logged in Status’. With an event like ‘Performed Search’, you may want to track properties including ‘Search Keyword’ or ‘No. of Results’ and so on.
A few insights on defining Event Properties—
- As you can intuitively see, Event Properties are a usually a key-value pair. Make sure that all the properties you want to track are logically inferrable from a user’s interactions or the current state of the product.
- Properties help you ‘slice and dice’ your data. For eg- If you wake up one morning to see your orders have dropped by 20%, you can use properties to segment the conversion data across ‘Geography’, ‘Category’, ‘Price’, ‘Browser’, ‘OS’, ‘Device Type’ etc to understand why this happened.
- Try to think holistically about what questions you’d like to ask your data — and use this to frame more exhaustive Event Properties. In the example above of ‘Viewed Product Page’ being accessed from 3 flows (From the Home page, Search or the Category page), you could use a property called ‘Source of View’ to segment all such page views by the channel which led to the page view.
- A list of Event Properties should be individually listed on the spec sheet for each specific event you track. However, there may be some Properties you may want to track for all your events. Eg- ‘User Type’, ‘Browser’, ‘City’ etc — which most tools allow you to configure as “Super-Properties” — attached to all events triggered.
- I highly recommend adding UTM based super-properties. Every link leading to the product (on paid ads, and on organically shareable links) should have UTM parameters defined such that any unique link is uniquely identifiable by the set of UTM parameters. Once you record UTM details as a property on all events (and store this for a session), you should be able to link every single interaction or conversion on the product within that session to the specific URL (linked to a specific ad or ad campaign) which led a user to the product. This is immensely valuable for the attribution of marketing spends on the web, or for measuring organic virality. For apps, a tool like Appsflyer can achieve a similar outcome. Facebook and Google Ads do allow you to send key events to their ad-products as well — but it is much easier for PMs to access data on their own Analytics tools than on the Ad platform’s tools.
3- User Properties: While Event Properties capture context around the event, User properties capture the context around the user performing the event. For eg- ‘No of orders placed’, ‘Total Revenue generated’, ‘Discounts utilized’, ‘No. of app Opens’, ‘User Type’ are all valid User Properties.
A few insights on defining User Properties —
- User Properties are usually an aggregation or an incremented count of certain events. Although, some can be user-level properties such as a user type (Paid or Freemium). Try to think of data points that you would like to access on a user level. Eg- How would you segment the conversion funnel for ‘Users who use the app at least 5 times a week’ vs ‘Users who do not’. User Properties can also be an array of values (‘Devices used’, ‘Cities searched for’)
- User Properties depend on a user accessing your product in a logged-in state or with the same device across sessions. How this usually works is that every ‘session start’ checks if a user is logged in or not. If they are, all events will update the user properties against the logged-in user ID. If the user is not logged-in, a dynamic and unique ID can be generated and stored on the browser’s cookies for each session on both app and web platforms. When the user subsequently logs in, all such generated unique IDs and the associated user properties attached to those IDs should then be merged with the logged-in user’s user ID. Most tools support this if you configure this logic correctly with the right triggers defined for each of these operations. The caveat here is that User Analytics will track unreliable data if your product is usually used by more than one person on the same device without separate logins, or by the same person across devices with no login. Most tools offer detailed documentation on User ID handling. Read this part super-carefully!
I’ve seen bad implementions that dilute the benefits of collecting user data—
A project had all its events named as ‘click’ — effectively telling users how many ‘clicks’ happened on a page with no way of finding out what was clicked. In other projects, key events (‘Perform search’) were not being tracked but junk events (‘page-load’) were. In yet other project, user ID handling was so bad that it was impossible to leverage user behavior data across sessions. If you are working on a product where you’d like to change the existing tracking implementation — get it done as a P0 priority task to avoid further data loss. Incorrect or Incomplete data is worse than no data since it would give you the confidence to make bad decisions.
The Tools of the trade
Once you have your ‘Tracking Implementation Spec’ ready. The next step is implementation. There are two approaches here;
1- Using a SaaS Analytics tool-
- I highly recommend Mixpanel. Also, Clevertap is a less expensive, more feature-laden — but much less delightful ‘UX and performance’ alternative. They both have well-defined, exhaustive and generally fair privacy policies.
- Honorable mention for the out-of-the-box free-to-use Google Analytics (where Events are ‘Actions’, and Properties are ‘Dimensions’). Google Analytics also has the added advantage of knowing your users’ behaviors across other products on the internet — enabling it to provide anonymized affinity categories which may provide useful inputs to marketing. Eg- xx% of converted users have a high affinity towards specific topics such as ‘TV series’, ‘Sports’, or ‘Hindi Movies’. If your team uses Google ads, you’d probably be sending key events to their service so that your team can track ad performance against conversion events. (It is best practice to explicitly include this fact in your company/product’s terms and conditions)
- Most tools have detailed implementation notes. I would recommend running through the notes yourself and helping developers to the correct segments detailing the features you want to use.
2- Building an Analytics Product in-house; Large companies usually build similar functionalities as SaaS tools in-house. However, unless you work for a handful of really big companies — a SaaS tool is usually a cheaper and better-performing option. Also, SaaS tools are incentivized to constantly improve. (Do not underestimate the cost of a team to build, maintain and improve a reliable, fast and accessible custom analytics platform)
Before deploying any tracking solution, please fully test the implementation by using your product in-depth on a test project, and verifying that all events and properties are being recorded as you intended them to be.
Aligning your Tracking to your Metrics
As a PM, your metrics would primarily depend on the outcomes assigned to the scope of your work. While there are many frameworks to define the right metrics to track, I would generally advocate a large breadth of key metrics and counter-metrics.
You cannot measure what you do not track. So, once the metrics are in place, you should revisit the spec sheet and see if you are tracking all the events and properties that you would need to actually measure your metrics and sub-metrics. Avoid measuring data you may not need to track, or should not track — such as personally identifiable data.
A User Analytics implementation offers a rich data set of events and properties that can be used to track metrics, glean insights and discover user behavior trends. Once the right data starts flowing in the right ways — the value of User Analytics starts assuming exponential potential and soon becomes your best friend as a PM.
