Everyone talks about getting insights from digital products and processes, yet I’ve seen it done wrong so many times that the subject demands revisiting, especially in light of the emerging field of DataOps.

Irzana Golding

·

Follow

Published in

Product Coalition

·

7 min read

·

Feb 13, 2023

--

2

Listen

Share

Data is increasingly complex and complexity can only be managed via stable and reliable foundations. If metrics are not stable, data-related problems will quickly compound and stymy efforts to generate insights.

Here’s the bottom line:

Insights must be actionable, else they are not insights.

What is a Metric?

A metric is a anything you measure. Examples:

1. #Site visitors (digital sign-ups)
2. #Sales calls (sales)
3. #Leads (channel management)
4. #Heart-rate (wearables)
5. #Rainfall (agritech)

Metrics should be well understood, good quality and accurate, ideally managed via a DataOps process wrapper to ensure stability and availability.

This can’t be stressed enough.

What is an Insight?

An insight is any pattern or evidence in the metrics that suggests a possible action to improve business goals. A typical pattern is:

Based upon [metrics pattern], if we take [this action], the expected benefit will be [suggested outcome].

I’ll talk about patterns in a minute, but first some examples of the actions ands outcomes:

1. #Site visitors (digital sign-ups) — if we produce more of this content (action), we will get more visitors (outcome)
2. #Sales calls (sales) — if we call more customers in this cohort using a this playbook (action), we will get more upsell opportunities (outcome)
3. #Leads (channel management) — if we target these channels with this offer (action), we will generate more leads (outcome)
4. #Heart-rate (wearables) — if we notify high-intensity workout users with this offer (action), they will buy our upgrade (outcome)
5. #Rainfall (agritech) — if we plant this crop 2 weeks earlier in these regions (action), we should get higher yields under drought conditions (outcome)

Let’s mention a common anti-pattern.

Smashing metrics into Powerpoint and talking about them are not insights, even though the term gets abused in this parochial fashion.

Noticing a trend and saying, “Such and such is declining” — is not an insight because it has no action. If you have legions of folks making such slides and calling them insights — stop it.

An insight typically has properties that indicate that it is indeed an insight:

1. ❌ It is not a metric
2. ❌ It is not a first- or second-order statistic of a metric: mean, median, standard error, rate-of-change, etc.
3. ✅ It is a pattern
4. ✅ It reveals an action that could be taken towards some business goal

What are these patterns?

1. #Site visitors (digital sign-ups) — people who interact with our loan calculator are more likely to sign-up.
2. #Sales calls (sales) — customers in this cohort who received core messages from this playbook are more responsive to upsell offers
3. #Leads (channel management) — customers in our high-tier partners channel are more responsive to bundled-product offers
4. #Heart-rate (wearables) —heart-rate patterns with this cluster of attributes indicates athletic users and they tend to like our athlete’s package
5. #Rainfall (agritech) — crop rotations within these rain-shadow areas grew better when planted earlier

The primary activity in generating insights is pattern finding, often in the form of prediction. There are common methods for doing this, both from statistics and machine learning. We will defer discussion of these to a later article and focus more on the operationalization of insights.

How To Operationalize Insights: 3 key pieces

By operationalize, we mean make it a repeatable and reliable process so that you end up with a pipeline of insights to drive experiments and product improvements.

1. Clear Business Context is Key i.e. Know Where to Look

A common mistake is not knowing what problem you’re trying to solve. Analysts and data scientists can end up working on problems without a solid understanding of the business context. Or, different analysts work on the same problem because they’re unaware of other areas where they might look for insights.

Leave nothing to chance! Make sure the entire landscape of metrics is laid out and understood.

Often, there is no clear map of KPIs — i.e. the metrics you really care about and how they interact. Discipline here is essential and is a part of good data hygiene that forms the bedrock of DataOps.

A useful approach to providing clear business context is to build and maintain a KPI Tree, which is an easy-to-understand map of all the key KPIs rolled up from the lowest level to the top-line metrics (e.g. revenue).

They look something like this snippet:

This blog post by Petra Willle and Shaun Russell explains KPI trees from a product metrics perspective and is a good read.

One way to think about the tree in relation to insights generation is to view the tree as a heat map — areas where you have insights are hot, else cold.

You’ll notice cold areas where no one is really looking, which are often ripe for insights generation or show blind spots in your coverage.

Also, it often pays to focus on a metric lower down that feeds up into a key metric you’re trying to improve. These “proxy” metrics are often easier to generate first-approximation insights in order to move the needle quicker.

2. Metrics Management is Critical

As part of good DataOps practice, metrics must be backed with a data dictionary to manage definitions and key operational parameters, such as where to find the canonical model, versioning, ownership, related Agile stories etc.

Metrics management is a whole topic in itself, but the old adage applies: garbage-in, garbage-out.

Insights generation and its output — actions — is always a downstream process from metrics. Any instabilities in metrics will thwart generating insights.

Consider using tools like DBT Cloud so that metrics models are modular (i.e. re-usable) and backed by tests. Tests should be run whenever the tables update. Better still, tests can be incorporated into critical upstream processes to detect breakages in downstream dependencies.

Ideally, the maintenance of the metrics dictionary is via Agile. It is well known that you should strive to start with best practices from day one because they become increasingly hard to graft on later — see Fractal Software’s guide to vertical SaaS for more details.

The importance of good metrics management cannot be overstated. It is the heart of insights generation. The so-called Red Queen Effect has been widely discussed as applicable to modern orgs — i.e. that you need to run twice as fast just to not stand still. Insights generation is tightly linked to metrics, which in turn are highly sensitive to data management. The last thing you need is constant data-thrashing that will make it even harder to run twice as fast.

As a rule, metrics definitions should be stable. It is a mistake to change definitions, such as what qualifies as a sales lead, because insights and downstream processes become miscalibrated.

If you need to change a definition, it’s best to produce a new metric and leave the old one in place.

3. Build an Army of Data Explorers

Let’s remind ourselves of the format:

Based upon [metrics pattern], if we take [this action], the expected benefit will be [suggested outcome].

If there’s no metrics pattern, then there’s no data, as in evidence, to suggest the action.

Clearly, the critical path is the ability to explore patterns.

An actionable insight is an actionable insight no matter how it came about. We don’t need to rely solely upon data scientists to generate insights. They have unique skills to do so, but not exclusively. Plus, they can quickly become a bottleneck.

A way to accelerate insights generation is to make it easier for more folks to access the data.

This is what Facebook did — gave everyone access to the data.

Many insights can be found using relatively simple slicing-and-dicing exploration of the kind available in Tableau and Power BI, etc. The key then is to build an effective self-serve environment so that as many folks as possible can go in search of insights “at the speed of thought”.

Think of insights as a pipeline with a constant inflow of insights generated by as wide an audience as possible. Data scientists can also use this to deep dive into “pre-qualified” insights leads.

The key to an effective self-serve environment is having a stable data environment, which is why metrics management comes first. After that, with self-serve, the sky’s the limit in generating insights.

Special thanks to Tremis Skeete, Executive Editor at Product Coalition for the valuable input which contributed to the editing of this article.