How giving analytics access to everyone at Consensus changed our AI startup

What if I told you that most seed-stage startups—even those at the forefront of AI—are way behind on analytics?

The reason is obvious: When you’re a small team, like we are at Consensus, you have a million things to prioritize, so it’s hard to find the time to invest in work that your users aren’t asking for. But there’s one big problem with this logic: User interviews, support tickets, and feature requests can only get you so far. To really understand how users are interacting with your product and your marketing messaging, you eventually need to look at the data.

To our credit, we knew this. But, unfortunately, the analytics platform that we initially used was so cumbersome we had to pay consultants to configure it. We were spending more time asking for data and wondering if it was correct than analyzing our metrics to make decisions.

We quickly figured we needed a solution that everyone on the team could use autonomously—from the founders to every single engineer—and we needed it quickly.

In order to be truly data-driven, we knew we couldn’t silo analytics into one role or group. Instead, we felt that it needed to be the glue that holds everything we do together. This led us to move to Mixpanel for self-serve product and marketing analytics so that everyone building a user-facing experience at Consensus could have access to data insights.

But our choice in analytics tooling has only been part of our success here. Our new company-wide analytics plan has also driven us toward the focus and depth necessary to get results.

There’s an understandable urge among data-driven startups to track everything. But with limited time and resources, you need to focus on what counts. For Consensus, a freemium search engine, that meant two things to prioritize early:

For example, we initially theorized that a good activation metric would be whether someone had performed a search. If you sign up for Consensus but never search for anything, you’re probably very likely to churn. But if you sign up and search, then we’re much more likely to retain you—at least, that’s what we thought.

When we used Mixpanel to look into user search behavior, we saw that our theory wasn’t quite true. Merely having performed a search wasn’t necessarily predictive of activation. A user had to have a good search experience. 

By looking at the data, we saw that a lot of people were inputting searches that indicated they really didn’t understand our product. If they asked questions like “What’s the weather in Toronto?” or “Best analytics platforms for startups” we knew they didn’t understand that it’s for finding scholarly research. In those cases, they’d likely be unsatisfied by the first impression and never come back.

However, when people asked good research questions, and got lots of highly relevant results, then they were much more likely to stick around. That was especially true if they further engaged with the results—all of which provided us with more data to analyze and use in our decision-making process.

Looking at search vs. retention also influences our marketing strategy. We’ve adjusted our marketing messaging and added some features and content to help guide users so they’re more likely to have a good search experience.

For instance, we now show users many example queries and questions to ask. As we grow, we’ll refine this strategy even more, leaning heavily on our analytics as we do.

At other times, we’ve used data to confirm when our instincts are correct. When we released the first version of Consensus, we assumed that a major draw would be the ability to easily share the results of searches.

We suspected (and hoped) that if people used Consensus to win an argument or provide supporting evidence for writing, they’d really see its value. But this was all hypothesis until we began using Mixpanel to zero in on our data. Once we did that, we saw that users who shared results were, in fact, 30% more likely to be retained for 30 days than users who didn’t share results.

While this finding was exciting, we took this as a cue to refine our product to encourage sharing even more and to make it more intuitive and rewarding.

Our approach to analytics is rooted in inclusivity. We want every team member to have data at their fingertips so they can make decisions rapidly.

We used Mixpanel to create custom dashboards that the whole team could access. This cuts down on the amount of time spent going back and forth and encourages everyone to consistently look for insights in the data and think of ways to use it. Our search engineer even set up his own dashboards and built a whole suite for himself, so he can now pull all the data he wants to analyze user queries. 

We also integrated analytics reviews into our weekly product update meetings. This wasn’t just to provide reports but to parse out the stories behind the numbers. How did a feature update impact user retention? What patterns were emerging from the data? And, most importantly, what should we do with those realizations?

With AI, the opportunities for using behavioral data are endless. But, again, a big key to success as a startup is focus.

As a search engine, our focus in the near term is to improve search relevancy and train our search ranking models by feeding them large amounts of behavioral data. Even as we grow and add more team members to work on expanded goals, we’ll still ensure everyone at Consensus can easily access analytics to move quickly.

And it’s easy to imagine scaling this through Mixpanel, which also allows us to export data and insights to more technologies we’ll need along the way. We’re keeping analytics at the center of what we do, and we’re very excited to see where it can take us.

To follow along with us, I welcome you to sign up for Consensus to start searching for free 👋