Keep your eyes open — Qualitative analysis at its best!
I travel for a total of 4 hours a day for work. Mind-boggling isn't it? It is indeed frazzling but over a year or so I have realized that if you keep your eyes and ears open, it can prove to be highly rewarding and insightful.
As a Product Manager or a startup founder, the product-usage data and its analysis are of paramount importance.
Data analysis at its core can be either quantitative or qualitative.
Quantitative analysis is what you do it on an analytics dashboard, doing the math with the events that were fired during product use. Through this, you try and find patterns, segment the users, draw inferences and accordingly move towards improving those numbers.
Qualitative analysis, on the other hand, is talking to your users either face to face or over a call and trying to understand the challenges they are facing and the expectation-reality gap i.e if it's generating the perceived value. It also includes observing the users while they use the app to shed some light on any UI/UX issues and user behaviour.
It is pretty easy to accumulate data for quantitative analysis. It is relatively easier to fire events at places we plan to than to actually get the users to talk to or meet as required in qualitative analysis.
Convincing users for qualitative analysis makes it tough but the insights you get is pure gold standard once you are able to accomplish it.
Assume that you are travelling by bus and sitting beside a person who is using the app you handle/own. So the steps are to
- Observe (silently)
- Talk
1. Observe (silently)
Hey, I don’t stare at my co-passenger’s mobile screen but do cast a brief look every now and then.
It may seem a bit unethical and sly but telling the user upfront that you will be observing him while he uses the app may not yield honest results as humans by nature are 'response-biased’. It is a type of cognitive bias which makes human respond the way they think you want the response to be. In our case, it will make the user (co-passenger) self-conscious and may be uncomfortable as well which will in turn miserably fail the exercise.
What can you learn
Ease of use is the core aspect of a product’s UX. Ease of navigation and content discovery for e-commerce, news and social media apps; ease of booking/order flow for the cab and movie-tickets booking, food-ordering and also e-commerce apps; ease of payment flow for all products involving payments (you obviously should not look at card details and the pin!).
With those razor-sharp eyes, your aim is to find —
- Frequent drop-off points like not able to search some content, or unclear menu images, complex 'pin location' feature, complex seat-selection while booking, difficulty in adding address while placing an order etc.
- Restricted use and constant hindrances while using the app due to a technical glitch that can lead to frustration, which, one’s piled up causes churn. For example, excessively high buffer time to load the first chunk of data for your video streaming app (you should probably load the first chunk at a lower bitrate for a faster startup time). Other issues you may spot are payment gateway errors, page load issues for news and e-commerce apps and even crashes.
I would strictly discourage you to observe the user secretly if it is a very privacy-centric app like a social media app.
In such cases, you may directly move to the next step of Quality analysis - talking to the user.
2. Talking to the user directly
The first thing here is to formally introduce yourself and clearly state the purpose. It should sound empathizing, not like a sales pitch that may put off the user. To ensure a freewheeling chat that could help you accumulate some critical information you should sound trustworthy and genuine. You may probably thank the user for using the app, talk something about how the team is trying to work on a problem or mention a few exciting features you may come up with.
Once the user’s body language looks accepting and optimistic, gradually start asking questions from any of these broad categories —
- The content - For news, social media or entertainment ask about the accessibility, searchability, quality and the variety of content.
You may find out that a Mumbai user is not able to view local news outside Mumbai because of automatic location detection or the fuzzy search of non-English keywords doesn’t work as expected eg. while trying to search for a Hindi song 'kyun ke’, user actually types 'q ke' or 'q k' which results in incorrect suggestions. You may also get info about the unavailability of some useful content. - The booking/order flow - for ticket/cab booking and food ordering apps you may ask about difficulties in completing the booking/order.
You can unearth some technical issues, complexity issues like difficulty in customizing food order, payment issues like failed payments on slow networks, unavailability of certain payment options, difficulty in accessing or sharing upcoming rides for cab booking apps, difficulty in choosing the movie language while booking movie tickets, confusing flow to avail offers using promo code etc. - Onboarding has a pivotal role in the product. A significant amount of churn happens at this stage i.e even before your app generates any real value for the user and thus, it has to be smooth and quick.
Try and figure out whether the user you are talking to faced any difficulties while onboarding. You may be surprised to perceive how big a task it is for users.
Issues while social login (Google or Facebook), ambiguities in the sign-up and login flow, confusing messages, tedious interest-selection flow, user device migration and multi-user issues etc. - Notifications are crucial for increasing engagement. News, content, social media or entertainment apps is normally driven by notifications. You may ask the user about how frequently they receive the notification and whether or not they are comfortable with that frequency. Users shouldn’t feel spammed with a barrage of notifications. You should also enquire about how they feel the quality and language of notifications - informative and interesting or purely click-baits.
On the technical front, you should enquire about any delay in receiving the notifications and whether each of them correctly deep links to the content.
Naah, you are not going to bombard the user with all the above questions in a single meeting. Ask pertinent questions. Identify a category or two which you feel is the product’s weak point or users might find difficult to use.
Be ready with a 5-minute, 10-minute, 15-minute script. Choose it depending on how interested and involved the user is in the conversation.
Such short observations followed by conversations with your users at different stages of the product lifecycle helps to gain quality information about how the users are actually using the app which could be in stark contrast to what you imagined.
It helps you make decisions on small tweaks to the UI/UX or to introduce new features. In some cases, it can even lead to pivoting the business model and identifying a set of the completely new target audience. But for such important decisions, you need to observe and interview a reasonably large number of people.
At the crux, it’s all about keeping your eyes and ears open while you are travelling. You are probably surrounded by your users!
The icons used in this article are courtesy Freepik, Alfredo-Hernandez, Srip, Photo3idea-studio and Eucalyp. Thanks for reading.
