Talking to Users During Product Development: The Challenges and Lessons Learned
Let’s explore the reasons teams engage customers and the challenges they are likely to face in doing so.
In the past, I wrote an article on how to start engaging customers without going into reasons why that may be helpful. Engaging end users in the product development process has become a textbook truth, hailed as the surefire way to build solutions that solve real problems.
But as we rush to incorporate user feedback, it’s worth remembering that a number of successful products, such as Sony Walkman, Post-it notes, and Viagra to name a few, were developed, released, and reached initial success without focused end-user feedback.
Furthermore, even when we involve users, we often do it in a way that backfires, leading to confusion, scope creep, and missed opportunities. In this article, we’ll explore the challenges that end-user involvement attempts to solve, the potential challenges it can introduce, and how to strike the right balance.
The Controversy With User Engagement: TL;DR
Whether you should engage customers when building a product or avoid listening to them at all costs is a flawed question. If the product you build does not flunk hopelessly right away and you end up with some customers OR some financial runway keeping you from drowning, you will end up engaging customers in one way or another to survive.
The question then will focus on the specific feedback you would be looking for, why that feedback specifically, and what you would intend to do with it.
In other words, the question isn’t whether to engage users but when to start engagement and what jobs to be done would you employ their engagement for.
Also, in this article, I do not speak about specific techniques to increase value from customer engagement. That I intend to cover in a separate publication soon.
Let’s Look at the Definitions
Before we go further, let’s look at the definitions. I like Pendo.io’s definition of user feedback.
User feedback is qualitative and quantitative data from customers on their likes, dislikes, impressions, and requests about a product.
Looking through the definitions of customer engagement online, here is the one that I found most meaningful:
Customer engagement is the ongoing cultivation of a relationship between the company and consumer that goes far beyond the transaction
In this article, I’ll use “user/customer engagement” and “user feedback” to refer to the involvement of someone with an interest in the product in its development.
If the developer of the product is the intended user, they may be the only one involved. But if the product is for someone else — individuals or businesses representing potential, existing, and ex-customers are involved.
Since we are dwelling on definitions, it’s worthwhile to identify the key types of user engagement. To me, all of the activities can be classified into one of two types: either they help us discover and validate the problem or they help us discover and validate the solution.
I also mention jobs-to-be-done. I like Clayton Christensen’s definition that I repeat in a somewhat freeform fashion:
According to Christensen, customers “hire” products or services to help them accomplish a particular “job” or task. This job can be functional, social, or emotional in nature. The idea is that customers don’t buy products or services because of their features or attributes, but because they want to achieve a specific outcome or complete a specific task.
Reasons for Customer Engagement
In a perfect world where no one counts profit and loss, we could just build a product, happily launch it and then refine or rebuild depending on the situation. But in reality, if we do not get positive results within an acceptable timeframe, either the company is out of the picture, or we [as a product team] are out because we have exhausted the trust credit.
Customer engagement before and during development together with the feedback it brings becomes that leading indicator of hope we are on the right track or a need to shift gears. From this point of view, proper customer engagement helps to reduce development waste such as over-engineering, incorrect capability allocation in the product, and incomplete or broken workflows.
Another reason to engage customers is the belief that teams can rely on customers’ creative juices to help co-develop more meaningful and innovative products.
Finally, working with users can provide peace of mind for those of us who worry about evidence-based decision-making, especially when things go wrong.
The last obvious reason is that nowadays with everyone talking about the need for engaging users, there appears to be a stigma tied to not being a part of “the best practices” fraternity where success will come if you remember to identify, state, and test your assumptions.
Product Development Challenges Related to User Engagement
Despite the many benefits, customer engagement can also make product development more difficult. Over-reliance on customer feedback can lead to marginal, insubstantial changes, while controversial feedback or the infamous 50/50 split in responses can result in analysis paralysis.
Fear of making any change without customer engagement can lead to a big design upfront that ends up partially obsolete over time, and misinterpreting feedback can lead to adding waste to the product.
Finally, engaging customers does not mean building truly innovative products as it’s not the customer’s job to envision what is possible and make it happen.
There is a good quote on the topic from Edward Deming in the book “The essential Deming: Lesson from the father of Total Quality Management”.
No customer asked for electric lights. We have gas mantles, a good light. What need would there be for electric lights? No customer asked for photography. No customer asked for the telegraph, or for a telephone. No customer asked for an automobile. No customer asked for pneu- matic tires. No customer asked for facsimile. All those come from the producer, not from the customer. And the producer has to watch out for the customers.
My Personal Experience With Engaging Users in Product Development
I have been actively working with users to help shape and improve the products I have worked on since 2015. Back then I was helping to build a product for Montessori record keeping and we needed to test our concept with actual teachers.
While we built a lightweight product that tests showed was easy to use and I even launched a pilot with a Montessori school in Utah, the product adoption never grew beyond a couple of schools in Kyiv, Ukraine.
As I moved into a different space (public safety software) in 2016 where I still am today, I have come to rely on engaging users for a number of reasons:
- Building domain knowledge. The hundreds of hours I spent interviewing dispatchers helped me understand the problem space relatively in-depth. Furthermore, they helped me find so-called champions — users with high social capital who can help promote valuable changes and shift the mindsets of others
- Reducing naivete. Numerous people that I spoke to provided me with perspectives that could converge but more often deviated. It helped me to understand that no one size solution fits all, and hence any solution has trade-offs and limitations
- Providing data points. The people I engaged were my data points and helped identify and document reasons for making certain decisions
Nevertheless, despite all of the value I got from these sessions and despite adding product capabilities that were marginally beneficial to users and business, there is one thing I have failed to achieve so far: groundbreaking changes that move the needle 10X.
Where Am I Going With This?
If you scout the web for success and failure stories in product development, you will likely see conflicting information. On one hand, there is an anecdote of Homer's car design that is nicely described in this blog article.
This is an example of the kind of monstrosity that creativity combined with a carte blanche approach given to direct end users can yield. Not only it’s never going to be used, but it also tanked the entire company that built it.
Critics of the approach to actively engage users in product development often retell stories of products that were built without much user engagement and still had success (Sony Walkman, many Apple products, Velcro, etc.).
On the other hand, there are many examples when engaging users throughout the development process helped to create great products and companies such as Slack, Calendly, Instagram, Airbnb, and many others.
Who Is Right and Who Is Wrong?
Before we can determine who is right and who is wrong, we must first identify the root of the disagreement. Is it a matter of not listening to users or asking the wrong questions?
I believe we employ user engagement and feedback to help with the following jobs:
- Problem expertise [how well do we understand the true problem]
- Solution expertise [how well do we understand the best solution]
- Confidence [how confident are we in our success based on the path we take]
- Unspoken truths [how well do we understand the overall context of the user, problem, and solution]
What Are the Jobs-to-Be-done for Customer Engagement?
Let us now consider the various scenarios in which product development occurs:
- A single person or a team with a specific problem they want to solve for themselves
- A single person or a team with a specific problem wants to solve a larger problem for others
- A product team that is part of an organization solving a problem for someone else in that organization
- A product team solving a problem for a group of people outside of that organization in a different space
- A product team solving a problem for an organization that is solving a larger problem for a part of their organization or a different organization/group of people
In scenario 1, a single person or a team identifies a specific problem they wish to solve for themselves. This genesis has resulted in numerous successful products. Individuals who are familiar with the problem and possess a clear vision of their desired solution are likely to possess confidence in their ability to achieve success.
In scenario 2, someone with an understanding of their problem attempts to solve the same or adjacent problems for other people. While the person knows their problem, the solution they envision may not be appropriate for others, either because it’s not desirable or viable.
In scenario 3, we have an in-house team implementing internal products for other departments that they may have little understanding of both in terms of the product and solution.
Scenario 4 describes a product company building products it intersects with only partially and hence has a low understanding of the space, expectations, and customer context.
Scenario 5 illustrates a case where a product is built by a 3rd party organization and is then resold to companies in a very different space. The knowledge of the teams building products, in this case, is likely very low, and hence customer engagement is critical.
As we can observe, with each successive scenario, the complexity of the product development process increases while an understanding of the jobs listed above plummets. Neglecting to engage users in scenarios 3, 4, and 5 can cause significant issues in terms of scope, budget, and timeline, or it can lead to outright failure.
While customers and users can provide valuable input, it’s the responsibility of the product teams to solve the listed jobs through proper customer engagement and interpretation of results. And frankly speaking, this is the area in which many, including myself, have failed.
Customers Help To Enhance Knowledge of Problem Spaces
I believe that regardless of your experience with customer engagement — you will come to rely on it. That’s why it’s best to build your awareness of the toolset required to work with customers.
In my opinion, the main troubles that teams find themselves in when they solicit customers’ input stem from engaging customers the wrong way and for the wrong reasons.
Customers can help you improve your knowledge of the problem space and existing solution space. They can also help you build confidence in your plan and product.
Customers can also help you understand those hidden truths about the larger context of other tasks they do outside of your product. You can learn about how swiftly they switch between tasks, how much tolerance they have for problems they experience, and so on.
While customers can provide input on various topics, the quality of decisions you make based on that input, the trade-offs you make without announcing them, and the innovation of the solution — it’s all on you and your team.
Those decisions are what’s going to make your product either fail, break even, succeed mildly — or get featured on the TechCrunch front page.
Thank you for reading! please consider clapping, leave a comment, or follow my Medium blog.
Special thanks to Tremis Skeete, Executive Editor at Product Coalition for the valuable input which contributed to the editing of this article.
