When your awesome feature turns to be a designed bug

“The range of feedback received matched the statistical results, where some customer segments significantly increased adoption, while others showed a significant decrease. We were surprised about this inconclusive result. But it was conclusive for us, some customers found our feature useful and others considered it as a bug.”

Our new awesome feature

New functionality was inserted to enhance an existing feature. The whole Product department was so excited, that we even added a special ‘thank you note’ to our special customer, for encouraging us to add this functionality. The customer truly appreciated this gesture. The feature was very specific; allowing Admins to get notified about specific actions when their users modify attributes on our SaaS platform.

We commence our story in the Beta stage of release.

This feature request was initiated from feedback that we received from a customer via the support department. We inserted it into our product backlog and after analyzing it, we interviewed the customer. From this stage we started to validate this requirement with other customers as well, trying to understand the impact of this new feature on our customers’ ecosystem. We realized that this functionality has a positive ROI, especially when we compared the development efforts against the impact of existing and potential customers. For us, it was a simple, and straightforward formula: Happy customers + Positive ROI = everybody is happy. A classic win-win situation.

However, as the Beta progressed, we understood that we might be missing something.

The support department reported that a certain number of customers included in the Beta complained about getting double notifications which they did not get previously. We reached out to all customers who reported the issue. Some customers told me that they are confused, others reported that it was annoying, but one customer actually said that it was highly useful!

The range of feedback received matched the user adoption statistical results, where some customer segments significantly increased adoption, while others showed a significant decrease. We were surprised about this inconclusive result. But it was conclusive for us, some customers found our feature useful and others considered it as a bug.

Is it possible that we had created a design bug?

Considering that:

• We are a top leading company in our domain, we are customer centric, always thinking of our customers from all angles across all business units.
• given the fact that our scale is high, consisting of thousands of customers and hundreds of thousands of users, we are used to the high impact of even the smallest change
• Our product organization is well organized, following the most updated best practices as well as using state of the art frameworks that help us to orchestrate the entire product ecosystem.
• Our technologies are cutting edge and know how to fulfill (almost) all product requirements

And yet, all the facts above are irrelevant when answering our question. The answer will always be “Absolutely yes!”

Is it possible to avoid designing a bug?

When it comes to bug designing, it cannot always be avoided. But it is certainly possible to reduce the chances by asking yourself the following questions:

• Is this a new feature? Perhaps it is new functionality to an exciting feature, and you need to then consider the impact to your customers already using it
• Dive (again) into the stats. What is the current adoption rate that you already have in this feature? What is the expected adoption?
• Are all scenarios covered in your PRD? Have your well written scenarios been applied to all users? Segments? What about Existing vs. new? How are these users going to be aware of functionality when it comes to an existing feature? For example, a new checkbox. And if so, will this checkbox be marked as default check and introduce this functionality to all users? Perhaps marking this checkbox to default check in some users and in others leave it empty? What will be the impact of their workflow in each case, and how you are going to introduce it from a UX perspective?
• In which stage are you going to introduce this feature? Alpha? Beta?
• A / B testing? or any other tests that you find relevant?
• Is your feature BIG or Small? I can say for myself, as someone who has experienced for many years working in Agile Frameworks, that it is much easier to disassemble the big feature into a smaller features and by doing so, receiving that feedback from the customer as well as minimizing the potential damage
• You have your testing results, and yet… have you considered protecting this feature with a flag? So it will be introduced only to the group that you have chosen, have the group’s feedback and then decide to launch it to all customers?

And the list goes on…

Turning lemons into lemonade

Once we understood the case, the solution was simple. We planned to move from Beta to General Availability phase. ‘The fix’ implemented was the adding of a checkbox that will allow our customers to decide whether to use it or not. The default was ‘unchecked’, so this new functionality was not introduced by default to existing customers. However, to the segment that found it useful based on our research and their feedback, we left it ‘checked’.

Our walk-throughs were updated, and support was trained to get familiar with the new functionality, to assist customers who reached out.

One month after the General Availability, our stats showed high adoption rates and happy customer feedback! We realized we had fixed the problem, we were glad we had learned a thing or two, and especially how to turn this lemon into lemonade.