SMS verification for SaaS sign ups
Why we used it and why we stopped.
A Deep Dive on our user on-boarding.
When we started OHNO, a big focus was getting feedback to ensure we could iterate quickly. In the early days, I sat down with a lot of potential users, asked questions, showed them prototypes, but nothing is the same as real world users.
The reason I wanted to write this article, is because too often, articles about product management focus on agile tradecraft which is so idealistic, it’s essentially impossible to implement, because most teams are just drowning in bullshit, and trying to slay dragons. Every product management article goes something like this;
- 20 things to launch better products using user stories
- Why roadmaps are a waste of time (Yes, I wrote that one)
- And why product strategy is critical to success.
But one thing I’m personally craving, is real behind the scenes case studies about very specific, tactical problems. For instance, I want to know;
- How does Spotify work out how to build their weekly discovery playlist, and know it’s good. What numbers do they watch? What mistakes did they make?
- How much do signups increase by adding the Google Sign Up buttons vs Facebook for a SaaS B2B product.
- What techniques do Fullstory use to migrate users from free to paid accounts?
So I figured, why not just tell my story about something very specific. So I’m going to explain why we decided to use SMS as an input for sign ups, why it didn’t work, and what we learned from the whole experience.
Some basic background
To explain the gist of OHNO, mostly to provide some context — it’s a micro survey product that uncovers the problems preventing teams from reaching your mission, objectives, and upholding the values that are important, and gives you a framework (The five whys) to solve those problems for good.
It’s web based, SaaS, with a free plan, and 3 paid plans. We launched on new years eve (the worst day for launches, if you’re curious), and we have about 150 users so far. We’re bootstrapped, and really just getting started.
To reduce as much friction as possible, we broke the sign up process into stages. This reduced the perceived number of fields users see, making them think it’ll be faster than it really is. The further they go through, the deeper their commitment to finishing the process, this is based on a psychological bias known as the IKEA effect. We place disproportionate amounts of value on things we invest time in creating. It’s a common practice today, and not worth going into any more than that.
By doing it this way, we get away with more inputs, while making the process appear simple. This was how we laid out the original panels.
Stage 4 was an on-boarding input for us. We were hoping to gain the first problem in OHNO, and use that as the sample data. OHNO captures problems happening in your organisation, so having the first problem is very useful.
So what happened?
There were a range of reasons we wanted to use SMS verification, and I’ll break them down here. I cover what we thought at the time, and what we learned along the way.
Reason 1. Robots
When we launched, I wanted to be able to verify accounts. Truthfully I wasn’t sure why, I just knew this was always something we did. My thinking was robots might sign up, cause drama, and we wanted to avoid bad accounts.
It turns out, robots didn’t sign up. So that didn’t matter. If you’re a robot (and reading this), don’t even think about it.
Reason 2. Feedback
Given our focus on feedback, I really wanted to be able to pick up the phone and just talk to users if they were having problems. By capturing the mobile, I had access to that channel.
In the end I probably only had time to call about 15% of users anyway. Most of our first pilot users I knew really well, so we would chat via Linkedin or email, and that seemed to be fine. Asking for mobile turned out to be a waste.
One problem using mobile numbers for feedback was something I totally underestimated. I’m a caller. I tend to call people — to say whats up. I’ll call people I haven’t spoken to in a year, just to say hi and see how they are going. But I’m starting to realise this isn’t really what normal people do. I find myself coming into contact with more and more people who text (via whatever) as their primary reason to say whats up.
This meant that when I would call leads, most just wouldn’t answer. Because they were busy, in the middle of something, they hate me, whatever reason really. When you think about it, most people are working, and my call is really just an interruption in their day. I’m not coming to them in the chronological context of them being ready to talk about OHNO. I really underestimated this.
We have crazy high open rates on on-boarding and drip campaign emails. 60–70% in most cases. So people will listen to what we want to say, but just not via phone.
So yeah, it didn’t help there.
Reason 3. A sneaky proxy for country.
At my last company, I worked a lot on sales and activation stuff. At the time, I was the head of product for a job management tool, and our sign up form was a world class disaster. It had LOTS of visible fields, and our conversion rate from view on sign up form, to clicking sign up was low.
I crunched the numbers using Mixpanel funnels, and could more or less tell with each additional field we had displayed, we were losing about 7.5% in users with each field.
We had 8 fields at the time (I think)
- First name
- last name
- Company
- Industry
- Timezone
- password
- password 2 (repeat)
The way I saw it, we could reduce that way down by introducing a Google Sign up button, which would get us at a guess, between 30% and 40% higher conversion rate.
So we did it, and that’s exactly what we got. About 30–40% higher returns by reducing a lot of that visible friction.
So when it came to OHNO, I wanted to capture as much data I could about users, to use in segmentation, but reduce the friction as best I could.
By using mobile number as a disguise for verification, I got the communication channel for feedback I was after, but I also got verification of country data.
By asking for mobile, I knew where they lived.
It turns out we didn’t really need it for country data. We captured that via IP anyway. So it didn’t count for much in the end. Yes. Mobile is better, because you avoid the IP routing problems, but in the end it didn’t count for much.
What else wen’t wrong?
Bogus numbers
We ran into problems with users doing mobile verification a lot. People would put in bogus mobile numbers to test out the software, only to realise our UX to ‘un-fuck’ that situation was pretty weak. So users would get stuck, then just bail on the sign up process.
Even if they wanted to get back into it, we lost those users more or less for good because we didn’t clearly state the reason the SMS was there.
Lesson here was if you are going to use SMS verification, you HAVE to explain why.
Secondly, you need to have a way to go back, enter a different mobile number, and go forward again. In our early UX flows we avoided this. We underestimated how big of a problem it was going to be.
Hesitation
Another problem that came up a bit was hesitation. Some users (mostly programmers, actually) would simply refuse to put their mobile number in and join their team in OHNO. This made the managers jobs who wanted to adopt it harder to convince them that the tool would add value — it ended up making it harder to get good activations early on. After we heard this a few times, we pulled the plug on the feature and unwound it.
We listened, and responded accordingly, and the whole SMS verification experiment probably lasted 4 weeks.
We probably could have addressed the hesitation factor by explaining the point of the verification, but the evidence was mounting to drop the field.
Time
A common activity during my week was going into Mixpanel, drawing out the SMS code, and sending it to users either via SMS, email, chat or whatever, in the moment to get their trial back on track. This sucked up heaps of time for me, and was a constant source of anxiety. As soon as I saw a bogus SMS, i’d have to stop whatever I was doing, and quickly find it, send it to them, and try resolve it before they bailed on the process.
Where the fuck is my phone?
The problem with using SMS verification with a SaaS product is we would get users who simply had their phone in another room. People are super lazy. The idea they will get up from the sofa to get their phone to complete a registration is too much to ask of most people. We lost a lot of users here. More than you think.
This friction speaks to the delicate nature of on-boarding, and how very small bits of friction do add up in the long run.
Latin America
I have no idea how this happened or why, but early on our most active users we’re all in Latin America. Brazil and Mexico City, and a few from Colombia. I have never spoken to these users, but I could see what kind of events they would trigger in Mixpanel, and they still rank in our highest cohort of active users.
The problem was we couldn’t speak Spanish, at least not very well, and they wouldn’t respond to outreach emails to say hi. I thought having mobile numbers might help, but calling international is always a bit of an ordeal to do at scale, both the cost, and the timezone plays a part here.
It’s easier to book a call for 10 minutes, and manage it like that, but to make that work they have to engage via email. But for whatever reason, we just noticed it was much harder to contact our users in other places around the world.
So what changes did we make?
We removed SMS verification completely and we gained back heaps of time in our weeks, made the on-boarding process a lot less shit, and gained a lot more users with the same conversion funnel.
I think on reflection, if we were a mobile app, SMS verification makes a lot more sense. The kind of instant, smart SMS verification code process you see with apps like UBER work great, because the context IS mobile. But when you try carry that to a web based SaaS product, it doesn’t work very well.
All of these seems very obvious in hindsight. But at least we learned — and iterated accordingly.
This is how it looks today. 2 Panels. That’s it.
That finish button is where we capture the user details and consider them signed up. Then we take users through a 7 step on-boarding wizard, which is designed to personalise their experience a lot more, and give us user attributes we can use in the rest of the on-boarding process.
Then once these panels are finished, we leverage Intercoms new Tour feature, to walk them around and hold their hands.
It might all seem like a lot, but the new process is doing wonders moving our metrics in the right direction now.
About me
I started OHNO with my brother in law, Poe Wie Chen. Our mission at OHNO is to reboot the workplace and liberate teams from mediocrity. We think there are too many ceremonies and practices we all know in our hearts to be useless & wasteful, and lots of them don’t serve our customers or help us enjoy what we do. We value outcomes over traditions.
We want to liberate teams from those burdens by helping them get to the heart of what’s slowing them down using a really clever product we built called OHNO. If you liked this article or care about our mission, I’d love to hear from you. You can reach me on Twitter here.
And you can always try OHNO for free. Discover what’s holding you back from your goals at http://ohno.ai
