How Vodafone’s Custom Scoring System Clarified Outcomes

Vodafone is a leading telco that operates across Europe and Africa. They operate in 17 countries and provide services to 300 million customers.

With access to such a large number of people, Vodafone is on a mission to drive inclusive and sustainable digital society through great technology.

The following interview takes place with the Products and Services team that addresses consumer needs across three portfolios called TV, Home, and Flex—everything from broadband to Vodafone Television to the FinTech services.

Dscout’s Customer and Community Marketing Manager, Colleen Pate, sat down with the team to learn more about how they blend design, UX, and research — and how custom metrics changed the game.

Ashton Snook is the Head of Product Design and UX Research.
Georgie Thompson is the Lead UX Researcher.
Nick Lockey is the Lead UX Strategist.

Colleen: How do the research teams work at Vodafone?

Ashton: Within CMS, we have two essential research divisions. One is UX strategy, and that's very much Nick's bag. It's at the beginning of the journey answering the questions that you've got on screen there: Are we building the right thing?

The other half of that is UX research, which is, are we building the thing right? Those two core questions allow us to connect with the wider community, our customers, and build products that we see having a very meaningful impact on the overall Vodafone strategy we discussed earlier.

Tell me more about those dueling questions: are we building or designing the right thing and then, are we building or designing the thing right?

Nick: We had a lot of discussion about how to visualize this. It's two things at one level. It's a linear way of imagining where the products sit and where all the research studies sit. But then as you can see on this chart, that line loops back on itself and twists and turns, so it's not an A to B thing.

It's like A to B to F, Z, probably some Greek letters, the occasional number, some emojis as well. It just gets confusing. I think the best way to understand the scale of our work is that we're a product design team, so a lot of stuff happens before we do our work.

A lot of business casing, a lot of exploring audiences and needs, but when it drops into our realm, it's like, how do we turn that into a tangible product that actually gets the market? If you think of a left-to-right scale, the very left end is more my domain, which is, are we designing the right thing? What could this product be? What are all the different variations this thing could end up looking like? Where could we take it? What could we do? What are the different value propositions?

As we refine and iterate that, it kind of moves along the scale until we build that confidence over what is that itch we're trying to scratch for the audience, and how do we shape it?

Then as it gets to the right, which is, how does that product come together to get to market? That's where it goes to Georgie. So we know what this thing is for now. How do we optimize that to do the best it can to make the biggest impact for our customers?

We're a very customer-centric department, but what's interesting is that at the start of the journey on my side, we have to be very sympathetic to how that aligns to the business needs. How does that align with the commercial model? How does that align with the engineering teams and what they can build? We can't deliver value for our users if we can't get that to market.

That means that we work with our stakeholders and other people around the business to make sure that our vision for making something fantastic for the customers gets past all the stuff it has to do to make it viable as well.

Georgie: My realm is around evaluative research. We've had the great work that Nick and his team have done to scope out the questions that we might have and basically check that we're creating something that our customers want or need, and it has some value to it because again, you can have beautifully designed products, but if they're not adding any value for people, then it's irrelevant.

On the flip side, you can have a great idea, but if it's not usable or it's not understandable to our customers, then it's not going to work. You have to hit that sweet spot and that's where I come in. Sometimes pre-launch, sometimes post-launch, and sort of everything in between.

It's checking that, okay, we've got this great idea. Now what? How do we present this in our designs? How do we make sure our customers can fulfill whatever experience or whatever sort of objective we need them to do?

I think what's key here and why we've got this sort of loop design is that it's like Nick was saying, it's not linear.

Just because it comes to me towards the end, I might discover something in my research that brings out some new questions that we'd not thought of. New ideas come up from our users, and I think that's something we need to explore further. So that might be something I pick up or it might be something that I loop back into the strategy team and they then explore that further.

It’s the same with Nick. He's looking at the discovery side, but there might be instances where he sees that we can fast-track this. Let's just test this now because we know it's a good idea for whatever reason or actually, we don't need to spend lots of time workshopping this, we just want to get some answers in terms of how people use this or work with this.

It loops around. It doesn't just stop or start with Nick and end with me. It crisscrosses and, as Ash will probably explain in a minute, product design sitting in the center there is integral as well. So Ash, do you want to mention anything in terms of your space in the middle there?

Ashton: Yes. You guys have both done a great job as always communicating that.

The graphic clearly articulates that design sits at the intersection, but because we see design very much as an end-to-end process. We have that design talent mixing in from the very beginning to the very end, and all phases in between—much like we have colleagues from markets and their product be involved throughout this process.

That's quite a critical thing that you can find if you work with agency models or if you work with companies with different structures, that you might outsource critical parts of. Are we going to go and build the right thing? That goes out for a long time when it comes back to designers.

There's a bit of a virtual wall between how they think about the work and the same thing at the end if you're validating designs. So it's been critical to develop a very effective culture for developing the right products for our customers to bring those disciplines together into what is more like a spaghetti instance, but the graphic does a nice job of simplifying for us.

Is there a way your team focuses and prioritizes discovery work, concept work, additional feature development, and ensuring customer value?

Ashton: I think the reality is all of these things are happening in conjunction because of the size of our team and the demand that we have in it.

Nick: Those stages of research are probably quite familiar to a lot of people here. Going back to that sort of non-linear thing, we're not a huge team and we're jumping between projects and different products and different stages of products all the time.

So it might be the project that we're doing today and the one we're doing next doesn't necessarily follow on from that one. We might be jumping on the next biggest priority for the products and services team. We're always jumping between the two, but at the same time, we're quite organized as well in terms of understanding the roadmaps and the questions that we pick off at any time.

We're often jumping between very early discovery to things that are near-launch or post-launch or something in between. We're sharing insights across the team continually. We have lots of meetings and ceremonies between the team every week or every month where we come together and we showcase what we do. We share what we do with the wider business as well.

It’s interesting because, partly by design and partly by necessity, we've never really followed the model where we'll have a researcher embedded in a product. While we were establishing the team, we had to be a bit more fleet-footed and help lots of different teams pick off the most important questions at the time.

Over time, that became a bit of a secret superpower for us because we're jumping between different products and different stages. A lot of the time, these teams are heads down in terms of just trying to answer their particular product questions and moving the dial on what they're doing and doing a really good job on that.

We've ended up being in a position where we can often take a bit more of a helicopter view on it to see what various teams are trying to do and how they're all solving problems. We can very often go, “There's a team over here that has come up with a solution that will work brilliantly for this product, or two teams are trying to answer a similar question and we can bundle that together into one study and get more value for them.”

Not having that more traditional model where a product team will have an embedded researcher lets us move around a lot more and overcome a lot of the bottlenecks as well. Sometimes, you'll have a lot of work to do on one team and then things will slow down while they're putting that into engineering or something.

It just means that we can move our research resources around and make sure they're always really delivering value for whichever team needs it the most. Also, and this is something Georgie might explain a bit as well, it helps overcome some of the attachment we have to the products and we can be a bit more objective.

Georgie: When you know anyone that's experienced in being embedded within a team, which I'm sure many on the call are, it has a lot of advantages. You get to know your product space, and the stakeholders that you work with. But that can also cause you to become potentially less aware of any problems that your team might be facing or any challenges.

You might start just taking things for granted a bit or not seeing the bigger picture. Having the ability to work across teams, you can see how different product pods are organizing themselves and how they're managing things.

It allows you to take the best bits from all and cross-pollinate that across all the teams, rather than getting stuck in your ways. That's obviously not always the case if you're embedded, but it can be that. We find that it's nice being able to work across.

On a personal level, it gives us that experience of working with lots of different product teams and different questions that we're trying to answer, because you can get stuck answering similar questions, particularly on my end, when we're doing evaluative work, answering similar questions over and over again.

This makes it interesting for us from a personal perspective as well. We get to work with different challenges with different personalities. I think it keeps it interesting. As well as the great reasons that Nick mentioned from a business perspective, I think personally, we found it an enjoyable way to work as well.

Why was creating Vodafone-custom metrics necessary for the business and how'd you start?

Georgie: We reached the stage as our research department matured, that there became an obvious desire really for clear and easy-to-understand insights that became more of a requirement.

We found ourselves with this challenge of selecting and developing UX metrics that allowed us to more easily assess our products. We wanted to create a common language with our stakeholders across the product cycle. This goes from discovery to validation.

We wanted that consistency to be present across the whole cycle. Not just, I'm asking certain questions at one stage and Nick is asking something completely different at the start. This allowed us to benchmark and have some consistency when we ask our customers for their scores at the end of the session.

We often find that when we get to this wrap-up part at the end of our sessions, this is typically where we get interesting insights. This is where customers can come alive. They almost feel like they're not in a testing scenario anymore, which we try to help them feel throughout, but it's not always possible when you're being recorded on the screen and you're having questions asked at you and you're using a prototype.

When that stops we get them to think, okay how did you feel? How was your overall experience using whatever it is or with this idea? That's when they get talking. We wanted a way of capturing that because while we always did a wrap-up previously, we didn't necessarily have a way to measure that experience.

That became more of a requirement to measure consistently. But don't get me wrong, we’ve had a few doubts as a team at first about using metrics, particularly for qualitative studies. We typically talk to small cohorts of users in depth, but it's smaller sample sizes.

There was a little bit of concern that doing a score for this type of work is the right thing to do, but we've been doing this for a while, at least a year. I think the metrics that we've established that we're going to talk through shortly are working well for us.

We're now advocates of it which, if you asked us 18 months ago, that might not have been the case. So I think you have to give these things a try, and hopefully what we've learned can help others potentially use them for core studies as well.

Ashton: As they said, we saw it as a bit of an opportunity and challenge. I think Nick's probably the best person to talk through this. What I said before, hello to Nick, is we were surprised at how positive an impact this has had on our workflow and the engagement we've had with our stakeholders to understand and have a bite-sized takeaway from the research.

You've got the in-depth research to be here through commentary and video slices, particularly on the dscout platform. These metrics have been so good at helping us and our designers progress and then engage with senior leadership across the business quite high up on big investment opportunities. Knowing where to guide and share products has been exciting.

Nick: Yeah, I think what's probably my favorite thing about these scores is that they actually came out of a design-thinking process. They weren't just born like this. We started off with the customer effort score, CES score, which is one that's fairly well known.

We thought out of all the metrics we could use, that was the one for two reasons. It was being used around parts of the business already, so there was already some buy-in for it. It gave us a score that enabled us to then go on and create very designable problems and solutions from that. The score felt like a natural fit.

We started off using the customer effort score and we trialed it on a few studies. It was great. As Georgie says, we use it at the end of talking to a user for 30, 45, and 60-minute interviews. We have a section at the end where we ask the question, “Based on the design you've seen today, how easy would it make it for you to achieve whatever goal we're testing for?”

We were seeing it was working very well as a point in the interview to help them articulate the [design functionality]. It was really nice. They'd reflect on it and give us something that we could very easily go back into the research notes and find their summary. We noticed the limitation of using the score by itself. The score didn't separate whether the value proposition in the idea was good, and whether the design was good.

We were finding situations where we were fishing in the right water and we would hit a really good pain point for them, but they might not like the design. Or they liked the design, but the pain point was wrong and we couldn't separate it with just the CS score.

We then created a custom score. There might be other scores out there like it, but we used the same seven-point scale and we called that a Customer Proposition Score. So the idea at that point became that we asked for the CS score and the CPS score proposition score separately. That's two questions at the end of the interview.

We've asked the proposition score first, so that question was irrespective of how it was designed and executed. What's a product or service that tried to solve this pain point for you?

The idea for that was, we can capture that sentiment of whether we’re fishing in the right water and whether we're onto something with the value proposition, and then asking that CES score separately, presuming this is a valuable thing for you. Is the way that we've executed the design would that make it easy for you?

We could then see whether the design in itself had strength. There's lots of situations where we were answering the pain point, but the design needed a tweak and we could definitely look at that then. There were other situations where we didn't have the right value proposition, but there was something in the design that people really liked.

Going back to that cross quad thing again, there's bits of designs that we can recycle for this product for future studies with a different value proposition, or it might work for one of the other teams. Or it shows that the design worked, so it did double the work.

We got to test out different design patterns and at the same time, we could use it as part of our wider design system. It was a very efficient way of doing it, but then we noticed a third thing. This is where the continual iteration of these scores came out that, like Georgie said, we're predominantly a qualitative team.

We do a lot of talking to small cohorts in a lot of depth, and we were finding that was part of the interview. We were often building up quite a good rapport with people.

We'd spent a bit of time with them, and then at the end when we asked for these scores, particularly the effort score. They would give us a great score because we'd explained what these products were, and they'd got to like us a bit during the interview. But really we were looking back going, ‘They stumbled over that bit and they misunderstood a core part of the value proposition.’

So we introduced this third score, the usability expert (UXS) score. That is a heuristic measure that we will take after we've done each interview or after a bunch of interviews. The researcher and usually the designers who have been taking the notes as well, then look at the scores that the customers have given us.

Through our UX expert eyes, we see whether we need to adjust any of those scores. So for instance, the customer might have loved this product, but we actually know that they fundamentally misunderstood something in the copy or the design.

It might be they've given us a low CES, whereas actually what they disagreed with was the proposition, and they just got confused between the scores. So it gives us a little wiggle room there using that heuristic expert view to adjust the scores as well.

We report all three of these back to our stakeholders as well, so they've got a really clear sense of the value proposition. Did the customers say they liked the design? Are there any sort of little tripwires or gotchas that we've picked up as a UX team, where we have to clarify some of those scores or maybe give a little bit more context? These three together have worked really well.

How did stakeholders react or interact with the metrics and scores you reported?

Georgie: I can talk to this in the sense of how we're educating our stakeholders in using these scores.

Now that we've been doing it a while, everyone's certainly a lot more familiar. But I think we found at the start—or I certainly found at the start—that there could be an over-reliability on the scores themselves, rather than the sentiment and the patterns that we were seeing. I often will run iterative work where we test things perhaps two, three times, change the designs, and test again.

We would use these metrics at the end of sessions and we would sometimes find that scores, let's say 5.8 in the first round, and 5.6 in the second. We know as researchers in our team that those are both good scores. We shouldn't have huge concerns.

They're both scoring well, but when you present that kind of side-by-side to your product manager or anyone else, it can feel a bit alarming. They think, ‘Oh, but we've been working hard on these designs and it's gone down a little bit.’ I think where the education happens here is, it's not so much the exact score, again. Like we said, we typically speak to a small cohort of users.

On another day, you could have eight different people and the score might've been six, not five or seven or whatever these things were open to a little bit of change, particularly when you're taking averages. It’s been an education that it’s just a way of showing that something's either working well or needs a bit of improvement and not necessarily getting bogged down on the exact score.

I needed to learn that myself as well, because I would also have the same question of, oh no, I from professional experience, think this has got better and I've actually witnessed people using this really nicely, but our scores don't necessarily reflect that exactly, or they're the same.

People get a bit disheartened that they haven't gone up, so I think that's been a learning for us on how to communicate the scores. It's not the scores, it's how we use them and how we communicate that back to our teams—to make sure they are valuable to our teams, and not just a number on a page. We want to make these usable and allow our stakeholders to know what to do with them.

For me personally, that's been a learning that I've found and I've been tweaking myself over the last year.

Ashton: The really cool thing is research serves a role to inspire and drive conversation. The scores have been very digestible and they're very clear, and they allow us to facilitate something that people can take away without having to think about all the nuanced detail.

Nothing's empirical—no research metric in the world can say for a fact—that this is going to be absolutely bulletproof. What it does is, obviously, with great confidence allows us to make more informed choices. But because it's bite-sized, if you dive into details, see the comments, see task completion, et cetera, that engagement has been very well received.

Then people are feeling much more confident to make decisions and discuss, ‘How do we move the needle up or down?’ Or, ‘How's that going to track over time for these different layers of the product development cycle?’ It takes a little bit of time to get your head around it. It's the same as rating error ratings. It's a different way to look at it, and the simplicity has been key.

Nick: The best thing for me is it's given us a shared language with the stakeholders and the users as well. What we're seeing increasingly with the stakeholders is that it's far easier to articulate the difference between a problem with the value proposition and a problem with the design.

When we bring these scores up on screen when we're talking on Skype to our users, it gives us that moment in the interviews and often the video clips that we show where we can hone right in on where the users are discussing these scores. It helps us bring that to life for the stakeholders as well.

We're all a bit worried when, as Georgie said, we were a bit reluctant at the start because we were like, ‘Are these scores going to be used as a stick to beat us with if they go down? Or are they going to be misinterpreted and scaled up?’ We haven't really seen that.

Everybody's been smart and switched on about exactly what they're there for. It’s not really there for the score in itself, though, it is useful to track it. It's more that by asking the question through these metrics, it condenses the focus of a conversation or making a point about the product that we're doing.

It’s been more useful as that shared language in a way than it has as a score.

Georgie: We can't be delivering pages and pages of information every time we do a study. It's just getting that point across quickly in a digestible, clear way for them.

Like we were saying earlier for Nick and me, when we're running these sessions, it helps us condense that wrap-up and focus our attention on what's important here. We've got lots of great work that we've been doing, looking at designs or prototypes—but now what's our focus and what are our core questions?

What were our objectives for this? Let's summarize those. I think these metrics help us get to the core of that in an efficient manner for us and our team.

Wrapping it up

Uniting multiple teams across departments takes a tremendous amount of effort and experimentation. Creating a shared language with stakeholders through customized scoring enabled Vodafone to improve on the product in ways they didn’t think were possible.

Interested in checking out the entire interview? You can watch it here.

You may also like …

• How KEEN Built a Continuous Discovery Process That Aligned Product, Design Marketing, Merch, and Insights

• How SiriusXM Drove Product Innovation Through Bottom-Up Research

• How to Engage Stakeholders in Research for Better Outcomes