Football App Validation In A Real-Life Scenario
Intercept field study by UX Studio
Our client creates digital products to help football fans follow matches, favorite teams, news, or even play web or mobile games. Some years ago their team decided to put a new app in football-lovers’ pockets. This app would guide fans as they travel to a match anywhere in Europe. Packed with clever tips, news, sightseeing, and public transport info and more — all focusing on football.
Before design, the team already had:
- Insights from a massive discovery research on the 2018 World Cup
- Over 3000 detailed observation points from our research system
With enough research ammo, they began to build the app. The plan was to introduce it during the EURO 2020 where millions of fans crazy about football would be active. Of course, these plans were made long before the COVID-19 outbreak.
From the Lab to the Jungle
Design and development were going smoothly and genuinely fast. By May 2019 the product team has already had a lab-validated test app with an early version of all the major features. We as researchers constantly supported the product team with insights.
But we had zero knowledge of real-life usage. What will happen when the first users download and touch this app? How will they react?
The product team has decided that the app has to leave the convenient and familiar lab environment, so UX researchers from UX studio were asked to go onto the field to collect insights:
- from real ticket holders,
- in real-life situations,
- using our live app.
Luckily, the finals week of UEFA Nations League appeared to be a convenient time and occasion for testing our application live. The team adapted the product to the competition and flew to Porto, Portugal. ✈️
Research streams
Our goal was set: evaluate a new football app in a real-life context. But how? We listed the questions we wanted to get answers for and divided them into two categories:
- How do people think and behave while adapting the app in the long run?
- How easy is it to use the current app and what contexts people might need to be able to use them?
In order to find answers, we’ve defined two separate research streams for these groups of questions. These research streams operated independently on the field to ensure unbiased findings. Stream 1 focused on longer adoption of the app while Stream 2 collected intercept insights about first-time users.
Stream 1: Concept evaluation
Method: Diary study
With this stream, we wanted to find answers to these questions: What do people think about this app? How do they adapt to it? How does it align with their habits? Is it really something that fulfills the needs of tournament goers in the long run?
Stream 2: User experience evaluation
Method: Intercept field study
With this stream, we wanted to find answers to these questions: How do tournament goers understand our current app’s offering? How do they complete the given tasks? How easy or difficult is it to use the app? What journey steps do they go through when they start using it?
UX studio organized and led Stream 2: UX evaluation.
Challenges
The most time-consuming part of our initial planning was to brace ourselves for every possible scenario — both in terms of research and logistics. We basically sat down on a daily basis and gave a lot of thought on how to cover everything.
Challenge 1: Screening
In order to have a large variety of different users, we had to set up our criteria.
We identified two main user groups for the testing:
- Traveling fans (unfamiliar with the city, living 50+ km away or abroad)
- Local fans (living in or nearby the city)
In order to adapt our most suitable scenario, we wanted to ask about their favorite teams, days of staying in the city, basic demographics, and some of their planned programs.
But how can one verify these on the field? How can we screen them if we only have a few moments before they lose their interest?
We have printed a screener spreadsheet with all the necessary information we wanted to learn. However, instead of approaching people with our boring questions upfront, we immediately started with the fun part: showing them the app. Since people don’t usually go alone to watch football games, the second researcher could target another group member (Secondary user) and ask the screening questions from them.
Once our primary user got excited about the app, we could slowly ask our questions and complete our target task.
Key findings from the field interviews:
- Screen your participants, but collect the necessary information only. Time is extremely tight when intercepting.
- Take notes as fast as you can: even though you do screen recording, essential bits of information can be lost in the noise.
- Use the power of your team: don’t do field study with less than 2 researchers working together. Distribute tasks wisely.
Challenge 2: Location
We wanted to intercept football fans in the environment most suitable for tests and short interviews. In order to do so, we had to run through our initial assumptions about locations and cross-check it with our football experience. These are a few thoughts we had in mind and the challenges we faced:
I guess airports, train stations, fan meeting points, football events in the city center, and so on.
Probably yes. Let’s get written permission from officials to do research on the location sites.
Well, fans are likely to get directions to the city center when arriving at the airport and train stations. They’ll possibly wait at the baggage claim. I assume they would be interested in football-related programs close to their hotels and the city center. They probably want to read news and facts about the events to kill time while they are waiting.
Check arriving flights from supporters’ countries. Check train schedules between cities where games take place. Build a plan to follow the fans’ journey throughout the week.
Key learnings:
- Start your plan with a lot of thinking and map out your on-site journey thoroughly.
- Share your plans early on with the team as they have fresh eyes.
Challenge 3: Improvisation
Planning: check. Now let’s turn your plan upside down. If you’ve ever done any field research, you know planning isn’t everything.
You get there, suddenly realize there’s a construction on your desired location, potential users are nowhere close by and your app keeps on loading due to poor mobile network. What do you do?
These are the reasons that made us go and observe the real context of our product in the first place. Alright, none of the above really happened to us this time, but it could have. Here’s something we experienced though:
Example: participants do not stop
By the time we got to the first location (airport), we have already had a spreadsheet printed with arrivals of countries with upcoming matches. But as we tried catching people coming out of the arrival gates almost everyone refused to help us.
Makes sense, right? You get off a plane, you collect your luggage, you’re probably tired and just want to get to your accommodation.
So we revised our tactics and used our Beaver scout observational senses. Instead of catching people on the move, we started looking for people who waited for something. Where do people wait at the airport? Taxi pickups, public transport stops, cafés. We followed the crowd and realized that the train from the airport was leaving every 10–15 minutes.
What an opportunity! This gave us the perfect time window to test with:
- Groups of Fans waiting for the train and planning directions on their phones
- Fans already aboard the train, trying to figure out what to do in the city
We have spent a day and a half traveling back and forth between the airport and the city center, talking to a lot of people both at the stop and on the train. It’s quite a repetitive way to learn “Next station” in Portuguese if you ask me! 🤷♂️
Football fans that did not want to participate in our test appeared to have another reason: they did not know who we are. People don’t trust some random dudes going around asking a bunch of questions with a phone in their hands.
Luckily, we had initially planned to spend our first morning piloting and fine-tuning our research process. We had applied to get a basic research accreditation for the game a few weeks in advance, which we received the next day. It came with a large badge — displaying our profile photo and name. Very official! We also bought some visible Nations League merch, so football fans could quickly identify us (and it was also a lot more fun to do research in a footy scarf).
Key learnings:
- It’s vital to build up your plan, but don’t be afraid to be responsive and adapt it to any unexpected situation.
- Dedicate ample time for piloting and enhancing your research at the beginning.
- Pause and give feedback to each other during the day. Enhance with the team continuously based on feedback.
- Find the best way to gain trust early on: dress as an official or put on a big ID or company pass.
Research findings
Fortunately, we managed to overcome every one of our challenges and carried out a successful study. The results were invaluable: we have collected insights we wouldn’t have done in any other way.
Here’s a handful of them:
1 — Content structure
The app content structure system worked on the lab tests, but when we went on the field, our system kind of fell on us.
During the intercepts, a number of users started spending too much time jumping back and forth between sections and submenus to collect the small bits of information in the necessary articles. This was overwhelming, and they couldn’t even find the desired content in some cases. Their mental model was built-up in a horizontal structure: “Show me everything I need to know today!”
So the product team put a new “mix” to the home page with relevant info and interesting details of multiple sections. It served kind of as a feed of ‘ most relevant information at the moment ‘. The team kept the original navigation as well but restructured and renamed sections for better findability.
2 — Best feature did not work out
Our initial thought was that the ticket holders would spend most of their time using our custom map that had built-in football interests, shortcuts to football activities, route planning and more.
On the field we learned that although people took their time to use it, they did not find that much value in it. Our football fans, as well as other audiences, were likely to use their favorite apps they’re already familiar with — Google Maps, Apple Maps, Maps.me.
On the other hand, they found value in our Tips: a section where we show them interesting facts, notifications and events going on in the city at the moment that are connected to the games.
3 — Copywriting, Local names
We used a variety of names in our article, search, and map to identify locations.
Some examples:
- Estádio do Dragão
- São Bento
- Guimaraes
Can you quickly tell what these names are? Neither did I. The first one Estádio do Dragão is actually Porto’s home stadium, where most of the games have been played at. São Bento is a train station in Porto city center, while it is a city in Porto’s agglomeration.
We’ve learned that users had problems remembering local names, and identifying what is what. They used terms like “Porto stadium” or similar in their mother tongue. So we had to improve our app by adding English alternatives to locations. After a bit of Googling, we found out that this was an issue that Google Maps has also identified and solved with alternative street/interest names.
So the app team simply added subcategories and alternative names:
- Estádio do Dragão — Porto stadium
- São Bento — Porto train station
- Guimaraes city
We also came up with new name ideas to almost all sections within the app as users had problems identifying the content behind them.
Takeaways
Every field research is an opportunity to check real-life situations. We had the chance to talk and evaluate our football app with more than 50 participants in 10+ scenarios. Then we compared our intercept study with the diary stream to have the full picture.
Our main takeaways from this UX research project:
- Invest in field studies. You can never know how your users will use the app in real-life situations. Intercept offers quick and relatively easy evaluation — combine with longer studies to the best results.
- Have a plan but dare to change it. Review and enhance your plan on site and improvise if necessary.
- Attitude is key. Be friendly, open, to the point, and smile a lot to gain trust early on. However, prepare for plenty of rejections — it’s part of the job.
- Distribute tasks well amongst team members. Time is extremely limited, use it the best you can.
Happy researching!
