Testing the user experience (UX) should start in development and last into post-production to be truly effective. Here are some ways to improve your mobile UX testing:
- Determine your mobile app metrics based on your market
- Test your mobile app UX repeatedly during development using the devices and OSs that are most popular with your current and prospective customers
- Conduct mini focus groups about your mobile app UX
- Factor localization into your mobile app UX testing if feasible
- Implement a user feedback feature in your mobile app to get feedback directly from users
- Use continuous monitoring of your mobile app's UX well into post-production
Amichai Nitsan, research architect at HP Software, points out that many people think of mobile UX testing as nothing more than looking at the application performance. In this article, you're going to see that UX testing is much more than that.
Determine your mobile app metrics
Because there can be an endless array of mobile device and OS combinations to test, Nitsan advises choosing the most popular devices and OSs that your current or prospective customers are using by business priority. Your UX testing choices should account for a variety of operating systems, screen sizes, and device types.
Choose the most popular devices and OSs for UX testing that your customers are using.
— Amichai Nitsan, research architect, HP Software (@aminits
His tip here can also be scaled up to your app's UX prior to moving into a new market. He gave me the example of having a mobile app that's popular in the European market. You know the popular device types and OSs that your customers use day to day. But if you want to penetrate the Chinese market, and you don't have much, if any, presence in China, you can't just plunge in. You'll need to research the market, especially the mobile device types and OSs that your target customers are running in that new market.
Test your mobile app UX repeatedly
Jack Stockholm, head of UX and UI for Colligo, a developer of mobile collaboration apps, told me his company follows Jakob Nielsen's guidance on testing apps repeatedly with three to five users. He adds that this approach allows them to iterate designs rapidly and respond to issues and ideas so their team doesn't have to spend time responding to the same issues during subsequent tests.
Andrej Kostresevic, CEO of Nomads, a mobile software development company, adds, "When you're developing the app, this is not a single cycle. You spend three months building the app, or six months building the app, and then you get some feedback from the users, and then you're done. We like to test UX frequently along the way."
"It can start early before there is even an app," Kostresevic advises. "You could start with paper prototypes internally or whiteboard sketches. Soliciting feedback from potential users off of a sketch or wireframes."
Michael (Micha) Modan, VP of engineering for Clarizen, a collaborative work platform provider, also advocates mobile app prototyping early on in the development process. He recommends InVision or AppTaster for mobile app prototyping.
Kostresevic also recommends early pseudo function interface testing as part of your UX testing, in which there's no database, no server, and no real data, but the interface has been implemented.
"It's an app that you can put into users' hands just so they can get a feel for how the application functions," Kostresevic explains. He says it's an important step when testing innovative user interfaces and getting early feedback and reactions. You would then implement more app functionality during alpha and beta pre-launch testing prior to launching your app.
While UX testing is far from automated, you might still be able to find opportunities for automated testing. Specifically, Nitsan mentions automated testing of your app's battery consumption, because users may uninstall apps that drain their batteries.
Other UX testing solutions include Userlytics, Appsee, and Swrve. If you want to beef up your mobile app UX testing, I encourage you to try one of these apps or another using a free trial before you make a final purchase.
Conduct mini focus groups to test your mobile app UX
Nitsan recommends conducting mini focus groups every two to three weeks to test your mobile app UX. The groups should include three or four people who are new to your mobile app. These focus group attendees can even be company employees. You can use these mini focus groups to conduct some manual usability testing by asking them questions about task performance and their interactions with the app.
While some say you have to videotape the interview, Nitsan says most people never have the time or video resources, let alone the time to watch the video after the focus group session. He points out that you can see immediately how a user works with your app and see if they understand the design. They use only one version of the app during the focus group, and then they analyze the user feedback themselves.
Nitsan and his team use the same UX testing plan when showing focus groups different versions of a mobile app under development. He recommends that no focus group ever see the two versions of the mobile app, because it works better for feedback.
Test your mobile app UX with multiple mobile device types and OSs
Somewhat related to localization, Nitsan recommends testing your mobile app across multiple versions of iOS and Android to ensure the integrity of your mobile app's UX: "Languages are only one of the variations that you have to deal with when developing software in general."
Changing device factors, such as moving from a tablet to a phone, or from a larger skin phone to a smaller skin phone, can impact the UX of your mobile app, he advises. There's also the operating system and the screen size of the device, as well as a huge range of device and OS combinations that can complicate your UX testing, including error messages and other user interface elements.
Nitsan gave me the example of a mobile app displaying correctly in Android 4.4, and then the same app not displaying correctly on another device running Android 5.2.
Device factors are a general problem in testing, according to Nitsan, but they also affect the user experience. That's because sometimes the mobile app will appear to function and even pass some automated testing, but it's when the app gets into users' hands that these sorts of issues appear, and they can lead to user abandonment of your mobile app.
Implement a user feedback feature within your mobile app
Another recommendation from Nitsan is to include a feedback feature in your mobile app. It's impossible to speak with all your users personally, so there are user feedback tools available that let users comment on defects and UX directly from your mobile app. You can then view the comments from a cloud-based dashboard.
"It's valuable when your customer is writing [about] the defect," Nitsan says. It can be difficult to speak with customers directly, especially if a defect is impacting a small portion of your user community, making an in-app feedback form a valuable communications channel.
Use continuous monitoring for mobile apps well into post-production
Eran Kinsbruner, mobile technical evangelist at Perfecto Mobile, a testing platform provider, recommends continuous monitoring of mobile apps. Continuous monitoring solutions include analytics and specialized app monitoring solutions.
While there's a range of tools for identifying UX problems, Nitsan recommends using analytics tools to monitor how your users interact with and use your app. It's similar to web application monitoring, but with mobile apps, you can capture when somebody leaves the app. A web user could still have a web app in another browser tab, meaning it could take you a day or two of the user showing as inactive in your analytics reports.
Often it's obvious why a user left a mobile app, according to Nitsan, because it's a common operation. However, analytics can return information on situations where you didn't expect users to exit your application. He advises pulling the following analytics data when identifying UX issues:
- Device
- Operating system
- User information
There are some analytics tools that support user personas and even defining events, user paths, and funnels through your mobile app.
Other data you may want to track with analytics tools include mobile carriers and device firmware versions.
Using analytics tools is a sure way to trace potential UX issues to their OS or device type. You can then use that actionable information to bring in UX specialists to create a fix for a future iteration of your mobile app. But, Clarizen's Modan cautions, you should avoid app crashes at any cost.
"Most of us will give a website another chance, but after the second time, most users will not return to the site," Modan says. The same thing goes for mobile apps, but after multiple crashes, users are going to start ignoring the app and then uninstall it, and you're off that user's radar.
Modan recommends investing in tools like Crashlytics (a weight crash reporting solution that reports app crashes), not only for your initial app release but also for as long as your mobile app is under development and maintenance.
Invest in a weight crash reporting solution to use while your app is under development and maintenance.
— Michael (Micha) Modan, VP of engineering, Clarizen
"It's called synthetic monitoring and [it] get[s] your NOC [network operations center] team, or your DevOps team, getting on-time alerts when something is either below threshold that is set from [a] performance perspective, or something is not responding," Kinsbruner explains. "Post-production monitoring would be an additional layer to continuous monitoring of end-user experience."
Post-production monitoring would be an additional layer to continuous monitoring of end-user experience.
— Eran Kinsbruner, mobile technology evangelist, Perfecto Mobile (@ek121268
Investigate and fix cases of "you are wrong; the app is right"
Some app design issues may adversely impact your mobile app's UX. Take, for example, the sign-up and registration in a mobile app. There's a policy set for the app password to include a certain amount of numbers and characters. This policy may not be documented clearly. A user repeatedly enters their desired password that doesn't meet the policy criteria. User frustration ensues.
This example of a password issue is a case of "you are wrong; the app is right" according to Nitsan. It's a situation that's hard to follow in normal UX testing because you only see it when you deploy your app. During typical functional testing, testers are following the policy as part of the test cases they must follow. He recommends that you continuously track error messages and malfunctions. It's not an error message, because there's a certain policy, and you didn't follow the policy, he explains.
It's not an error message because there is a certain policy, and you did not follow the policy.
— Amichai Nitsan, research architect, HP Software
If you see a lot of these messages, Nitsan recommends that you try to trace them back to a user misunderstanding, because it may be time to re-engineer that feature during a future iteration of your mobile app. Developers usually develop internal tools to track such messages, but Nitsan says that there are also commercial tools that can fill this gap.
Fixing UX issues around error messages is going to require input from the product manager or another product stakeholder with responsibility for overseeing the product, according to Nitsan. A UX expert can then revise and update the troublesome error message.
Monitor UX between mobile app versions
When releasing a new version of your mobile app, it's advisable to compare metrics with previous versions of your app. Nitsan advises comparing the following metrics between versions:
- Performance
- Stability
- Resources
- Errors
One example of UX issues between versions would be unforeseen user impacts that occur after the planned removal of a feature as part of the new version. You may also come across UX issues that were completely unplanned as part of the new release. Analytics and monitoring tools can clue you into both of these potential issues.
Factor localization into your mobile app UX testing
Nitsan also touches upon the importance of factoring app localization into your UX testing. Often, the localized screen text may not be clear enough, he cautions. There are also translation issues that may interfere with screen space on the mobile device, which can impact the UX of the localized app.
Ideally, you would like a tester to test something in many languages, but that can be difficult in most cases, according to Nitsan, because translations normally happen after the release of a mobile app.
Unfortunately, there doesn't seem to be a formula for UX testers to collaborate better with translators, because as Nitsan explains, localization starts at the design stage. However, he mentions that there are solutions that enable you to run tests on several devices. If a mobile app UX test fails on multiple devices, you could then run the same test on multiple devices, running different localized versions of your app to aid you in identifying language-specific UX issues that are lurking within it.
Final thoughts
To come away with a smile from UX testing, you need to develop a continuous and iterative UX testing process that begins in development and lasts through post-production. Here are some steps to building out your UX testing process:
- Adopt a continuous UX testing mindset, beginning in app development, using continuous monitoring to track UX issues well into post-production. Think of it as an iterative process that will carry you through multiple app releases.
- Determine your mobile app UX metrics based on similar apps in the industry, your competitors, and market demands to create UX metrics upon which to judge your mobile app.
- Conduct multiple focus groups with users outside your project to show them a version of your mobile app to get their UX feedback. Attendees should be new to your mobile app.
- Test your mobile app UX across the multiple devices and OSs that are popular with your current and prospective customers. This may lead to outsourcing your device testing to a third-party testing provider or device cloud host.
- Factor localization into your UX testing if feasible. Because localization takes place after UX testing, you may need to consult your localization provider on how to best integrate UX testing at this stage.
- Test your mobile app UX well into post-production using weight crash management and other continuous monitoring solutions.
How do you come away from UX testing with a smile?
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
