With quality assurance (QA) teams under pressure to do more with less, adding the challenge of mobile app beta-testing can quickly overwhelm them. They may have tens—if not hundreds, or even thousands—of testers to shepherd through the beta-testing process. And while gaining feedback about a soon-to-launch mobile app is critical, it's also important not to waste tester or developer time in doing so.
Here are some secrets to help you run a successful mobile app beta test.
1. Consider the merits of closed versus open beta tests
With a small, closed beta test—say 20 to 200 people, consisting of end users, product managers, other internal stakeholders, etc.—you can get more precise feedback, advises Rhine Wu, senior director of engineering for Cheetah Mobile. But when you need to test different behaviors, phone models, Android versions, and configurations, for example, he recommends using an open beta test to increase the number of beta users. In this scenario, he recommends targeting 1,000 to 20,000 valid beta users.
2. Source your beta-testers through multiple channels
Carlos Padilla, director of engineering and operations for GameSalad, advises that, as you crowdsource and engage testers for your beta app, you should start with your online forum users, who are already loyal to your product. He also suggests identifying users who regularly interact with your company through the help desk. They are often champing at the bit for new features and might jump at the chance to become beta-testers for your next app release.
Padilla has seen some startups use crowdfunding platforms to source and engage beta-testers for their apps. With their mobile app in alpha, they put it up on a platform such as Kickstarter with the primary goal of driving awareness and a secondary goal of getting more funding. While the startup takes in a minor investment, the early adopters drive interest and serve as beta-testers.
Beware selection bias
Hugh Reynolds, chief strategy officer of Swrve, sees selection bias as a major problem when a company selects beta-testers. "Selection bias" means choosing people from your local area, people who look like you, or even people whom you know—not necessarily people who resemble, or think like, your real end users.
He told me the story of a company that made a significant UI change to its mobile app. The UI change was met with positive feedback from a 20-person focus group. The company figured its beta-testing was over at that point. But when the app went live, its actual users hated the UI.
Reynolds also warns against conscious bias, when beta-testers tell you what they think you want to hear. He advises watching what they do with a critical eye.
Finding and interacting with beta-testers
One way to get past biases is to solicit beta-testers through online advertisements. Reynolds has seen companies go out and do test buys in the open market for new-user acquisition, using a selection of online content relevant to their target users. The tools out there allow you to spend $500 to get a cross-section of users who are strangers to you and your company, according to Reynolds.
If your app’s back end includes analytics or other monitoring tools, you’ll be able to track how users interact with the app. If you're able to switch on and off features remotely so that you can have 100 of those users see and beta-test feature A and another 100 users see and beta-test feature B, you get true results as opposed to showing the features to 10 of your friends or falling into some other selection bias trap.
“Let them use your app at home, let them use your app in the wild. Don't make it particularly obvious that you are seeing what they are doing,” he says. “This realistic view can show you where your app is sticking versus having them fill out surveys, looking over their shoulder, or relying on self-reporting.”
3. Respect your beta-testers
“Respect your beta users and don’t feed garbage to them,” recommends Wu of Cheetah Mobile. He gives two questions you need to answer when preparing for a beta:
- Did the mobile app pass your alpha testing?
- Is the mobile app stable enough for your beta users?
4. Choose the right beta app management platform
Managing and delivering mobile app betas requires a management platform. The mobile OSes you support, the number of apps you plan to test, and related factors will drive your platform decision.
Apple and Google options
TestFlight (now owned by Apple) for iOS, TVos, and WatchOS enables you to manage beta-testing for up to 100 apps in your Apple Developer Account. It lets you prepare your app for beta and invite testers through iTunes Connect.
Google Play Developer Console supports Android and Android Wear only. It allows you to run open betas, closed betas by email address, and closed betas with Google+ Community or Google Group. Beta apps are made available via Google Play. It’s important to note that beta-testers require a Gmail or Google Apps account to join your betas. Testers can’t leave public reviews of betas on Google Play, so you’ll need to provide an online channel for testers to send beta feedback.
The MAM option
For cross-platform testing, Chris Schroeder, president and CEO of App47, walked me through how to use a mobile application management (MAM) platform to manage mobile app beta-testing. A large enterprise often tests hundreds of apps before a big release and can have a large beta-testing community that it needs to manage and serve. Schroeder explains that beta-testers get authenticated access to the MAM and its app store as part of the onboarding process. Then they can discover and download the right applications from an app store set up to support the beta test.
Schroeder emphasizes that you don’t want any unnecessary time spent with beta-testers, QA team members, or developers filing bug reports on the wrong version. A MAM already has the features to ensure that apps stay in sync with the current version to ensure that every tester is running the same app regardless of level of participation or technical knowledge.
He also mentioned that a MAM can capture issues and track information during a beta and pass it to a trouble-ticketing system if you integrate both systems. The MAM analytics also enable you to understand app stability.
Once the beta app passes functional testing, you can use the MAM to shut down the testing apps over the air (OTA), preventing an early release leaking to the public. In the case of iOS, this shutdown also enables you to stay in compliance with your security certificate.
5. Beta-testing goes hand-in-hand with agile and continuous integration
"I like to think beta-testing goes hand-in-hand with agile development and continuous integration as part of the best practices of engineering culture,” says Tom McLeod, CEO of Omni. He sees it as another level of continuous integration similar to writing a smoke test or testing script that you’d be putting up against your code to ensure that you’re not adding new bugs and that everything is working continuously.
He further advises that you go through a continuous integration loop with mobile apps, where you're asking, "Now that I have now added a feature, what are the things we always do when we integrate new code?"
During the architecture phase, his company adds levers to turn features on and off during beta tests. It also adds permission controls to different groups based off an IP address or a phone number. Beta-testing is at the core of its products, so to speak.
6. Establish a beta-testing community
Wu advises creating a community to share information, report issues, and promote a connection between you and your beta-testers. He stresses that community is an important platform to help get feedback from your beta users.
7. Accepting beta feedback can be a balancing act
“If I have two or three people come back and say, ‘You know what, guys? I'm really struggling with this,’ or, ‘This is fantastic, but it's hidden,’” says Hilton Barbour, CMO of dubdub, then “you need to find a way to push it forward. It’s the critical thing we have to work through with that balance between marketing and product engineering.”
For example, a month or more out from launch, Barbour has more frequent discussions about beta feedback because the decision was made to stop app development at that point. The company isn't done taking input, but it is done actively looking at the impact of the beta-testing feedback and trying to do something with it for that release.
Of course, obligations made to investors or a board of directors may also dictate how you manage beta comments.
Balancing everything
Mobile app beta-testing is about balancing your internal resources, external beta-testers, and feedback to make the best use of everybody’s time to ensure that you get actionable feedback about errors and omissions in your soon-to-launch mobile app.
What mobile app beta-testing secrets do you have to share? Let us hear from you via the comments section below.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
