The Android platform, now more than 10 years old, has plenty of testing tools available. Java is even more mature, with a rich assortment of testing tools for Android projects.
Then you have open-source UI testing frameworks, including Appium, Espresso, Robolectric, and Mockito/Powermock, from which to choose. All of this can be confusing, especially if you're just getting started with Android.
Whether you're new to Android and trying to sort through what's available or you just want to get up to speed on your options, this overview of four key open-source testing frameworks should help you decide which best fits your testing strategy.
JUnit is the foundation
First, let's start with the basics: JUnit is the default test framework for Android. Mockito, Robolectric, and Espresso all use it as the backbone and build on top of it. This is one decision you don't have to make.
More specifically, AndroidJUnit4 is the current runner that provides utilities to run Android tests. After recent releases, AndroidJUnitRunner was deprecated, and AndroidJUnit4 became the single runner for both Espresso and Robolectric tests.
As you might guess, there's an AndroidJUnit5 on the horizon, and you can get native support for JUnit5 style tests within Android.
Appium
No discussion of mobile testing frameworks is complete without Appium, the native mobile testing framework that works with Android, iOS, the Windows desktop, and Internet of Things devices.
When and why to use it
- For very high-level, flow-type tests.
- For testing on exact PROD apps as a user would. It provides the highest fidelity to your end-user scenarios.
- For integration with other apps, such as settings and dialers, and for system-level options, such as push notifications and widgets.
- For teams that have similar apps across platforms and want to have a single test repository that can target all of them.
- For teams that have web Selenium experts and want to leverage that expertise for native app automation.
- For teams that have hybrid or React Native app and Appium tests that can be co-located.
Why it might not be a good fit
- It's slow. Its round trip HTTP-based architecture means each call to find an element is slow. Native apps use faster locators.
- Appium code typically is not co-located with mobile app repos and can quickly get out of sync.
- The framework is highly dependent on open-source contributions. As such, it might lack documentation and support for the latest releases.
Recommended CI setup
- Appium runs daily on a very small set of end-to-end high-value flows on multiple devices.
Espresso
Espresso is the default UI automation framework for Android—even Appium uses it under the covers. While you had few alternatives in the past, Google has been steadily improving and investing in Espresso, making it unparalleled in support and adoption in the Android community.
When and why to use it
- It's very fast because of its insight into an app's status.
- It's a good choice for dev-focused teams that do not have any prior test automation experience.
- It's co-located with dev code, which makes it easy for tests to be in sync with changes.
- You can reuse helpers and mocks that you created for unit tests.
- Google has a vested interest in it, and teams are working on making this framework forward- and backward-compatible across devices.
Why it might not be a good fit
- Your team is experienced with Selenium and is also responsible for UI automation on other non-Android platforms.
- Your developers do not consider UI tests to be part of their responsibilities.
- Your team doesn't want to maintain separate sets of tests for iOS and Android.
Recommended CI setup
Create a small set of smoke tests that you run on every pull request.
- Build a bigger set of regression tests that you run on every release candidate.
- Use emulators on cloud labs.
Roboelectric
The Roboelectric UI testing framework, which went mainstream in the last few years, has parallels with Headless Chromium. Roboelectric has no physical UI, and tests operate in a simulated environment. Android SDK framework APIs are available, and view interactions are mocked. The biggest differentiator between Roboelectric and Espresso is that Roboelectric has no need for an emulator or device.
When and why to use it
- You want to test your UI views without the overhead of emulators or cloud labs.
- You want tests that are very fast (~2 sec.) versus merely fast (~10 sec.), as with Espresso.
- You want tests that exercise the UI in various error and edge cases.
- You want to test activities and fragments in isolation.
Why it might not be a good fit
Your tests navigate from one screen to the other.
The setup required to mock a test is becoming heavy and cumbersome.
- Your CI setup isn't scaling and is running into out-of-memory exceptions, with hundreds of Roboelectric tests.
Recommended CI setup
About 300 tests that run after pure unit tests.
Google is contributing heavily to Roboelectric, and with Project Nitrogen, the lines between Espresso and Roboelectric are blurring.
Mockito and Powermock
Mockito is a mocking framework for unit tests written in Java. It's similar to Powermock, but provides additional mocking functions, such as singletons. Powermock's GitHub page describes it as "a Java framework that allows you to unit test code normally regarded as untestable."
Unit testing mocking frameworks are ideally suited to view model testing and helper classes testing. In well-architected code, say following an MVVM or MVP pattern, about 50% of the code should be tested using unit testing frameworks. And these frameworks shouldn't need Android SDK hooks.
When and why should you use it
- Your code is modularized and componentized and can easily be mocked.
- You want tests that are very fast (milliseconds).
- Your team cares about code coverage metrics.
Why it might not be a good fit
- You need test UI rendering.
- Your tests need access to Android SDK functionalities such as lifecycle, intents, threads, etc.
- The setup required to mock a test is becoming heavy and cumbersome.
- Tests become brittle, and each app change requires modifying test code as well.
Recommended CI setup
For every pull request, run all the unit tests to provide instant feedback. Unit tests should cover about 50% of the code and are usually in hundreds.
Putting it all together
You can visualize all these frameworks in a class test pyramid that combines the idea of how many tests are recommended at each level and their foundational nature.
Source credit: Satyajit Malugu
Note that there can be exceptions to this general categorization. For example, you can use Espresso as a complete black-box tool if you don't use some of constructs provided in the framework.
Now, get started
Google and the open-source community have done a phenomenal job of creating various testing frameworks for different levels of Android. With the information above as your guide, you can start customizing your own Android automation strategy.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
