One of the biggest problems with test automation on any platform or operating system is synchronizing test automation interactions with the user interface (UI). It's challenging to know when the UI is ready for the next automated click.
Traditional black-box tools try to address this problem through explicit or implicit waiting, but this technique is slow and error-prone.
Newer test frameworks—including Espresso, Cypress, and EarlGrey—understand the internals of the app and synchronize interactions only when the view is ready, making the frameworks very fast and reliable.
Here's how to use these tools to make your tests less flaky.
Are you gray box or black box?
There are two schools of thought when it comes to the operating principles for a UI test automation framework. The black-box school values fidelity over speed and reliability, and treats the system under test (SUT) as a pristine sandbox. It says you should operate on the SUT as the end user would, and perform interactions and validations without any internal knowledge.
But testers in the gray-box camp value reliability and speed over fidelity. They operate with a knowledge of the state and the internals of the SUT.
The gray-box folks have a higher visibility into the SUT than do the black-box adherents. More concretely, you would use a tool such as Selenium to operate only on the SUT, with knowledge available to accessibility tools such as element types, element tree, names, values, colors, and so on.
But gray-box tools can access more details—such as the state of the UI thread and the network calls state—which allows them to synchronize the SUT. Synchronization in this context means bringing the UI to a "ready" state before performing any operations on it. Gray-box tools have knowledge of the SUT and can interrogate the internal state before performing test actions.
What happens without synchronization
Without synchronization, black-box tools must rely on waits (also known as timeouts). Many automation implementations rely on a version of code similar to this:
Thread.sleep(5)
This line puts the test code to sleep for five seconds, and you hope that the SUT is ready. But where did the number five come from? It's usually through trial and error. A random network call that takes longer can break your previous assumptions, and this could lead to very high timeout values and slow automation suites.
With traditional tools, automation engineers have become accustomed to playing defense, sprawling waits and sleepers across their code. While you can apply some intelligence through loopers and backoffs, these can get complicated and on a bad day can still fail.
Getting an important heads up
It would be nice if the test tool could let you know when an application is ready. That's exactly what these three tools do, on three different platforms.
Espresso
Google introduced Espresso in 2014 after several attempts at an Android UI automation framework. One of key tenets of espresso is its built-in UI synchronization. Without the need for any extra code, tests will implicitly wait for the main UI thread to be idle before performing any test code actions.
The following piece of code will work on any device of varying capabilities in speed, on any network condition.
Notice that there are no sleepers, no defenses, and no waits because the first test action, onView, will start operating only after the activity has completely launched. Also, after the statement on line 5 executes, the next onView method, on line 6, is guaranteed by the Espresso framework to start after any UI actions, such as animations or loaders, finish executing.
Also, this synchronization mechanism is quite extensible in Espresso. Say the application makes a network request and can be in a "ready" state only once the response comes back. You can implement a custom idling resource counter that can wait for the request to come back. This removes the guesswork on how much time to wait. which adds to speed and reliability.
EarlGrey
EarlGrey can be viewed as a twin brother of Espresso because it also came from Google and synchronization is a key design factor. But this tool is for iOS. This synchronization can be done at the UI level, at the network request level, and in custom events. Also, you can write EarlGrey tests in either Swift or Objective-C.
Since Android is open source and started first, Espresso is much more advanced than EarlGrey. But EarlGrey is a good alternative to the default XCUITest framework that Apple provides. Also, it can work along with XCUITest and share the same codebase and libraries.
Cypress
Cypress is the new kid on the block, challenging the age-old Selenium with an unique approach to web test automation. While not broadly advertised, Cypress network synchronization is one facet of how this tool can achieve high speeds and reliability.
Cypress explains on its website:
Most testing tools operate by running outside of the browser and executing remote commands across the network. Cypress is the exact opposite. Cypress is executed in the same run loop as your application. Behind Cypress is a Node.js server process. Cypress and the Node.js process constantly communicate, synchronize, and perform tasks on behalf of each other.
Adopt a comprehensive testing strategy
In a classic test pyramid visualization we can think of gray-box tests as sitting one layer below black-box tests, but they still operate on the UI and sit above your API or unit tests.
Note that a comprehensive test automation strategy consists of all black-box, gray-box, and unit tests. Each provides a different type of feedback about your application, and all of them play an important role in the software delivery lifecycle.
For more on how to use these frameworks to make your UI testing more efficient, come to my talk at STARWEST 2019 in Anaheim, which runs from September 29 to October 4 in Anaheim, California. TechBeacon readers can save $200 on registration fees by using promo code SWCM.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
