At some point in your test automation, you will run into a common problem: Too many automated UI tests take too long to execute.
The most powerful way to speed things up is to execute tests in parallel. Unfortunately, getting automated tests to run in parallel is not so easy. Typically, thread safety is not something that is baked into automation. Thread safety ensures that the code is written so that a program can run and manipulate data structures without unintended consequences.
There are mandatory requirements for tests that can run in parallel. These are the same conditions that some of our customers use to help run 50,000 automated tests per day. Here are the four requirements.
1. Your tests must be atomic
Your automated tests should form a single irreducible unit. This means tests should be extremely focused, and each should test just one thing. A single automated test should not test something such as end-to-end automation.
A good rule of thumb I use in my teams is that an automated acceptance test should not run longer than one minute on your local resources; it's actually better to have even faster tests, but, baby steps. If your test runs longer than one minute, then that might be dangerous.
Here is an example of a non-atomic test (NAT) that's available in this repository. This article will refer to this test throughout.
Atomic tests fail fast
Why use atomic tests? First, they allow you to fail fast and early. This implies that you will get extremely fast and focused feedback. If you want to check the state of a feature, it will take you no longer than a minute.
[ Special coverage: Automation Guild online conference ]
Atomic tests decrease flaky behavior
Second, writing atomic tests reduces flakiness because it decreases the number of possible breaking points in a test. Flakiness is a concept in test automation where a test exhibits invalid behavior such as passing when it shouldn't or failing when it shouldn't. A non-flaky automated test is one that conveys the correct state of the system—whether the system is working or not working.
Flakiness is less of a problem with unit or integration tests. But it is a large problem with acceptance UI automation.
For UI automation, every step is a chance for something to go wrong. A locator may have changed, the interaction mechanism may have changed, your synchronization strategy may be broken, and so on. Therefore, the more steps that you add, the more likely your test is to break and convey false positives.
Atomic tests allow for better testing
A third benefit of atomic tests is that if one fails, it will not block you from testing other functionality. For example, if a test fails on line 25, then none of the other features after that point will be tested, assuming that you don't run other tests to check this functionality. As a result, if you have large tests, you will actually reduce your test coverage because some features might never be tested.
Here is an atomic test that was taken from the large test above. If this test fails, you can still test other functionality.
2. Tests must be autonomous
An autonomous automated test is one that does not rely on the outcome of another test. One common anti-pattern that evolves from trying to speed up test suite execution is linking tests together to avoid repeating test steps.
For example, you might have a login test that executes before a search test that executes before a checkout test. All the tests must occur in the correct order to execute the checkout scenario.
This is a huge problem for parallelization, because it means that your tests can never execute out of sequence. If tests 1 through 3 do not run in order, they will fail—not because there is a bug in the application, but simply because the checkout scenario was expecting a logged-in user and the login test failed.
3. You need to correctly manage your test data
Test data management is critical to achieving parallel test execution. The most effective way of managing test data is to use just-in-time data. This means that you have some way to create and destroy data at runtime. A RESTful API is a great way to make this happen, and is the optimal approach.
Unfortunately, I didn't have a RESTful API available for my web application above. Hence, I worked with the developers to allow JavaScript injection to control the state and data of the application. It allowed my team to figure out how you can check out from a cart without logging in, searching for items, and adding items to a cart beforehand.
Done with appropriate test-data management, I can open the cart page directly. As a result I can bypass the login page and search for the item. I inject a user and an item into my shopping cart like this:
Now I can use the UI of the application to finish the checkout process and test whether it works, like this:
Ultimately, this provides you the capability to bypass any UI page in the application that isn't relevant for testing. After you manipulate the state of the application using an API or JavaScript, you can perform the relevant actions with your UI automation tool.
4. You can't use static keywords
As a general rule of thumb, do not use a static keyword in your automation code. This is the simplest rule to follow—and the most sinister.
A static keyword in object-oriented programming languages is one that tells the program to allocate a single memory space to a variable for the life of the program, meaning it can’t be altered. Anything that uses a static variable will use the information stored in that single memory space.
A single, poorly placed instance of a static keyword will destroy all of your hopes of parallelization. There are many exceptions to this rule, but it's better to be safe than sorry.
For example, if you create a static browser driver to use in your browser, when you try to run all of your tests in parallel, they will all try to use that single instance of the browser driver. One test will be trying to type in a username while another tries to open a different page, and so on. As a result, the tests will pull the single browser driver instance into many directions, causing weird failures.
Making tests fast and accurate
Automation needs to be fast to be useful, and executing tests in parallel is the best way to achieve the fastest possible test suite execution. But to run in parallel, you need to have atomic, autonomous tests that don't use a static keyword, and create tests that correctly manage test data.
For a more in-depth discussion of how you can achieve parallel execution in automation, come to my session at the Automation Guild online conference, which runs February 4-6, 2019. My session will be held on February 4 at 10:00 am EDT.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
