While there are pleny of scenarios where test automation make sense, some test scenarios should never be automated through the browser using a tool such as Selenium WebDriver.
Here are 10 examples, along with my explanation of why automation just doesn't make sense in each case. I'll also give you an alternate approach for each.
1. CAPTCHA
CAPTCHA is short for "Completely Automated Public Turing test to tell Computers and Humans Apart." It exists to prevent automation, so it's not even worth trying. Otherwise, it fails to be CAPTCHA, because it won’t be able to tell humans and computers apart.
Two primary strategies exist to get around CAPTCHA checks during testing. They are as follows:
- Disable CAPTCHAs in your test environment. This could be a simple configuration in the software under test. Or this can even be a URL parameter that the test sets.
- Add a hook to allow tests to bypass the CAPTCHA.
2. Visual testing
Visual automated testing means checking how a page renders and looks to the end user. This is extremely valuable to check on multiple devices and screen resolutions. Many of us try to do this by checking for the presence of dozens or even hundreds of elements on a single page with code. WebDriver is not the correct tool for this.
Instead, it's better to use a dedicated visual testing tool. Here's an example of me testing my website on the most popular screen resolutions:
And here is the simple code to make this happen:
3. Two-factor authentication
Another scenario that you shouldn't automate through UI is two-factor authentication (or 2FA). This is where a one-time password is generated using "authenticator" mobile apps such as Google Authenticator or Microsoft Authenticator, and sent by SMS or email.
It's a significant challenge to automate it in Selenium, but that doesn't mean it can't be done. And while you can automate it, it's another layer to add and is not as secure. Therefore, it's best to avoid automating it altogether.
The three options to get around 2FA checks are as follows:
- Disable 2FA for certain users in the test environment, so you can use those user credentials in the automation.
- Disable 2FA in your test environment.
- Disable 2FA if you log in from certain IPs. (This way you can configure the test machine IPs to avoid this.)
4. File downloads
It’s less than ideal to automate file downloads through UI because the API doesn't expose download progress. Downloading files isn't considered an essential aspect of emulating user interaction with the web platform.
Therefore, you should consider completing this workaround: Find the link using Selenium, along with any required cookies, and pass it to an HTTP request library, such as REST Assured (Java), fetch (JavaScript), or libcurl (cross-platform).
5. HTTP response codes
HTTP status codes are standard response codes given by website servers on the Internet. The codes help identify the cause of the problem when a web page or other resource does not load properly. In automated functional testing, checking the status code is not a particularly important detail of a test's failure; the steps that preceded it are more important.
It's better to keep API testing at this layer. WebDriver is not an API testing tool. Hence, you can use libraries such as REST Assured (Java), fetch (JavaScript), and RestSharp (.NET)
6. Gmail, email, and Facebook logins
Another scenario where you shouldn't automate through UI is Gmail, email, and Facebook logins. It's not recommended to use WebDriver to log into these types of sites. It's against the usage terms, and it's slow and unreliable.
Generally speaking, the longer tests are more fragile and unreliable, which puts you at risk for the test failing. A study of over 2 billion tests confirmed that tests lasting longer than 2 minutes are twice as likely to fail.
Instead, it's best to use the APIs that email providers offer or, in the case of Facebook, the developer tools service that exposes an API for creating test accounts. (The Gmail API is here.)
Using an API might seem like some extra hard work, but you'll be paid back in speed, reliability, and stability. The API is also unlikely to change. However, web pages and HTML locators change often and require you to update your test framework.
7. Performance testing
Performance testing using Selenium and WebDriver is generally not advised because it's not optimized for the job and you're unlikely to get worthy results. WebDriver tests are subjected to many points of external and internal fragility that are beyond your control.
These include browser startup speed, the speed of HTTP servers, the response of third-party servers that host JavaScript or CSS, and the instrumentation penalty of the WebDriver implementation itself, to name a few. It will cause variations in your results. You will get a slower performance test that includes back-end and front-end performance.
Instead, use a free tool such as Google Lighthouse for front-end performance. Then, perform a separate test for load or stress testing using a free tool such as Apache JMeter.
To discover what to improve, you need to be able to analyze overall performance independent of environmental differences, identify poor code practices, and break down the performance of individual resources (i.e., CSS or JavaScript).
8. Link spidering
I don't recommend using WebDriver to perform link spidering—in other words, to spider through links. You can do it, but WebDriver is definitely not the ideal tool for this task because it needs time to start up. That can take up to a minute, depending on how your test is written, just to get to the page and traverse through the Document Object Model.
Furthermore, writing the logic of traversing pages and capturing links is simply a waste of time.
Instead of using WebDriver, there are much simpler methods. A quick search reveals these two free tools:
- www.brokenlinkcheck.com, which found all the broken links on my site in a matter of minutes
- www.deadlinkchecker.com
9. Video streaming
Video streaming is becoming more popular these days, but you may not want to automate it through UI. More often than not, Selenium won't be able to recognize video controls. JavaScript Executor and flex-ui-selenium will work to some extent, but they are not entirely reliable.
Instead, check out StreamTest, a free tool that can measure the end user's quality of experience. You can even add monitoring.
10. Crash recovery
Recovery testing is a software testing technique that verifies software's ability to recover from failures such as software and hardware crashes. You might want to test your application's crash recovery. It is best tested manually. It's not that you can't test it using Selenium, but doing so is not feasible or beneficial.
For reliability testing, data is gathered from various stages of development, such as the design and operating stages. The tests are limited due to restrictions such as cost and time restrictions.
Use alternative tests for these situations
These are the top 10 scenarios to be aware of that you shouldn’t automate through UI. You can refer to them as you're working and trying to decide on the best approach for when you're developing, coding, and testing.
Although there are workarounds in some cases, it's always wise to consider the reasons why these particular elements shouldn't be automated in the first place.
Note: If you'd like to learn more about test automation with WebDriver, I also teach a course, Complete Selenium WebDriver with Java Bootcamp.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
