Dear codeless test automation tool vendors:
You've worked hard on your shiny new record-and-playback offerings. You're demoing your applications to product teams and expecting them to be as excited as you are about what you have built. But you're getting some resistance. There's a good reason for this, and all you have to do is take a trip down memory lane to realize what's wrong here.
This is an open letter to the vendors of the new generation of codeless test automation tools. We automators have experienced problems with these tools in the past, and here's what we expect from the next generation of codeless offerings.
The problem with record-and-playback tools
Many years ago, record-and-playback test automation tools ruled. If a tester wanted to do UI test automation, such tools were essentially the only options available. There were several offerings, many of them commercial, and a few open source.
Testers felt empowered. We could use these tools to automate our regression tests. All seemed well—until we realized that the test automation projects we were working so hard to build would not scale.
As we were automating against applications that were still under development, maintenance of these automated scripts became a nightmare. Previously recorded web objects were unrecognizable by the tools as the application changed, and updating a test script often required re-recording the entire test.
Simple click actions were easy enough to automate, but many of the tools did not allow for more complex flows that included conditional steps or repetition. Maintaining state between multiple steps within a test also proved quite challenging, if not impossible.
When good tools go bad
Eventually, product teams realized that these codeless offerings caused more trouble than they were worth. Within the industry, record-and-playback tools gained a bad reputation, and were almost universally dismissed as a nonviable option. Testers replaced them with test automation APIs that were geared more toward programmers. And so began the hunt for the automation engineer.
To become an automation engineer you had to possess the skill set of a developer and the mindset of a tester. Automation engineers didn't need codeless solutions; in fact, they thrived on writing code themselves, and they used the newer test automation APIs to help them navigate websites and make assertions. As more began to adopt agile practices, the demand for automation engineers skyrocketed. Unfortunately, there were not enough of them to meet demand.
With a shortage of skilled automation engineers, product teams began to look to other team members to fill this void. Since test automation is essentially software development, product developers were tasked with building test automation scripts. This was short-lived, in many cases because developers got bogged down developing features and unit test, leaving little time for (or interest in) working on more involved test scripts.
Product teams began to look at testers to drive automation efforts. Testers had the testing mindset. They were the ones who felt the pain of regression tests the most. But many lacked the programming skills required, even if they possessed the desire.
And test automation for all
That brings us to the present day. We've come full circle, and the problem to be solved is familiar: How do we enable testers who aren't programmers to do test automation? Hence the resurgence of codeless test automation tools.
However, many teams have heard the horror stories that people like me experienced when using record-and-playback tools, so there’s a general sense of skepticism where these offerings are concerned.
For a codeless automation supplier to have any chance of long-term success, it must be aware of the pain points of yesteryear and provide the following capabilities to ensure that history does not repeat itself:
1. Smart element locators
One of the biggest problems with test automation is that testers develop against an application at a given moment in time. When the application changes, those automation scripts may break. (This is true of both coded and codeless automation frameworks.)
There's an opportunity for codeless solutions to have an advantage over coded frameworks by gathering more than one element locator during the recording. In the event that a locator is no longer found during playback, the alternative locators for that element can be tried. This gives the tester a far better chance to develop stable scripts that don't require as much maintenance.
2. Conditional waiting
Record-and-playback tools that require script authors to insert pauses between steps not only are tedious to use, but promote a test automation anti-pattern as well.
A more modern approach, called conditional waiting, has proved quite reliable. In this approach, scripts don't blindly wait x number of seconds before continuing to the next step; instead they wait until a condition is true and then proceed as soon as possible.
This drastically cuts down on the execution time of the automation suite while also preventing flaky tests. Codeless tools should, at the very least, allow script authors to insert conditional waiting, but it would be even better if the tools took care of this behind the scenes.
3. Control structures
One of the major limitations of older record-and-playback tools was the absence of control structures, such as loops and conditional clauses. A script author who needed to repeat an action 10 times would need to record that action 10 times and maintain each one of the actions individually.
Also, if there were multiple viable alternatives within a step and the test author wanted to take one action if a certain condition was true, and a different action if it was not, there was no support for such logic.
In coded automation frameworks, testers use loops and if-else clauses to control the flow of scripts. This is a necessary requirement for codeless solutions if the project is to be scalable.
4. Easy assertions
Recording a scenario is fairly easy to do. As the user takes action on an application, the tool records those actions. However, when a user validates the actions, that does not usually include a web action. It’s something that you do with your eyes.
Because of this, adding assertions within codeless automation tools has always been a pain, requiring manual entry by the user to describe what needed to be validated. This is neither intuitive nor easily expressed.
The process of adding assertions should be as simple as adding the navigational steps. Great consideration should be given to this feature, since it's the most important part of the test script.
5. Modification without redo
Most older record-and-playback tools did not allow authors to easily edit recorded scenarios. If the flow of a scenario changed and you needed to insert a new action in the middle of the scenario, you had to re-record the entire scenario. That's not ideal.
As the application under test changes, the test scripts need to change as well. This may require adding or removing steps from a flow, or slightly changing existing ones. Codeless automation tools should recognize this as inevitable and make it as simple as possible to accomplish this.
6. Reusable steps
Some steps exist in multiple scenarios. For example, logging into an application may be something that the majority of tests must do. Recording this action in each of those tests is what leads to maintenance nightmares. If anything changes with the login step, the test author would need to update every test that contains this.
Codeless tools should allow authors to record common steps that they can then insert into any test flow. This would not only save time in authoring, but also promote reusability and make maintenance much simpler. Similarly, the tools should support the use of variables within tests to allow for reuse of state between the steps.
7. Cross-browser support
Many record-and-playback tools come in the form of a browser extension, which means they may be tied to a specific browser. But many product teams must run tests across multiple browsers. Test authors should be able to record a scenario once on a given browser, and be able to play that recording on any other major browser.
8. Reporting
Adequate reporting is a must-have for test automation, especially when testers are executing hundreds—or thousands—of tests. Triaging any failures should not require reruns or extensive debugging. An automatically generated report with details on what occurred, screenshots, and even a video of the recording would be very helpful.
9. Ability to insert code
No codeless tool can incorporate everything that's possible. So test authors should have the ability to let testers insert code within the tests for those edge cases. For extra points, don't limit this to UI-specific code, such as JavaScript. Many teams are looking at models such as the Test Automation Pyramid and exercising certain actions as web services instead of UI actions. This will allow for faster, more reliable test scripts.
10. Continuous integration
Older record-and-playback tools required testers to click a button within the tool to execute their tests. In the era of DevOps, test automation is a key ingredient in continuous integration and continuous deployment. Tests should integrate with such pipelines and automatically execute when triggered. For faster execution times, tests should be capable of running in parallel as well.
It's your turn to weigh in
Vendors that include these 10 features in the next generation of codeless test automation tools will address much of what makes testers hesitant to adopt codeless test automation products.
As for my fellow testers, are you with me? What other features do you think should be required in the next generation of codeless test automation tools? Post your wish list in the comments below.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
