You just finished automating your regression test suite. Or perhaps you now have 4,731 behavior-driven "test scripts" that you run on each deployment. Hooray! T-shirts for everyone!
Then it happens—your product changes, your tool version changes, or the infrastructure changes. Your test suite quickly becomes an obsolete, rusted, unusable pile of shattered dreams and wasted effort.
Having automation to support your testing is like owning an old pickup truck. Both require proper care to avoid gradual corrosion. Typically, we call this maintenance.
As with your vintage truck, your carefully crafted automation, if not properly maintained, will corrode and rust to the point that it's no longer fit for its purpose—especially when that automation is new. If that happens, you'll have to expend considerable effort to restore usability.
In general, automation decay is due to maintenance surprises for which you did not plan. But you shouldn't have any surprises if you plan for things such as infrastructure changes and parity.
Here are some of the common causes of those surprises, and how you can prevent your automation's value from rusting away.
Support infrastructure changes
From time to time, you need to modify or upgrade your testing infrastructure. These changes may take the form of tool upgrades or changes, operating system upgrades, and cloud migrations. There's really no way to avoid infrastructure changes; as our tools, operating systems, etc. become unsupported or don't support new technology, we must upgrade or move on from them.
You can, however, mitigate the risk that your automation might fall into disrepair by allowing time and effort for these kinds of changes in your delivery plans.
From a software implementation standpoint, you can encapsulate your automation capabilities into layers during your initial automation implementation; these layers can attempt to protect one layer from changes in another.
At one company where I worked, the lowest layer of our stack encapsulated the automation tool being used. Eventually, when we needed to change automation tools, only the encapsulation layer needed to change; layers higher in the stack did not have to change.
Achieve product parity
When was the last time your organization released the exact same code twice? The answer is probably never. Developers always add something, fix something, or, in rare cases, remove something.
In each of these cases, the product changes may necessitate a change in your automation such as adding, changing, or removing a script. If you don't plan for and perform this kind of maintenance, your coverage will shrink over time and your automation may even result in unreliable results that erode trust with your testers.
The key here is knowing that product changes impact automation; once you expect product changes to necessitate changes in your automation, you begin to allow time to deal with it.
Again, encapsulating your automation capabilities into layers helps isolate the needed changes. One of the most common changes is a change of HTML attributes in a web UI; keeping the locators for those elements isolated in a layer, away from the test scripts themselves, means you can change the locator in one place instead of in each test script.
Build trust
If the testers on your team don't trust the automation, they won't use it. How do you build that trust? The main way is through exposure. Testers must be part of the automation implementation. Even if they are not pressing the keys (i.e., programming the scripts), they are an integral part of the endeavor. You need testers to help prioritize what should be automated and to review completed scripts.
The key to trusting the script is the review. If testers know exactly what is happening in a script, they are more likely to trust that script to perform some aspects of their jobs; reviews help them trust that automation is giving them consistent, appropriate, valuable results. Without gaining their trust, you'll have wasted time and money.
How to avoid automation rot
Most automation rot comes from the misplaced expectation that automation can be "finished." But automation is software that you build to support testing activities that in turn support software delivery. If you continue to deliver software as a product, your organization will be changing that software, which means you'll have to change your automation software.
So plan for maintenance and implement your automation such that maintenance doesn't overtake the value that the automation provides.
For more details on how to maintain your automation, come to my session at the Automation Guild online conference, which runs February 4-6, 2019.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
