Automation is cool. You simply choose the best tool, create test scripts for your smoke test suite and regression test suite, and click the "run" button. Then you get to sit back and watch stuff zip around the screen. The benefit: Testers get to spend time doing exploratory testing or partake in other valuable testing pursuits because their routine testing now takes minutes, right?
In many cases, this is true, at least to some degree. Teams can attain results that are similar to the above, admittedly cavalier, description. However, these kinds of results can be realized only if you focus your automation efforts on the appropriate activities.
Automation must provide value, but to determine what is valuable, you must look at automation through a business lens. Unfortunately, most teams and organizations miss the business aspect of automation.
Here's why you should treat your automation like a business for better success.
Understand the costs
Why must the decisions on "when, where, and how" regarding automation be made as a business decision? Like other aspects of software delivery, creating automation has a cost, both monetary and non-monetary; note that non-monetary costs also usually result in a monetary cost.
In many cases, teams don't know that they need to talk about automation in terms of business value. Automation is often considered simply something you do to test and release faster, but it is not that simple.
Even teams that understand that there is a business impact caused by automation don't always know how to have conversations about these business aspects. With the business aspects of automation, two important topics must be considered: opportunity cost and total cost of ownership.
Opportunity cost
Opportunity cost is the cost of not performing Activity B because you are doing Activity A. In a general product sense, each feature that is implemented and each bug that is fixed pre-empts working on some other features or bugs.
There is a cost associated with each deferred feature and bug—such as delayed or lost revenue, delayed or lost cost reduction, and additional penalties due to feature or application unavailability.
To put it in an automation-specific context, by spending time to create a script that performs an end-to-end GUI interaction, you are not spending time creating one or more other scripts. For some teams or organizations, that's the right decision; for others, it is not. Without understanding the opportunity cost, you won't really know if it is a good decision for you.
Total cost of ownership
Loosely related to opportunity cost is the concept of the total cost of ownership. The total cost of ownership is the sum of the direct costs plus the indirect costs of an activity.
Typically, direct costs are the easy ones to identify—developer hours, hardware, up-front license costs, etc. The indirect costs are often less obvious and less concrete: cost of software maintenance, cost of environment upkeep and upgrades, ongoing license costs, etc.
When examining an automation endeavor, you will have two flavors of the total cost of ownership. The first is the total cost of ownership for the automation itself. Following the definition and description above, this is the sum of the following costs: automation framework development, software licenses, automation execution environment setup, and upkeep and maintenance.
The second flavor is the total cost of ownership of the feature or application that is being developed. Again, following the preceding definition and description, the same kinds of costs (development, testing, infrastructure, maintenance, etc.) comprise the total cost of ownership.
You must, however, also include the total cost of ownership of the automation endeavor itself. Yes, automation is a cost; sadly, there is a dearth of magic associated with software development and delivery.
Since costs are context-specific, some teams will not have all the costs listed above, and some teams will have costs not listed. But having a realistic, near-accurate cost of your automation demonstrates that you've prepared to have the business discussion and you are aware of the cost you are asking to incur.
The good news is that by discussing the total cost of ownership of the application, you can better help your organization to understand how the automation impacts that cost.
A way to control costs
The good news is that by having the total-cost-of-ownership discussion, your organization is better poised to make decisions that help keep that cost as low as you responsibly can. For example, you will often find that "older" code, i.e., code that was written prior to automation being a consideration, will not have the appropriate hooks in it that allow it to be automated in a valuable way.
A classic example of missing hooks is not having unique, unchanging identifiers on HTML elements. Without these identifiers, the resulting automation usually has to be based heavily on the structure of the HTML.
This results in automation that is more brittle; brittle automation has a higher maintenance cost because it's more sensitive to changes. This means that changes in the application will have a higher chance of causing automation rework, thereby increasing the total cost of ownership for the automation and for the application.
Retrofitting the existing application with appropriate identifiers will certainly raise the development cost, but it will also decrease the automation cost. The delta between those costs is case-dependent and must be examined by each team or organization to decide on the most valuable approach.
Tally up maintenance also
You must also account for maintenance because it's a fact that there will be maintenance. Make no mistake, automation is a software development activity and it must be treated as such. One part of treating it as such is accounting for and performing maintenance on that software.
That maintenance comes in many forms, including bug fixes, work to maintain application parity, infrastructure upgrades, results audits, and test script retirement. For additional details on some of these items, see this article on preventing your automation from getting rusty.
All work performed in the process of delivering software impacts a budget, be it a team budget, an organizational budget, or the corporate budget.
With that in mind, understand that both opportunity cost and total cost of ownership have budget implications. If you spend money developing an automated test suite, that money comes from your budget. You now have less money to spend on some other software delivery activity.
This is an example of an opportunity cost, because the other activities that you've chosen not to spend money on, at least for now, would also have had a positive business impact.
An example of the total-cost-of-ownership impact is seen when choosing a development technology. If the team chooses a development technology or framework that is not supported by the current automation technology, there will be an increased total cost of ownership for the automation endeavor and, likely, to the application overall.
Choices, choices ...
Deciding what to automate and what not to, when to automate, and how to automate are all business decisions.
Yes, the technical aspects of software delivery must be factored into automation decisions as well, but not at the expense of the business aspects; in fact, the technological aspects will have costs of their own to consider.
The key here is value: Is expending effort on a specific automation activity the right thing to do, and is this it the right time to do it? Answering these questions will help you quantify the value. Calculating the opportunity cost and the total cost of ownership of both the automation and the application itself are essential parts to answering those questions.
For more on building the business case for test automation, come to my presentation at the Automation Guild online conference. The event takes place February 3-4, 2020.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
