The top 7 test automation mistakes: How to avoid your next fail

In their desire to do more automated testing, QA teams often make mistakes that cost time, money, and trust and derail progress. These blunders can make your team too nervous to try again—despite the fact that QA is now seen as a key driver of business growth, according to the World Quality Report.

The good news is that, in most cases, these gaffes are completely avoidable. Here are the seven most common test automation mistakes your team should avoid.

1. You skipped the first step

It's tempting to jump in and start making lists of things to automate and then hunt for tools that will do the job. But there's a step that should precede that, said Bas Dijkstra, test automation trainer and consultant. "Teams forget to ask themselves first: Why are we going to automate this in the first place?"

Paul Merrill, automated testing consultant at Beaufort Fairmont, suggested starting with the questions, What risk are we trying to mitigate with our test, and how does automation help that?

Solution: Carefully define the goals and expectations for each automation initiative, and remember that every potential area of automation should contribute to "quality at speed" in a measurable way.

[Related story: 50+ resources for test automation engineers ]

2. You automated the wrong things

Many teams try to automate things that aren't good candidates for automation. One big miscalculation in this area is trying to translate your existing testing activities one-to-one into automation. That leads to testers wasting much time and effort on things that don't need to be automated, or that shouldn't be automated.

For example, you might try to automate all of your existing regression tests verbatim. "That's a recipe for disaster, because automation doesn't work very efficiently that way," said Dijkstra.

And if you run a test only, say, once a year, it's not smart to spend months creating a framework and scripts to automate it. "Make sure you’re targeting something that will give you real value right off the bat," said Joe Colantonio, founder of TestTalks.com; don't just automate everything you can.

Teams can get into trouble responding to management goals that promote quantity, such as a mandate that the organization automate 90% of testing, said Colantonio. If managers set up a high-level dashboard showing the automation percentage, teams will automate a lot of the wrong stuff just to pump up that number, he explained.

Solution: The best kinds of tests to automate are those that are repeatable and deterministic. "If there's any kind of randomness involved or the code's always changing, it will not respond well to automation," Colantonio said. For a successful outcome with a measurable ROI, look for tests that you will do over and over again, or things like performance testing that you can do only with an automated tool.

3. The tools you picked can't solve your problems

Many managers want to choose one tool for the entire enterprise, said Merrill. That's a mistake because there are many different problems you want to solve with automation, and one tool can't solve all of them. "The key is finding out what problem you want to solve with a tool," said Dijkstra, "rather than buying a tool and then finding a problem for it."

Merrill tells his clients to look for "the right tools for the right job and the right team." Otherwise, you risk getting tools on board, and then finding that you have to do a lot of workarounds or customizations to get them to do what you really need them to do.

Solution: Define specific problems you need to solve before searching for a tool, and try before you buy. Colantonio advises a two-week proof-of-concept trial with each tool, running it through the full development lifecycle to see how it works in your environment and to make sure it addresses the problem you are trying to solve.

[ Related story: 6 test automation tools developers love ]

4. The tools you selected aren't a good fit for your testers

In the real world, problems arise from "the technical skills (or lack of them) in the people who will work with the tools," said Angie Jones, senior developer advocate at AppliTools. Make sure you know what skill sets your team will need to effectively use any tool you are considering. This should be a part of your request for proposal, such as the one included in the Buyer's Guide to Test Automation Tools.

Vendors design their tools for the "personas" that are most likely to use them, according to Diego Lo Giudice, vice president and principal analyst at Forrester Research. There are three types of tester profiles, he said:

Developer-testers, who prefer to stay in their preferred coding medium for test writing
Technical testers, test automation specialists who prefer higher-level tools that use modeling and don't require coding skills
Business testers, nontechnical people who do acceptance testing, usability testing, etc., and prefer to use natural language for test writing

If you buy a tool that was designed for a developer-tester and ask a technical or business tester to use it, you will run into trouble every time. This is likely to happen more often as organizations look beyond their limited supply of developer-testers and ask other personas to help with test automation.

Solution: Do a trial run with each tool you are considering, and use the actual testers who will need to use it, to uncover skill gaps. Look for tools that both coders and non-coders can use. For example, some functional test automation tools provide both scriptless and script-based editing. Those tools don't require advanced developer skills and will be easier for everyone to use.

Also, make sure you have a budget for upskilling the team when necessary. "Unless organizations take active steps to retrain their employees and develop specialized skills, this could soon emerge as a critical bottleneck holding back the progress of the QA and testing function," according to the World Quality Report 2018-19.

5. You failed to add up the total cost of tool ownership

You'll quickly get into trouble with a tool purchase if you don't consider all of the costs associated with that choice. "I had a client who wanted help moving from a commercial tool to Selenium," said Merrill. When asked about his goals, the client said, "We don’t want to pay the licensing fee anymore." He had no other goals; it was all about saving money.

"Immediately I knew that with the effort that he was going to have to put in, there would be no cost savings for years," said Merrill. They had thousands of scripts already created and would need to spend time training a team of 13 people.

Between the training and support costs, and going back and rewriting all the scripts while handling all the new things coming in every day, it would be years before they would actually save money. The client took Merrill's advice and didn't change tools.

Solution: Consider the associated costs before migrating from one tool to another. Factor in team training and support for the new tool, the costs of migrating existing tests, and the impact the move might have on your ability to hire and keep skilled testers on your team.

6. You chose a tool just because it's open source

Market preferences are definitely shifting in this arena, and open source is now well accepted, according to Forrester's Lo Giudice. But that shouldn't be your primary selection criterion. "The trick is to find out first what you need to solve, and only then look for a tool that takes care of that problem in the most efficient way regardless of how it is licensed," said Dijkstra.

In general, open-source tools require more technical skill. Typically, you won’t be able to get on the phone and call the vendor for support if it's not working, Jones warned.

Open-source tools also come with some limitations, despite their attractive lack of licensing fees. Some have limited functionality and may not give all you the features you need, so be clear about what you require.

"You might just need a little bit of reporting, so an open-source tool will be enough. Or you might be in an organization that is highly regulated, or needs more extensive reporting; that makes it worth the expense of buying a commercial tool," Jones said.

Also, be prepared to switch it up. People tend to go more open source at the beginning, start to innovate, and "at some point they scale and find they are better off with a solid commercial tool that can scale and doesn't need all the support they have to put into it," said Lo Giudice.

Many organizations end up using both open-source and commercial tools together, since the commercial tools can fill in any missing functionality that the open-source tools don't have, and often add value on top of the open-source tools.

Solution: Identify the problem you want to solve and the features you need, and then search for a tool among the commercial and open-source options that meets those needs. Then consider the tradeoffs between open-source and commercial options. For example, given the skill level of your users and their training and support needs, is the open-source option a good fit? Finally, evaluate the vitality of the specific open-source community that's behind the tool, to determine if there will be free community support available.

7. You didn't foster a testing culture that embraces automation

If you don't have a testing culture, tools won't fix it. People tend to blame their tool when things go wrong, said Colantonio. But it's not usually the tool's fault; it's the team, the way they approach testing, and their attitudes about automation.

Colantonio used to work for a large company that used a commercial tool, but its tests were flaky and took a long time to run. "They switched to Selenium, and guess what happened. People started complaining that the tests were slow and hard to maintain, which were the same issues they had with the vendor tool." The real problem, he said, was that they didn't have a culture oriented to testing.

Companies with culture problems may have a bias against test automation, or they may still have only one person assigned as a tester and require that all test activities go through that tester.

Shifting the culture can take a lot of training. It takes one person to take the initiative and that sets the team on fire about the benefits by coaching others and marketing the idea of test automation from the inside, Colantonio said. "It just takes one person to step up and be the test automation champion."

If all team members are doing their thing, but no one actually buys into why they're doing it, someone inside the team needs to point out specific successes. That could include time savings, along with information about how that happened.

It's up to your organization's leaders to shape the culture, said Merrill. They can encourage champions to take the initiative and set up an environment that encourages experimentation and applauds successes.

“Having a champion take over and lead it with passion is absolutely key,” he said. People hear about an experience, and they see something working, the energy behind it, and the benefits from it, and that is a lot easier to get onboard with than some article that they've read.

Solution: Evaluate the team and organizational culture, and work with leadership to promote a great testing culture. Find individuals who will drive it from inside the team and help them be successful.

[ Related story: How I created a culture of quality ]

Your next step: Find a project that can deliver a win

If you have had mixed results—or even epic failures—with test automation in the past, that should not prevent you from trying again. You will never be able to keep up with the speed of agile development without it.

Take a hard look at the mistakes that may have derailed you in the past, and use these recommendations to find a project that can deliver a win. This is how you can lay the foundation of a robust test automation strategy for your organization.

The stakes are getting higher every day. Automation is the biggest bottleneck holding back the evolution of QA and testing today, according to the World Quality Report 2018-19. "Automation, and especially smart test automation, is poised to bring about significant changes in the way QA and testing is done" over the coming two to three years, according to the report. You need a strategy and roadmap in place if you expect to reap its benefits.

Keep learning

Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.

Read more articles about: App Dev & Testing, Testing

You are here