Artificial intelligence (AI) and machine learning (ML) allow companies to close the testing gaps and are best applied when they augment people’s expertise and capabilities to process real-time data.
For digital transformations, tools that leverage ML can help quality assurance (QA) test for issues that are difficult to perform manually or with automated testing, according to the "Benefits of Automation & AI in Functional Testing" report, written by Isaac Sacolick, founder and president of StarCIO. Two hundred CIOs, CISOs, and IT leaders in development, QA, and operations from large enterprises were surveyed on their testing strategies, practices, and challenges.
"The surprising thing to me [from the survey] was how many people were using and seeing the benefits of AI. It was pretty high, a lot higher than I expected."
—Isaac Sacolick
Here are the key benefits of AI for functional testing.
1. AI is good for business
Almost all of the 200 IT leaders surveyed said they're experimenting with AI capabilities in testing (only 1% said they had no plans to use AI in QA), and nearly half of them (49%) are seeing the business benefits of using AI in QA, Sacolick said.
Areas where ML has the greatest potential include detecting anomalies, using computer vision to spot user interface changes, and leveraging natural-language processing in test creation, according to the report.
Given where AI is in the other areas of technology and other areas of the business, Sacolick found that number pretty high. "If you asked me ahead, I would have guessed maybe 15%, 20%. And also for the fact that AI is still an under-invested area in the software development lifecycle. So for people using more advanced capabilities, I found that surprising."
One of the reasons more organizations are using AI in testing is that they realize that developers don't have enough time and skill to test robustly by hand. Consequently, they look for tools to be able to do it, Sacolick said. And the tools have come a long way, particularly over the last five years.
"The real question is as you make changes to the application, to the code, how easy is it to keep the tests updated? And that's the part where the AI comes in."
—Isaac Sacolick
Using AI can drastically help reduce the maintenance required for scripts and applications, said Chris Trimper, test automation architect at Independent Health, in an interview. "Once testing teams adjust to a new paradigm, there’s a significant impact on the testing’s robustness and supportability," Trimper said in the report.
With new AI-based testing capabilities, Trimper said, his company was able to use a single set of multi-platform scripts across iOS and Android and reduce mobile test maintenance by close to 45%.
2. Platform coverage is a barrier to automation
Survey respondents also noted that testing multiple browsers, devices, and operating systems is a testing pain point—and lack of platform coverage is a significant barrier to test automation.
Twenty-two percent of respondents said they can write a single test case that can run on any platform without modification, while 77% of respondents either write different tests or have to customize tests for each platform.
Donald Jackson, chief technologist at Micro Focus, a sponsor of the report, said that what really surprised him was that so many customers said that they could today write a single test that runs on any platform without modification.
"I think one of the reasons [for that] is that some of the survey respondents probably didn't understand what we meant by 'platform'."
—Donald Jackson
Segmenting this by job level, there's a wide discrepancy between the CXO level and the teams that actually understand what's going on, likely because the CXOs aren't really plugged into exactly what the developers are doing, Jackson said.
"When we talked about 'platform' in this particular survey, we're talking about the ability to run on a Mac, macOS, iOS, Android, and Windows platform with a single script," he said. "When you look at the CXO response, 38% say, 'We can write a single test that can run on any platform without modification.' Then when you talk to the guys who actually understand it, the developers, QA, and IT Ops, only 19% say that."
3. AI lowers the barrier to automation
Another thing AI can do is lower the barrier for those who want to do automation, for two reasons, Trimper said. "One is it's a bit simpler to do and it doesn't require as much domain experience as what would be traditional automation. What this means is you can get more testers involved in automation."
In addition, developers can also get involved in automation, he said. They understand the product and they know what they coded, but they don't necessarily need to know how to automate, because the barrier to automation has been lowered immensely.
"And [AI] makes it so that they theoretically could automate on day one and not need an immense amount of training and not have to understand two different domain areas. They don't need to be a test automator and a developer. They just need to be the developer and they can [do test automation]."
—Chris Trimper
Automating more testing during the development process and increasing the scope of testing to include performance, security, functional, and application programming interface test cases is a shift-left testing strategy, according to the report:
"This investment enables teams to improve customer satisfaction, reduce defects in production, and enhance applications with less fear of breaking things. Leveraging machine learning and AI in testing enables teams to identify more complex quality issues."
How to realize the full potential of AI
But teams that just use AI to augment existing test automation won’t see the full potential of AI, said Thad Parker, founder and CEO of Proof'd, a software-as-a-service automated testing platform for web and cloud applications.
"Adopting an AI-first approach better meets the needs of both testing teams and leadership by dramatically reducing the time and cost of achieving the desired test coverage. Unlike AI-augmented tools, AI-first testing tools . . . remove the burden of maintaining different tests for different browsers and operating systems. This allows QA teams to focus on quality rather than tests."
—Thad Parker
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
