AI in testing: 13 essential resources for QA pros
What if you could make software testing simple? What if it could be done without all the conversations, questions, defect reports, and metrics?
We've been promised artificial intelligence (AI) as the solution to all problems related to testing, especially by those who have never tested—those who believe that what we do as testers is little more than tapping screens to make comparisons.
Although I've stated that AI is coming and will change software testing forever (eventually), we're not there yet—not even close. But that doesn't mean we can't use AI to support our testing efforts.
Finding tools that meet your needs, and help with getting accurate information about systems under test, is key. Sharing all this clearly with those who can make decisions is fundamental to our maturity as a profession.
It's time to use AI in your testing. Here are 13 resources for learning about, understanding, and using it effectively.
What is AI?
Before using any AI in your testing (or testing apps with AI built in), it's helpful to learn the basics. Here are several approachable resources for learning about AI.
What is AI—For People in a Hurry
He had me at "in a hurry"! This five-and-a-half-minute video by independent consultant Raj Ramesh gives one of the best descriptions of AI as a discipline within computer science that I've heard. What's better is he does it with simple visualizations, so the ideas stick. Ramesh clearly describes, defines, and organizes machine learning, neural networks, natural-language processing, and much more.
You don't have to be a mathematician or computer scientist to understand the field and disciplines within it.
Explain Machine Learning to Me Like I'm Five
If you want a clear understanding of machine learning, this video by Angie Jones, senior developer advocate at Applitools, is helpful. Not only does Jones share what machine learning is; she also points out how genuinely stupid it can be!
If you're using AI in your testing, make sure to watch this. Just because you're using AI doesn't mean it's doing what you think it is, or even what you want.
A Gentle Introduction to Neural Networks
Data scientist David Fumo provides a deep dive into neural networks and a tutorial to create your first neural network in Python.
Buckle up—this one is intense.
Using AI for testing
With the basics covered, here are some resources for applying AI specifically to testing.
How AI Is Transforming Software Testing
Tech-career coach and former developer advocate Raj Subramanian spoke about AI and its impact on testing at the Selenium Conference in Chicago in 2018. This presentation will help you see how AI is influencing test automation, especially the use of Selenium and where Subramanian believes AI is headed in the future.
Machine Learning and How It Affects Testers
In 2016, I attended a conference where five executives on a panel all said that AI would take over testing. I wanted to understand if they were correct, so I did the research and put together this talk, sharing it at several conferences in the US and Canada during 2017 and later.
How Can We Trust AI Testing Bots?
The cornerstone of testing is trust. For testing to be of value, we have to trust the output from testing. So how do we trust AI testing?
Tariq King, chief scientist at test.ai, addresses this question at Automation Guild in 2019 with his "Test Machina" talk.
Experiencing AI
As already mentioned, to use AI effectively in testing, we have to get familiar with it. Here are some apps and tools that have nothing to do with testing; people can use these to get a sense of what AI does.
Quick, Draw!
Okay, this one is fun.
In Subramanian's presentation described above, he uses this Google tool. In it, you're challenged to draw an object in 20 seconds. During that time, the AI attempts to guess what the drawing is.
One great way to get familiar with something is play, and this tool is fun for that.
Teachable Machine
Teachable Machine is another experiment from Google. The site allows you to try out training models for machine learning. Classify sample videos, photos, or sounds to train the machine-learning algorithms. Then try it out and see how it works.
Of course I had to try this! The site allows using your own video cam to record images. I created images holding up an unsolved Rubik's Cube, a partially solved cube, and a solved cube, all from different angles. Then I classified the images into two groups—solved and unsolved.
It did a remarkable job of classifying new images of the cube in new configurations. Of course, as a tester, I couldn't help but try to confuse it, and I did.
Teachable Machine is excellent for quickly setting up machine learning and doing experiments with it. Learn how AI like such as could be working for you or against you. If you're using AI-enabled testing now or are considering it, this would be a great way to get familiar with the ups and downs of ML.
ML is the most boundless area for exploratory testing I've found.
Know your tools.
WEKA
WEKA is the Waikato Environment for Knowledge Analysis, built and maintained by New Zealand's University of Waikato. Use it to explore data with machine learning. You can use different ML algorithms, configure them, and train them on datasets.
WEKA was the first tool I used to begin understanding ML. While the user interface has room for improvement, don't judge a book by its cover. The power of the tool far outweighs its looks.
Here's a better description from the WEKA site:
The WEKA workbench is a collection of machine learning algorithms and data preprocessing tools that includes virtually all the algorithms described in our book. It is designed so that you can quickly try out existing methods on new datasets in flexible ways. It provides extensive support for the whole process of experimental data mining, including preparing the input data, evaluating learning schemes statistically, and visualizing the input data and the result of learning.
Meetups (or rather, meetup)
I was blown away to find only one meetup on AI in testing. Maybe it's time to start one in your town?
AI in Testing and Testing AI Meetup — Santa Clara, California
Meetups are an awesome way to learn, and because of the pandemic, most meetups are now virtual, meaning you can join them from anywhere.
The only AI in testing meetup I found is the one hosted in Santa Clara, California. Last month, they talked about "Creating a Deployable Jira Bug Classifier using Tensorflow." I can't wait to see what's next for this group!
AI tools
Most AI tools are commercial, for a couple reasons. The amount of effort teams put into building AI tools for testing is huge, and the payoff for customers could outweigh the price many times over. Kevin Surace, CTO of Appvance, explained that his company's AI tool consists of about 4 million lines of code and took approximately 250,000 engineering-hours to develop.
I believe there will be more AI-based, test-assist tools in the open-source market over the next few years. But for now, here are a few tools that may help.
Test.AI Classifiers
Test.ai classifiers use machine learning to match elements on a webpage. These classifiers are available in a number of different languages.
TensorFlow
Want to use a machine-learning API for a testing idea of your own? TensorFlow is one of the go-to APIs out there for quickly implementing ML models.
TensorFlow makes it easy for beginners and experts to create machine learning models. See the sections below to get started.
AGENT
AGENT is an acronym for "AI Generation and Exploration in Test." Find this bot generator for testing your sites in Tariq King's repos on GitHub.
AGENT, using training data from AGENT-X, autonomously learns to explore a website and evaluate its actions, fields, and forms. AGENT deploys one or more exploration and testing agents to explore a web application and apply test flows as testable patterns are recognized.
Enjoy these resources and make sure to let me know in the comments field, below, which AI tools you're using for testing.