Tools to support and automate parts of software testing activities are important, even essential in some areas, and yet they also can be problematic. I've seen teams struggle with today's testing tools and have wondered why: Is this a problem with the testers' skill sets, or with the usability of the tools themselves?
I've worked in the software industry for decades, and as part of a research project as a PhD student at the University of Malta, I asked testers about their experiences with testing tools and the concerns and challenges they face when implementing and using them. I collected data through surveys, workshops, and interviews with over 100 testers from across the globe, amassing an enormous number of stories about their experiences, good and bad, with testing tools and automation.
As it turns out, testers say usability issues are affecting success with testing tools. Here's what else I learned—a sneak peek before my in-depth presentation at the upcoming EuroSTAR conference.
Ease of implementation, use are key problems
As a software tester, the information that you supply to decision makers and your colleagues must be useful, timely, and accurate. Delivering software on time that then fails is not a good outcome, but neither is delivering software late. The use of tools to support testing makes sense, but while tools that aid in testing have been around for decades, so have problems with their acquisition and use.
In the 1980s, I worked on a project where we used tools to execute tests. The tools allowed us to deliver on time and with enhanced certainty about software quality. The software, tests, and tools grew together. But over the decades, I noticed that successful tool use is unusual. While more tools are available today, testers complain that their tools have not been easy to implement and use over time.
As the IT industry has grown and software has become ubiquitous via the Internet and mobile technologies, the work of testers has become more pressured. We're doing more, with a requirement to deliver more quickly. Tools to support all aspects of testing have become even more important, in terms of both the scope and the speed of the testing. However, implementing tools and using them successfully remains problematic.
Issues include lack of management support, misconceptions and over-optimism about what tools can achieve, and technical problems with tools. These problems are well documented, but still ubiquitous. Additionally, there's a push to change testers' skill sets to include coding, in order to support scripting in test tools.
Most users experience issues with their testing tools
The first result is rather depressing. In all the time our industry has been attempting to adopt and improve tool support—for all the presentations, tutorials, webinars, books, and discussions we've shared—we are still making the same old mistakes.
Some 82 of the 111 participants surveyed raised at least one issue with their tools. The table below shows the percentage of comments shared about three main topic groupings. Management decision making continues to be important, with challenges, blockers, and enablers identified in recruitment errors, lack of training support, unreasonable expectations, and poorly defined targets.
Technical blockers abound, including installation problems, test and tool maintenance problems, IT security preventing progress, and tools that are mutually incompatible, blocking a stream of test activity.
Usability is indeed an issue: Testers said they want greater usability to be built into the design of their tools, and 37% of all issues raised were usability-related.
From tool frustration to anger: Testers expressed a range of emotions
A second finding was that testers had very emotional responses to many of the survey questions, and this was directly related to their experiences with tools and automation. Testers said they feel "stuck" because "tools are promoted as 'magical solutions'" and also they feel "scared" and "frustrated" but also "quietly proud" of what they do.
Two participants reported that they were required to use a particular test tool, but because of organizational security policies and the technical configuration of the test environment, they were unable to gain access to the tool, leading to unnecessary frustration, anger, and demotivation. Two people used the word magical to describe management expectations of tool implementation—reduced team sizes, costs, and time scales—while other testers spoke of fears and feelings of isolation and worthlessness.
Usability is a focus for tool designers and vendors, but even so, testers reported problems. Usability issues are not a matter of complete neglect, but in some cases the result of a superficial approach to usability. Here are five key issues:
- Vendors focus on the interface to emphasize attractiveness over usefulness. This resulted in an interface that "looks cool" but doesn't support the testers' workflow. True usability focuses on goal achievement and the support of a seamless workflow experience.
- Some testers reported that their tools were too hard to use. Others reported insufficient flexibility. Usability should consider the range of tool users and their specific goals, needs, and aptitudes.
- Others said their tools did not support their changing needs. Usability should provide task flows for novice and expert users, supporting movement between those levels.
- Testers, software under test, the test environment, and the tests themselves all change over time. Given that, usability is necessary but not sufficient, maintainability and portability of the tests and tools are also important.
- Other attributes such as security, performance, and reliability must also be supported.
Taken together, this means that when you acquire tools you can fall into an "illusion of usability." This is where you think a tool will be useful and easy to use and will solve your testing problems, only to be disappointed when it does not help you complete your work because of one or more of these five issues.
Get more from the tools you have
Usability is a vital aspect of a testing tool's success, but it's not trivial to achieve during tool design. Testers come from a wide range of backgrounds and experience, so it's not surprising that the tools are not a good fit for everyone.
But how can this be resolved? To help your own tool implementation be more successful, use already-published materials such as the Test Automation Patterns website to help you overcome management and technical blockers to success.
Want to know more? Pick up a copy of D. Graham and M. Fewster's "Experiences of Test Automation: Case Studies of Software Test Automation" or K. Wiklund', “Impediments for automated software test execution: A systematic literature review."
And don't miss my EuroSTAR conference session, "Who Are We? What Are We Doing? How Are We Doing It?" where I'll talk more about my results and my research, what it means for testers, and how you can contact me to participate in the next phase about what can be done to improve tool usability design. The conference runs from September 28 to September 30, 2021. TechBeacon readers can save 10% on individual tickets using code "TechBeacon10" until Sept 6.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
