For its annual World Quality Report, Capgemini and Sogeti interviewed thousands of technologists and technology leaders to learn the state of software quality practice. The World Quality Report 2021-22, sponsored by Micro Focus, is Capgemini and Sogeti's 13th such effort, giving the authors over a decade of data to show not just the state of software quality, but how trends—and the culture of quality—have been changing.
At 72 pages, the report is very thorough, and what this year's report reveals might surprise you. You'll want to dive in for all the details, but for a topline summary here are eight key takeaways from the newest World Quality Report—everything you need to know, along with comments from the authors, and other experts, who shared their views of the results.
1. Testing and quality are playing a bigger role
The first figure in the report is about the objectives of QA: Find problems early, go faster, and contribute to business growth or customer satisfaction. The percentages vary only from 58% to 62% and haven't changed much in the past four years.
Figure 1: The objectives of quality assurance and testing. Source: Capgemini/Sogeti
Put differently, nearly everyone wants everything. As a consultant, I would be tempted to read this as immaturity, but the authors of the report see it differently: that management is finally seeing QA as about more than finding defects. The slight drop in the role of QA as the "custodian of quality" could be because testing has been pushed beyond a particular person, onto everyone on the software delivery team.
Andrew Fullen, head of innovation and technology for Sogeti in the United Kingdom, said that means a wider role for, and perception of, quality.
"The industry feels like it's grown up now, it's matured, it's looking at a bigger picture. It's discarding the things that are not important, moving into a world that is more interconnected with more tools to support it. Just because all these things are important doesn't mean every project will do all these things in the same ratio."
—Andrew Fullen
2. The rise of AI in quality is extending what testers can accomplish
For years, industry leaders and consultants have been complaining about the hype around AI. But the report found that AI is offloading some cognitive tasks from humans to the computer. "These findings show the need for organizations to advance their modernization initiatives through AI-powered continuous quality and test automation tools," said Rohit De Souza, Senior Vice President and General Manager of Micro Focus' ITOM and ADM product groups and Leader of the CTO Office and Product Security.
"There’s a couple of big things when it comes to AI powered testing,” said Micro Focus chief technologist Don Jackson. “The first, and probably the heaviest, is the leveraging of computer vision—for the computer to see things the same as a human does.” In the past, testers had to use the properties exposed by the operating system, and every time they crossed a platform, those properties would change for the same object. You could have as many as four different scripts for the same business process, depending on what platform you were running on. "With computer vision, it runs on the visual characteristics of the objects, just as a human does,” he said.
Codeless tools, for example, can use natural-language processing (NLP), allowing a nontechnical human to create and update tests. Computer vision can recognize a shopping cart image, so if the icon changes or moves, the software can still find the button to click.
The combination means that tests do not need to use precise locator strings or classic code (Java, C#, Ruby), making them less brittle. Log query and analysis tools make it possible to perform risk-based testing with usage data. None of these are strategies to replace humans. They are, instead, approaches that augment human performance.
3. Testing has room to improve—and is actually improving
The next figure rates the success in accomplishing the objectives outlined above. Only half of the respondents claimed that testing was optimized for speed and cost. That indicates that there's a fair amount of room for improvement. Yet the same response last year racked up only 37%. That is a significant improvement in just a year, and it might be due to how organizations responded to COVID-19.
Figure 2: The QA group's success in achieving specific objectives. Source: Capgemini/Sogeti
4. COVID-19 response has been integrated into the world
Last year's report had an entire section on how COVID-19 had affected software quality. For example, financial services and manufacturing had been slow to embrace cloud computing, making the shift to remote work more difficult. The pandemic-induced pressures of time, resources, and availability made companies more open to out-of-the-box approaches—not just working remotely, but also crowd-sourced testing.
As more employees had to do testing remotely, there was also a demand for data-masking services, to transform and anonymize production data. A year later, the issue of Covid had "Faded into the woodwork" of software development, and most companies have integrated those changes into the way they do software delivery, said Jackson. What once were accommodations, such as working remotely, have become part of day-to-day work. So, instead of a separate section for pandemic-related responses, the answers to COVID-related questions were integrated into other aspects of the report.
Over the past year, many Silicon Valley companies moved to remote-always for testers. This is a considerable shift, letting skilled workers all over the United States compete to work at Apple, Google, Facebook, and other now-remote-first companies.
That will reduce supply for local test- and quality-related jobs on the high end, especially jobs where the employees need to come into the office. Companies that do not allow remote work, or that insist on a back-to-the-office strategy, will face challenges in recruiting, retention, and morale in the year to come. Many employees are now used to working from home and prefer it, or have relocated to areas where the cost of living is lower or quality of life is higher.
5. The lines have blurred between testing, quality, agile, DevOps, and digital transformation
It is striking how much overlap these topics had in previous World Quality Reports and how those have converged. Instead of an "Agile Testing" survey, a "Digital Transformation" survey, a "Testing in DevOps" survey, and so on, this year the report combined them all into one.
While the most recent WQR did have a section for digital transformation, it is becoming clear that agile, DevOps, and quality are all prerequisites, or perhaps the mechanism by which digital transformation is possible.
6. Test tooling and automation are becoming the norm
Of all the priorities for improvement to work on, test automation rated the lowest, with only 46% seeing it as a priority. The temptation is to read that as "not important," but again, it may mean that companies are testing successfully. In those cases, test tooling has moved from a strategic initiative to a part of the day-to-day. As Sogeti's Fullen put it, "I'd much rather have a small percentage of good-quality automation that makes a difference than automating everything."
Shivakurma Balasubramaniyan, vice president of financial services, data assurance and quality for Capgemini—and another co-author of the report—adds that companies are finally starting to understand they need to hire for and develop software development skills in test.
That means that when they recognize a gap, they do not have an "automation initiative." Instead, they call it hiring, a job description, restructuring, or training.
Figure 3: The importance of different factors in making testing more efficient. Source: Capgemini/Sogeti
7. There's a growing recognition that testing is actually a skill set
Balasubramaniyan's comment leads to the most popular success factor on the team—having the right skills. That goes beyond automation, to AI, DevOps, continuous integration, and other aspects, but most importantly for actual testing and test design.
Kirthy Chennian, vice president for North America at Capgemini and a report co-author, pointed out that the biggest challenge respondents faced was in obtaining sufficient agile testing skills on software teams. At first that might seem odd, considering the number of years since many respondents said they completed an agile shift.
Chennian said that is because the person doing the work has moved from a specialized role to everyone participating in software delivery. After a few years of assuming testing was easy and anyone could do it, organizations are realizing that it is a repeatable skill that can be developed, trained, and hired for.
8. The use of Docker/Kubernetes for testing and test environments is accelerating
As companies perform testing with tools, they need servers to run against the latest build. Just over half the companies surveyed have on-demand, fit-for-purpose test environments, and the de facto standard for that seems to be the Docker image, likely running in a Kubernetes cluster. That provides a relatively straightforward testing path for companies, especially considering that every major cloud vendor supports Kubernetes.
What's next in QA
Over half of the companies surveyed have on-demand test environments, which means that nearly half do not. The real questions are what matters to your organization, and where your organization stands relative to its competitors. If you don't have an on-demand test environment, the new World Quality Report could be a call-to-action.
These key takeaways are top line and don't get into individual industries such as automotive, energy, financial services, healthcare, government, high tech, or telecom. But the report's more detailed breakdowns dedicate 20 pages to sector-specific data. Download the full World Quality Report 2021-22 to take a deep dive.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
