The last decade has seen a relentless push to deliver software faster. Automated testing has emerged as one of the most important technologies for scaling DevOps, companies are investing enormous time and effort to build end-to-end software delivery pipelines, and containers and their ecosystem are holding up on their early promise.
The combination of delivery pipelines and containers has helped high performers to deliver software faster than ever. That said, many organizations are still struggling to balance speed and quality. Many are stuck trying to make headway with legacy software, large test suites, and brittle pipelines. So where do you go from here?
In the drive to release quickly, end users have become software testers. But they no longer want to be your testers, and companies are taking note. Companies now want to ensure that quality is not compromised in the pursuit of speed.
Testing is one of the top DevOps controls that organizations can leverage to ensure that their customers engage with a delightful brand experience. Others include access control, activity logging, traceability, and disaster recovery. Our company's research over the past year indicates that slow feedback cycles, slow development loops, and developer productivity will remain the top priorities over the next few years.
Quality and access control are preventative controls, while others are reactive. There will be an increasing focus on quality in the future because it prevents customers from having a bad experience. Thus, delivering value fast—or better yet, delivering the right value at the right quality level fast—is the key trend that we will see this year and beyond.
Here are the five key trends to watch.
1. Automating tests
Test automation efforts will continue to accelerate. A surprising number of companies still have manual tests in their delivery pipeline, but you can't deliver fast if you have humans in the critical path of the value chain, slowing things down. (The exception is exploratory testing, where humans are a must.)
Automating manual tests is a long process that requires dedicated engineering time. While many organizations have at least some test automation, there's more that needs to be done. That's why automated testing will remain one of the top trends going forward.
2. Establishing a continuous quality culture
As teams automate tests and adopt DevOps, quality must become part of the DevOps mindset. That means quality will become a shared responsibility of everyone in the organization.
Figure 2. Top performers shift tests around to create new workflows. They shift left for earlier validation and right to speed up delivery. Source: Launchable
Teams will need to become more intentional about where tests land. Should they shift tests left to catch issues much earlier, or should they add more quality controls to the right? On the "shift-right" side of the house, practices such as chaos engineering and canary deployments are becoming essential.
Shifting large test suites left is difficult because you don't want to introduce long delays while running tests in an earlier part of your workflow. Many companies tag some tests from a large suite to run in pre-merge, but the downside is that these tests may or may not be relevant to a specific change set. Predictive test selection (see trend 5 below) provides a compelling solution for running just the relevant tests.
3. Data-driven DevOps
Over the past six to eight years, the industry has focused on connecting various tools by building robust delivery pipelines. Each of those tools generates a heavy exhaust of data, but that data is being used minimally, if at all. We have moved from "craft" or "artisanal" solutions to the "at-scale" stage in the evolution of tools in delivery pipelines.
The next phase is to bring smarts to the tooling. Expect to see an increased emphasis by practitioners on making data-driven decisions.
4. The rise of AI: Test-generation tools
There are two key problems in testing: not enough tests, and too many of them. Test-generation tools take a shot at the first problem.
To create a UI test today, you either must write a lot of code or a tester has to click through the UI manually, which is an incredibly painful and slow process. To relieve this pain, test-generation tools use AI to create and run UI tests on various platforms.
For example, one tool my team explored uses a "trainer" that lets you record actions on a web app to create scriptless tests. While scriptless testing isn’t a new idea, what is new is that this tool "auto-heals" tests in lockstep with the changes to your UI.
Another tool that we explored has AI bots that act like humans. They tap buttons, swipe images, type text, and navigate screens to detect issues. Once they find an issue, they create a ticket in Jira for the developers to take action on.
More testing tools that use AI will gain traction in 2021.
5. Machine learning and predictive test selection
AI has other uses for testing apart from test generation. For organizations struggling with runtimes of large test suites, an emerging technology called predictive test selection is gaining traction.
Many companies have thousands of tests that run all the time. Testing a small change might take hours or even days to get feedback on. While more tests are generally good for quality, it also means that feedback comes more slowly.
To date, companies such as Google and Facebook have developed machine-learning algorithms that process incoming changes and run only the tests that are most likely to fail. This is predictive test selection.
What's amazing about this technology is that you can run between 10% and 20% of your tests to reach 90% confidence that a full run will not fail. This allows you to reduce a five-hour test suite that normally runs post-merge to 30 minutes on pre-merge, running only the tests that are most relevant to the source changes. Another scenario would be to reduce a one-hour run to six minutes.
Expect predictive test selection to become more mainstream in 2021.
The real problem: Test execution time
Automated testing is taking over the world. Even so, many teams are struggling to make the transition. Continuous quality culture will become part of the DevOps mindset. Tools will continue to become smarter. Test-generation tools will help close the gap between manual and automated testing.
But as teams add more tests, they face real problems with test execution time. While more tests help improve quality, they often become a roadblock to productivity. Machine learning will come to the rescue as we roll into 2021.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
