Organizations on the Forbes Global 2000 list are increasingly beset by nimble startups intent on disrupting their industry. Accelerating application delivery is a key part of maintaining a competitive advantage, but it's hard for enterprise organizations to keep pace.
All too often, it's testing that holds back the speed of application delivery—specifically, a lack of a mature continuous testing practice that provides near-real-time feedback on whether the application has an acceptable level of business risk. At times, it can seem as if you're competing in the Tour de France while towing a child in a bike trailer.
Nobody doubts that continuous testing is essential. But making it a reality is a challenge in any development environment, and it's an especially pointed pain in enterprises, for a number of reasons. These include complex application stacks, deeply entrenched processes built for dramatically different delivery models, and strict compliance requirements.
Here are three challenges enterprises commonly face when trying to implement an effective continuous testing practice—and how top organizations have solved them.
1. Introduce and expand test automation where it has been most difficult
Having a core set of automated tests that expose business risks as they're introduced is a key component of continuous testing. Yet the latest World Quality Report found that test automation has been stalled at around a dismally low 15% for years now.
You can't—and shouldn't—automate all your testing efforts. But if you want fast feedback on whether the latest changes broke any critical business processes, ramping up automation is essential.
The problem is that getting started can be hard, and keeping it going is, unfortunately, even harder. Most large organizations have at least one or two failed test automation initiatives under their belt, so quality leaders are understandably reluctant to spearhead another.
Yet many brave QA leaders are up for the challenge, and many of those have achieved success. What's their secret? At a high level, they remain laser-focused on the long-term goal of sustainable test automation and architect the deep-seated process and cultural change required to support that.
Getting it started requires finding a test automation approach that crosses the various technologies involved in your business processes (technology-agnostic or broad technology support). You also need an approach that can be adopted rapidly and broadly by your existing team members, whatever their current skill set may be.
Keeping it going also requires addressing what I call “the three nightmares" of test automation: test maintenance, test data, and test environments.
Here are some tips from how Linde, the world's largest gas company, introduced test automation into its complex network of highly customized SAP, Salesforce, web, and mobile apps:
- Think about where automation will have the greatest impact, then focus on creating stable automation for those use cases. You can gain much more attention and internal support by automating a handful of high-impact, end-to-end processes than you can by automating thousands of ill-conceived tests.
- Root out the "more-tests" mindset. For years, many organizations have been using the number of tests to measure the extent of testing and even to compensate testing providers. More tests that address the same business risks don't help. In fact, they hurt. They consume resources to create as well as resources to maintain—eating into your test automation ROI.
- Sometimes you need to cut your losses. If an approach or tool really isn't working for your existing resources—e.g., it's too technical or doesn't suit your ecosystem—understand what's needed instead, and resolve to address it going forward.
2. Assess 'release readiness' of new functionality across teams and applications
A primary goal of continuous testing is to determine if a release candidate is ready for production. As described above, you absolutely need to ensure that the changes in each release don't break existing functionality. But you also need to test the new functionality to ensure that it works and meets expectations.
Making the ultimate go/no-go release decision can be a bit of a guessing game when different teams are responsible for different components and layers of the application: the browser interface, the mobile experience, the various packaged apps at work behind the scenes, and all the microservices, APIs, and integration platforms that are probably gluing it all together.
Teams are likely developing new functionality at different cadences, and testing "their" parts in different ways, using different testing practices and different tools. But the user doesn't make those distinctions. They expect it all to just work … flawlessly.
Moët Hennessy-Louis Vuitton (LVMH), the parent company behind luxury brands such as Christian Dior, TAG Heuer, and Dom Perignon, recently decided to streamline its testing process to support ambitious plans for e-commerce growth. The company perfected the art of testing new functionality efficiently and making release decisions that ensure the ultimate user experience, in part by doing the following:
- Take a structured approach to testing new functionality. To better defend both the customer and the brand, each QA team took a deep dive into the DNA of the brand and its core customer profiles (personas). The teams then developed new test strategies for checking how the customer experiences the brand and captured them as reusable "rollout kits."
- Reuse, reuse, reuse. The teams built a library of strategically designed test building blocks that allow 70% to 90% reuse across their launches. These building blocks map to an array of test automation tools that they combine to view, interact with, and evaluate the site like a human—on a broad variety of devices. With this head start on automation, they can start testing early in each release cycle and gain fast feedback as they implement and refine new user stories.
- Connect the dots. All the disparate testing data for web, mobile, Salesforce eCommerce, enterprise resource planning, customer relationship management, etc. is integrated into a single pane of glass. This provides real-time insight into release readiness by persona, feature, and tech layer. Team members can always tell, at a glance, what's ready to release, what's holding the release back, and who's responsible for getting it on track.
3. Balance collaboration with autonomy
For years, the trend has been toward autonomous, self-governing development teams that choose the practices and tools best suited for their culture and projects. This is great for team motivation, but it's not ideal for collaboration. At the same time, another competing trend—toward highly interconnected applications—has made collaboration even more critical.
At the points of intersection, there's a huge opportunity to share test artifacts (as well as code and deployment artifacts). However, that's easier said than done, now that so many teams have grown accustomed to using their own processes and tools.
It is possible to promote cross-team continuous testing collaboration without homogenizing all the distinct ways of working that different teams have invested considerable resources into developing and perfecting. Dell has devised an effective strategy to tap synergies across 30 divisions working across more than 20 test automation frameworks. Here's what the company did:
- Abstraction is key. Dell analyzed the elements of DevOps success from a high-level philosophical perspective, then crafted an abstract continuous integration/continuous delivery/continuous testing architecture. This specified what activities to address (source control, requirements management, test management, etc.) without prescribing low-level implementation details on how to complete them.
- Build an overarching layer. For continuous testing, Dell established an overarching layer that allows team members to find and run relevant test assets without even thinking about which test automation framework is used behind the scenes. This way everyone can share, but no team has to compromise.
- Standardize on one test framework. For new projects, the company plans to standardize on a test framework that's flexible enough to cover its various application stacks and delivery cadences. Dell will make it available to existing teams and projects, but nobody is being forced to change.
Focus on the right approach for your challenges
There’s no such thing as enterprise continuous testing in a box. As these examples show, the top challenges and potential solutions vary widely, and yours will too.
To succeed, you need to get real. Take a long, hard look at what you're working with, and craft an approach accordingly. Don't expect longtime manual testers to download some open-source testing tools and automate processes that span packaged applications, custom applications, mainframes, and more.
Likewise, don't expect a high-performing, tech-savvy team to give up the homegrown or highly customized approach they poured significant time and resources into over the years.
Continuous testing itself is all about assessing the impact of change. Your approach to continuous testing must also be built around change. You need to determine what you can do from a technology, process, and change management perspective to ease the transition from your current state. Your goal is to craft an enterprise continuous testing process that achieves your organization's goals with respect to quality, speed, and efficiency.
For a more in-depth discussion of these continuous testing challenges, read my book, Enterprise Continuous Testing: Transforming Testing for Agile and DevOps.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
