Imagine you work for a company that has hundreds of developers pushing new code to production at any time, on any given day. The developers get there because every person on the dev team has a role in quality and testing.
This sounds like an impossibility. After all, making a shift from releasing quarterly to every two weeks is hard enough. But it's not impossible; other organizations—perhaps even your competitors—are already doing this.
Continuous testing is a core safety net in this process. But what does it mean, exactly, when someone says they're doing continuous testing?
You need a concise definition of what continuous testing is and its connections to modern software development. The first step: Define the common vocabulary around the term.
Continuous testing: The basics
Continuous testing is the culmination of many modern, radically collaborative development methods. It is what a development team ends up with when its practices include things such as:
- Test-driven development (TDD)
- Extreme Programming principles
- Collaborative task grooming with testers and developers
- Product manager-led acceptance testing
But to be doing continuously testing, your team needs to hook its tests into continuous integration (CI) servers and build pipelines that run the artifacts through an infinite loop that includes the following steps:
- Check out the code
- Build
- Run the tests
- Report failures
- Repeat
Compare this to the typical development flow, which is characterized by hand-offs.
- A product manager creates some user stories, then hands them to a developer.
- A developer writes some code, maybe writes some unit tests, and then hands that to a tester.
- The tester investigates and returns information to the developer on what needs to be fixed.
The swim lanes you see in a Kanban board are a perfect illustration of the start-and-stop pace most development groups use.
Breaking role boundaries
The closer a team gets to continuous testing, the more the lines between roles begin to blur. Developers collaborate with product managers on stories to refine the ideas and better understand what they need to develop. When they write code, they will sometimes pair with another developer or a tester.
A feature is “done” only when there are tests running in continuous integration, the feature has been tested with an exploratory approach, and ideally a product manager has accepted the change. Rather than pushing the responsibility of understanding quality onto a person in the testing role, each person in the chain becomes a test expert in his part of the product and the technology stack he works with.
Note: Having a continuous testing approach doesn’t mean there are no quality experts on the team. Those quality experts are still there, but they are more empowered to focus on automation now that everyone has a stake in the overall quality.
Release early and often
A key element of continuous testing is frequent, smaller code check-ins (several times a day), made possible through heavy automation of manual steps. This means the automated test feedback cycle needs to be fast enough to keep up with this cadence and catch errors within about an hour of the check-in.
This piece is the hardest for many teams to accomplish, since some "test automation" suites fail to find significant defects. Sometimes, the reason for this lies not with the suite, but with teams that try to speed up code releases without breaking down their code releases and check-ins into smaller, simpler pieces.
The main objective
The objective of continuous testing is improving a development team's ability to perceive product quality at any given moment. Instead of waiting for people in a specific role to jump in, investigate, and report back, continuous testing delivers information about quality each time new code is written.
This is crucial for companies that want to release software faster than the standard two-week Scrum cycle. In a continuous testing organization, if the team uses a Kanban board, there is no need to "batch" work into sprints or iterations so it can be regression-tested, because regression testing is happening all the time thanks to automated processes in the CI system.
Continuous testing: Start the conversation
Is continuous testing a buzzword? Perhaps. But like most buzzwords, there's substance behind the hype. DevOps, continuous delivery, and even more technical terms such as microservices are buzzwords, yet despite the frequent confusion about their exact definitions, they often ignite important conversations in the IT community about better ways to produce software. Continuous testing starts the discussion about how testing work needs to change to support the next generation of software lifecycles.
The average software project involves a programmer sitting at a desk on his own writing code. He might collaborate with a product manager briefly to ask a question, or a tester might demonstrate a bug she just found, but for the most part each person does his or her own thing.
Testing generally includes a handful of unit tests, but the real testing work is pushed to a person in a dedicated testing role. Continuous testing often means there are fewer people, or no people at all, in the role of full-time software tester. This isn't because software teams don't need testing; it’s because testing is happening systematically at every step. Work that is traditionally pushed off onto other people and seen as "something I don't do" is handled responsibly.
In continuous testing, everyone on the product and technical teams tests the software. They aren't all experts in the field of software testing as they would be in a more traditional testing role, but they are experts in their own areas, and that is usually enough.
If you are starting to see the connections between continuous testing and modern software development approaches, then you're already ahead of the curve.
How continuous testing meshes with DevOps practices
DevOps comes with a bunch of process, system, and architectural requirements. On the process side, there is a culture of cross-discipline collaboration. For the most part, developers work together when they write code, and sometimes they pair-program. The stories they work on are measured in days or hours of work. When development is done, that means the code is ready for production, not for some other person’s intervention.
On the technology side, DevOps is about understanding and automating hand-offs of code between previously siloed departments. Continuous testing focuses on how the testing and QA silo breaks down and spreads out to participate in the entire software delivery lifecycle.
Testing needs to be involved from the very first design specs to the last stage of ongoing production monitoring. DevOps is an environment where continuous testing can thrive if the organization is committed to making that happen. The key, however, is not just picking the most popular tools for a continuous delivery environment, but deciding how those tools are tied together. The ability to ensure seamless information exchanges between these tools is how you achieve the continuous aspect of automation, by removing the need for manual intervention.
What continuous testing looks like
The cadence of software delivery is completely different in environments where continuous testing is possible. Ten years ago, companies delivered software a few times a year, and the stakes of each delivery were high. An important bug in production meant either that customers had to live with a problem for the next few months, or the next release would lose features while the development team worked on a hot fix.
Five years ago, it became more common for people to deliver software every two weeks. The size and number of changes were shrinking, along with the overhead of delivery mechanisms, thanks to the rise of software as a service.
Development groups performing continuous testing can release software multiple times a day. There are a handful of companies right now delivering new software to production hundreds of times every day by using continuous testing, sophisticated branching strategies, and environment monitoring tools as a last-ditch effort to discover problems quickly when they do happen.
Most agile shops will never get close to this. While a two-week release schedule is much better than every six months, they are still stuck in a rigid flow. Within those two weeks, there is a planning phase, a development phase, a testing phase, and plenty of other ceremonies that steal time away from delivering software to production.
Start your journey
Having continuous testing means having real testing occurring systematically at each level of the software you are building and in the development process. It may feel like a buzzword, but after a closer look you’ll find a compelling story.
The companies that are close to achieving continuous testing have a culture of software craftsmanship. They are willing to try potentially dangerous new approaches, new technology stacks, and, maybe more importantly, practice radical collaboration. Are you ready to get started?
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
