You're not doing continuous testing unless you're doing testing that occurs before, during, and after each software change is made. That's long been a goal for many testers.
But achieving continuous testing involves more than just willing participants; it requires both activities and "enablers."
Here are some of the key concepts you'll need to adopt to get there.
Embrace these activities
Your organization can adopt these activities to facilitate continuous testing:
- Collaborating on requirements (for instance, behavior-driven development, or BDD)
- Constantly validating changes (for example, regression testing)
- Pairing developers and testers (with exploratory testing and reviewing test cases)
- Doing automated testing in CI (for example, unit testing, API testing, and code analysis)
- Continuously improving test approach, test suites, and test scripts
- Reviewing customer feedback and product ratings
Adopt these key enablers
There are also several enablers your organization should embrace to truly reach continuous testing. Here are seven you need to know.
1. Create an environment of quality
The QA team does not "own quality." QA doesn't introduce the issues that affect quality, and the QA team doesn't fix those issues. It simply finds them.
While finding bugs is an important step in software development, many of these issues could be identified earlier. Shift the focus of testing to start earlier in the development process, and you'll save time and money. If you don't start testing immediately in the development process, you can't do it constantly.
High-quality code starts with the design of the software. Get developers to do test-driven development (TDD). Better yet, get the organization to adopt BDD practices: Have the developers, testers, and product owners all agree on functionality and tests before code is written.
2. Build orchestration and continuous integration
Having automation around your build and creating a continuous integration (CI) pipeline are vital to being successful in your continuous testing effort. While it can be difficult to enforce starting testing before coding, it's simple enough to ensure that, once developers have committed code to a source control repository, testing begins.
This means ensuring that your application builds, that your unit tests pass, that your code coverage is at an acceptable level, and so on.
Developer-level testing is crucial, since it provides rapid feedback and ensures that at the lowest level your code does what is expected. Consider what other things you can throw into your CI, such as static or dynamic analysis, security checks, or API validation. The more information the better, as long as feedback is rapid and readily available to the team.
3. Implement virtualization, emulation, and simulation
Rapid feedback cycles are key to continuous testing. The best way to get rapid feedback within your DevOps pipeline is to test only what needs to be tested, on the bare-minimum application.
Often, this means that instead of taking 20 minutes to stand up a full stack of your application, you launch just the API and do some API testing. When you can, use the cloud to quickly deploy your application, then kill the app when done as a cost-saving measure.
When testing your full application, consider what feedback you are really looking for. For your web application, do you really need a full Chrome browser to run the tests, or will an HtmlUnit driver suffice for initial testing? Does that simulation give you enough immediate information to make a go/no-go decision?
If you have a mobile application, can you use an emulator instead of a real device? What will be lost, and will it be missed? If you have an application with many microservices, do you need them all running, or can you get away with mocking some of them to speed things up?
Seconds that can be shaved off by using quicker devices and tools add up to minutes and hours over the course of builds, which means quicker development cycles and money saved.
4. Welcome, containers
Docker is one of my favorite new technologies and how you can use conntainers to facilitate testing. While containerization is great in its own right, being able to download the application and run it anywhere the exact same way makes testing super easy. It helps remove that cringe-inducing phrase, "Works on my machine."
Getting your application into a container, be it Docker, Mesos, or LXC, allows testing to be done anywhere and everywhere.
I like to take this one step further and containerize my tests as well. Not only does that make it easier for tests to run within a pipeline, but no machine configuration is needed. That means any team member can download and run your application and test containers, and retrieve quality metrics without any complex system setup.
5. Use production-like environments
One of the biggest benefits of setting up a deployment pipeline is having the ability to automatically deploy your software in a rapid, repeatable fashion. Ideally, you'll deploy to one system, run your tests, provide feedback, then rinse and repeat.
In each cycle, your tests will probably take more time, which is fine. You want rapid feedback initially, and kicking some tests further down the line can help accomplish that.
You want to ensure, however, that your environments all are as production-like as possible. There is limited benefit to testing if you know the setup won't match your end state. While this can get expensive, you don't always need a full scale deployment for each environment; you can and should scale as you get closer to your production deployment.
Figure 1: Scale your deployments to get progressively more similar to production as you move further right, to better simulate production-like testing.
This also means that for test design and development, keep in mind the access you will have in each environment, and the data you want/need to be testing. How can/should you be seeding this data? Do you need access to a specific level? Are these tests destructive to data, or even the application?
Thinking and designing with these factors in mind will ensure your tests can run on any environment, which will allow more quality checks throughout the development process.
6. Manage your test data
Dealing with test data is one of the biggest struggles within organizations. Too often, people don't think through what data is needed for testing. As a result, organizations end up using incomplete, unknown, or way too much data for testing.
Some organizations also grapple with finding the correct data to execute a test, some of them spending more time to locate and load the correct data than they do actually testing with it.
When designing a test, be it automated or manual, pay attention to the data needed to exercise the functionality. Pay attention to the data needed for each test, and be sure to tie that data to each test.
If you're running only one test, and that test needs only one user, don't spend time loading a million users into the system. Conversely, when you're testing further to the right and performance is important, ensure you're using a dataset that properly stresses the system A few users probably aren't enough in that scenario.
7. Automate testing below the UI
When testing, your application is more than a user interface. You probably have some APIs, a back end, and a database, all of which need testing. Not only is it easier to test at these lower levels, but usually you can test these areas sooner in the development process.
Figure 2: While it's tempting to test at the UI level (traditionally how testing is performed), that's slower and more expensive than using other methods. Instead, test below the UI as much as possible.
Similar to a database or back end, the UI doesn't need to be fully developed to stand up an API and start testing it. So you can get started testing sooner, which is the goal.
Additionally, typically API tests run tens to hundreds of times faster than do UI tests. This plays in perfectly to the idea of rapid feedback, and keeps pushing testing further left.
Test early and often
Remember to build your CI pipeline; make quality a whole-team endeavor; automate, emulate, and simulate as much as possible; embrace containers; use production-like testing environments; and manage your test data carefully.
Most important, keep in mind that you can't test continuously if you don't start testing early. Happy testing!
For more on the key enablers to continuous testing, come to my talk, "Getting to Continuous Testing" at STARwest 2019, which runs from from September 29 to October 4 in Anaheim, Calif. TechBeacon readers can save $200 on registration fees by using promo code SWCM.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
