My previous article, "10 best practices for QA teams to deliver quality software fast," provides practical tips for helping quality assurance (QA) teams ensure the highest quality of a release. But in real life, and specifically in the world of agile, you can’t QA test everything when running short release iterations, and you need to have the right release management techniques in place. So how can you still be confident in the quality of the product you approve?
Release criteria and planning the next test release have always been vital and with approaches such as adaptive release governance, it's even more so in today’s agile development realm. In an agile world, release criteria can sometimes be thought of as "the contract." The release criteria are a pre-defined contract agreed to by the team and stakeholders. Once the criteria are fulfilled, there should be no disagreement about what was promised and what was actually delivered. Here's some advice on how to plan tests and release criteria the right way.
Prioritize testing in areas of the codebase with high activity
During development iterations, our agile team meets with project stakeholders to try to better estimate the potentially affected areas of the product, so that we can focus our efforts in the testing phase. This also helps us plan the next release, including specifying the release criteria before the development cycle even starts.
With any large-scale R&D project, it’s important to look beyond the current structure of the code. You need strategies to identify challenges and bottlenecks that inevitably come up in any product development and release. In his article "Code as a Crime Scene," Adam Tornhill compares software bugs to criminals. Just as we want to hunt down defects in the real world, we need to find the bugs and the offending developer to make the codebase more resilient and help the developer make better decisions. I like this comparison because it encourages us to ask the right questions when planning our tests.
The identified hot spots, the parts with high development activity, will be prioritized in testing. This strategy provides us with a guide to our code. It's a guide that shows where to focus our cognitive cycles by answering questions such as:
Police | Software development |
What area is most likely to serve as home base of the offender? | What's the most complex spot in our software that we're likely to change next? |
Where can we expect the next crime to occur? | What are the typical consequences of that change? |
Are there any patterns in the series of crimes? (i.e., can we predict where the offender will strike again?) | Is it likely to be a local change, or will other parts have to change too? (i.e., can we predict the impact on related and seemingly unrelated modules?) |
Source: Code as a Crime Scene - the Article
Before a new release cycle begins, our testing team works to analyze and estimate which parts of the product will be touched by new or updated code. We then use that information to prioritize our QA efforts. If you know something worked in the last release and you’re not touching it in this release, you don’t need to spend too much time testing it (unless, of course, it will be running in a different environment, such as a new browser or operating system update). Just make sure automated tests continue to validate what’s done.
When determining test coverage and the number of known defects, our QA team commits to testing all new features—which means conducting a full test cycle and automating tests of main flows. We use backward compatibility and regression tests.
In one instance, we had a significant new feature called "Project Management" that was developed on top of our current product. Most of the product was affected, and we were all aware of the massive change, so we decided to do a full regression test for most of the product.
In another case, we added a new theme to the product that was self-contained and had only a minimal effect on the rest of the product. For this, there was no need to do a full regression test—just a quick sanity test.
If a large feature is being released, we may even meet with a customer to present and discuss the new release. Since we actually use the products we test, we are one of the customers (see "Eating your own dog food"). Therefore, we can also provide feedback on usability as we test.
Quality is not a 'yes or no' feature, it's a threshold
QA can use various methods to determine if the user flow makes sense, including using feature toggling, which allows us to reveal the new functionality to only a few selected customers (i.e., our beta testers). The basic idea of feature toggling is to have a configuration file that defines various toggles for pending features. The running application then uses these toggles in order to decide whether or not to show the new feature.
Security must also be considered. A special QA team should be designated to run penetration tests, code analysis, and third-party review.
As far as the performance is concerned, the release criteria take into account the service-level agreement (SLA) that the product guarantees to our customers; for example, uploading a 3 MB file should take only 10 seconds at most in a modern network environment. The last step is test tagging, which provides a final sanity test in a simulation of the customer environment before release.
At times, we might have 1,400 builds over a six-week time frame. From a quantity perspective, we have specific thresholds per release for defects: Critical - 0, High - 0, Medium and Low - 75 percent of defects discovered during development. You don't have unlimited resources and time to test everything to perfection (assuming you can achieve perfection, which you can't). You have to set thresholds so that you don't exhaust your resources.
Behind the scenes, QA is the star of the show
The QA team sits between the customer and development, which allows test engineers to see the entire picture. When a customer has an issue or problem related to quality, it always falls on the shoulders of QA. Questions will arise about why QA didn’t find the problem earlier, or how a particular usability issue occurred. Therefore, it is only natural that QA should be the primary team involved in defining the release criteria.
For any large-scale project, the question "Is this software ready for release?" will inevitably come up. Asking the right questions and clearly laying out your release criteria in advance will help you solve this riddle. The key takeaway is that your QA team should know what success means for the product and then quantify how you recognize that success. Once you’ve done that, you can implement release criteria and benefit from having a repeatable standard for quality in all your product releases going forward.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
