The allure of DevOps is speed. Whereas using agile methods can result in releases of around six weeks or so, with DevOps and its focus on extreme automation and integration, you can release code weekly or even daily. This is exciting for organizations with high user demands and/or competitive pressures to innovate frequently.
Many organizations new to DevOps are eager to get going and see how much time and effort they can save on the development cycle. The problem is that speed can sometimes come at the price of process and quality. For startups, there’s not much to worry about here. With little money and a nonexistent or small customer base, startups can be loose and fast. A few bugs released into production can be fixed before anyone notices.
Established companies or those with large user bases must be more careful. This is why when large companies embark on DevOps, they select lower-risk projects that won’t affect the business if the software has a glitch. These are often experimental projects, such as building a new mobile app or an internal social media site. The company’s goals are to learn and prototype new ideas—perfect for DevOps, since it allows for on-the-fly changes and rollbacks.
When there is success in these endeavors, CTOs and other technology leaders start to think, Hey, why not expand DevOps even into business-critical areas where we need rapid response and the work of our best team members? On balance, this can be a great journey, yet it all depends upon the application’s user requirements. If DevOps is applied to a revenue-generating application and results in downtime or critical errors affecting users for a few hours, you can probably expect a visit from the CEO.
How to ensure quality in DevOps remains a hairy problem. Regardless, once you begin to scale DevOps throughout your organization, you will need to invest more energy (time and staff) into the quality equation.
Unless you work for Google, Netflix, Amazon, Facebook, Microsoft, or a few others, you’re not likely to have a huge staff, with all the right skills, and an unlimited budget to spend on perfecting DevOps at scale. These companies are on the cutting edge of cloud and software development and are inventing some of the technologies and best practices being used everywhere today.
For the rest of us, it’s time to get scrappy and work smart.
Here’s one pathway:
1. Controlled rollouts when a little failure is okay
If your company is able and willing to accept some risk of breakage, do what companies such as Facebook are trying: roll out new updates gradually to a small percentage of users. Then you monitor production and support calls to watch for issues. Once those are cleared up, you can begin releasing the update to larger sets of your users. What’s important here, though, is that you have a large user base so that the sample size can effectively weed out any issues in a timely manner. In other words, if you are a company such as Facebook with tens of millions of users and you do your first rollout to 2% of users, you’ll have a healthy sample for testing the update.
When the stakes are higher, here’s an alternative pathway:
2. Your app needs strong QA, so learn to do it at the speed of DevOps
It’s hard to imagine a bank, insurance company, healthcare organization, or government agency being okay with the controlled rollout strategy for certain applications. A trading application that is down for a minute can result in millions of dollars of losses and lawsuits. A glitchy remote monitoring application for a chronic disease patient could be giving the doctor’s office bad information, leading to a bad decision for the patient. That’s not workable. So instead, you invest more up front in testing and, specifically, test automation. That enables an organization with limited staff to conduct comprehensive testing quickly and efficiently. Yet this all depends on whether the testing can be accurately and quickly tracked back to the business requirements and the code that has changed.
QA planning and tracking are key
In DevOps, test automation should be the norm, with manual testing done sporadically and only when the use case dictates. Young companies or startup divisions in large companies that jump into DevOps will find that there are many test automation tools and frameworks available, from open source to commercial. Open-source automation test frameworks are easy to use and, of course, free, and many teams will start there. As your needs mature, commercial tools offer features to make it faster to set up and run the tests, as well as better features for collaboration.
Yet having a framework for test automation is not everything. There’s also the need to test quality assurance: Did we test the feature well enough to achieve the goal or fix the problem? Developers might see a 99% pass rate and think all is good. Yet if the 1% of tests that didn’t pass relate to a core user function: not good. Whether you do this in-house or with third-party technologies, make sure you have the means to correlate test coverage with business requirements and outcomes, not just with code.
QA takes a team
This is also where having a solid QA team in place makes a big difference. In DevOps, with so much focus on automation through continuous integration tools, some might think that there’s less of a need for quality specialists. Yet that couldn’t be further from the truth, particularly given the challenges of maintaining high-quality releases within fast development environments. The role of the QA team is to ensure that the application is receiving comprehensive testing in the areas that matter most, identifying all opportunities for automated tests and determining how to handle manual testing needs.
When your organization decides to make DevOps a mainstream versus an experimental approach to software development, consider that the basics of good development still apply: a clear definition of requirements and a process to ensure that the software has been developed to meet the specs, that tests have been written to cover the requirements, and that tools or processes are in place to measure the outcomes before and after release.
While each project or application has a different risk profile and unique needs for QA, this kind of discipline will help your organization go far in achieving consistent quality as you scale your DevOps practice.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
