Nowhere is the disruption of DevOps more visible than in the way teams implement release management. Teams can't even agree on what it means to release things!
Agile teams often preach that "deploy continuously and release on demand" is smart and sensible, but there is something to be said for managing the process of pushing stuff into production with some of the old practices of change-advisory boards and checklists.
This is amplified with the increasing focus on preventing supply chain attacks and other security issues. Here's what you need to know about the convergence of old and new, and how to get the best of both worlds.
Clarify what you're talking about
First, a few words about terminology: Release typically means making something available, whether an application or a feature, to whoever requested it.
If you are not in a DevOps setting, then a development organization is "releasing" what it created to the operations team. If you are in a DevOps setting or if you think of releases from the perspective of end consumers of a digital solution, then a release is when the operations team makes the app, service, or feature available.
A DevOps team will typically talk about "deployment" as the process of moving from the dev cycle to the ops cycle. The DevOps person would not call this releasing. To that person, release comes after deployment; it's when users are onboarded to the new features that were deployed.
A lot of the conflicts I see are due to the above two views not being understood and appreciated. The figure below illustrates the difference.
Source: Lars Rossen
In classic release management, there is strict control between dev and ops. Everything that operations has deployed in production is implicitly available to consumers of the service, and any issues are dealt with through support.
By contrast, in DevOps, the same team handles both dev and ops, so deployment is just one of many of the team's activities, and deployment happens continuously. However, the DevOps team will carefully plan how to make new features available and, if there are any issues, will restrict the number of users until every issue has been dealt with.
The benefits of release control
The primary purpose of the release control function is to ensure the "fit for purpose" of anything that is pushed into production—whether that's new functionality, simple patches, or completely new systems.
Typically, operations teams are concerned with these questions:
Are we breaking already-running functionality?
Will the release be secure?
Is operations ready to manage the release?
Are performance and all other nonfunctional aspects of the release ready?
Are all the associated artifacts, such as training, documented, and are known errors understood?
The benefits of release on demand
Here's what the DevOps team is doing that traditional operations teams typically struggle with: releasing new functionality in a controlled manner. The goal is for new functions to be tested and scaled up to everyone only once they are proved to work. In fact, this is likely possible only if there is a tight link between development and operations, since the ability to make controlled releases to users depends on capabilities in the developed code.
The benefits are huge, as anyone who has tried to manage upgrading software with a large user base can attest to. No amount of preproduction and acceptance testing can find all the issues that eventually will pop up in production.
The best approach: Practice agile and automate
Well-implemented DevOps teams can be significantly more productive and deliver a higher-quality product than can traditional IT setups where dev is separate from ops. However, if you are not a greenfield shop, this represents a significant cultural shift, and, on top of this, many legacy applications do not easily convert into this new paradigm.
Typically, the ops people will say, "I need a release control mechanism between what the developer is doing and I'm doing." This is a sign of a lack of trust. Use this as an opportunity for dev to automate as much of the traditional release controls as possible, not throw them away. And if dev wants to call it "deployment controls" or "pipeline checkpoints" that's fine—it is the same thing.
But ops could teach some tricks as well, including how service catalogs can be used as a way of exposing new features to control the release to consumers. Operations teams have a lot of experience in dealing with service catalogs, service subscriptions, and support based on contracts. This can actually be used to improve the current DevOps practice where the rollout is controlled behind the scenes based on release flags, etc.
We are building this way of combining the best of both worlds into the upcoming IT4IT standard, with new value streams managing deployment and release separately. This is simply illustrated in the blue value stream arrows below:
Source: Lars Rossen
There is another reason it is important to maintain some of the existing properties of release control when moving to a more agile approach: to automate and document tacit knowledge.
It is critical to be able to reconstruct something and know what the content is of a given release, because then you can also do an audit and look at the security controls. This is why your releases have to be centralized and managed, not only by the DevOps team but by the IT organization as a whole.
They need to understand what goes into production. If IT does not, then the business may be susceptible to a security risk.
Finally, if a release is not well documented and standardized, you will not be able to easily resurrect a DevOps team years later when a product or service that has been in low maintenance for a long time needs new upgrades.
Best practices of good release management
So which approach is right? Certainly, moving to continuous release is where we as an industry are going, but there are gray areas in the real world, and the solution needs to accommodate that.
Sometimes your systems are in a stable state, but they may need minor tweaks. However, when issues crop up, someone needs to do something about that.
This is why it is important to know what the previous team put in production and have centralized management. If every DevOps team has its own pipeline and methods for managing operations, it becomes almost impossible for a new team to manage something without introducing errors.
You should manage the transition from dev to ops in the following way:
Ensure that anything deployed into production (and potentially, also preproduction and system test, if relevant in your setup) is well defined and reproducible over time.
Provide good records management on release/deployment artifacts such as binaries, release notes, configuration scripts, and compilation of known errors. Often, reproducing entire sets of assets relies on tacit knowledge.
Check your supply chain: Are any assets in the compiled release certified? (That is, guaranteed that it was delivered by the expected supplier. This is typically achieved through code signing and good processes.)
Protect against malicious or negligent release mistakes. Ensure there is always more than one person involved in new deployments to protect against inside threats.
Ensure that code has been scanned before release, preferably, in a "shift-left" manner so it is not a last-minute, blocking process.
Run automated functional and nonfunctional tests with automatic risk scoring. In this way, you can efficiently and continuously evaluate the release candidate.
Stick to your principles, but automate
Don’t throw out the baby with the bathwater. Hold on to all the principles of classic release control in IT management. Learn from them, because a lot are still valid. That said, DevOps teams should be sure to automate as much as possible.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
