While the practice of CI/CD (or continuous integration, continuous delivery, and continuous deployment) is not new, the ways to implement it continue to evolve along with cloud-native architecture, site reliability engineering, and DevOps. As challenges arise with maturing CI/CD, enterprises can learn from past mistakes and industry expertise. These challenges come from all vectors of software delivery, from risk/compliance and monitoring to data management and team collaboration.
To learn about common challenges and potential solutions to enable CI/CD, I asked several SKILup Day speakers and subject-matter experts for their insights. Here's what they had to say.
1. Avoid service performance issues by hardening deployments and delivery teams
CI/CD is the common software engineering practice associated with enabling development and operations teams to deliver apps and services faster by injecting automation techniques into the software development lifecycle. CI/CD helps to address the challenges associated with integrating new code more frequently and seamlessly across teams.
For engineering teams, a big challenge of CI/CD today is delivering their work without degrading service performance or quality. Every deployment has a risk associated with it, and organizations need effective means for controlling and reducing those risks. Red flags to watch out for include high change failure rates and a risk-averse attitude toward managing or taking on CI/CD responsibilities.
Tiffany Jachja, engineering manager at Vox Media, said, “Successful CI/CD depends on the hardened practices of developers and engineers, so it’s worth looking into how your delivery team is reviewing unit test or integration test results, broken builds, vulnerabilities, and incidents.”
“Contributors to deployment risk can involve inadequate testing, inadequate solutions for configuration management, non-reproducible environments, and environments that are not well defined, controlled, or maintained.”
—Tiffany Jachja
2. Merge the islands of automation
The end-to-end delivery process has too many of what Anders Wallgren, vice president of technology strategy at CloudBees, calls “islands of automation.” “CI processes might be highly automated,” he said, “but then there's manual work involved before the CI artifacts are available in a test environment. Or there's manual testing required before the software can be promoted out of a test environment.”
“Whenever we have breaks in the automation, these are opportunities for errors, delays, and a general lack of traceability.”
—Anders Wallgren
In many organizations, software delivery teams suffer from some form or other of manual processes in between the automated processes. This is often where the cracks in the CI/CD process show up. One solution to this problem is to introduce orchestration across the software development lifecycle, so teams can be freed up to evolve and focus on continuous improvement metrics rather than plugging gaps.
3. Be data-obsessed
Simply put, if yours is not a data-driven organization, it’s likely losing to one that is. Breaking down data silos is at least as important as bridging automation or process silos. The CI/CD pipeline, for example, must be built with information accessibility in mind to help people make sound decisions based on data that is pulled across many different systems.
Unfortunately, said Grant Fritchey, DevOps advocate at Redgate Software, “Over and over I’ve seen data being the big challenge for DevOps implementations. Data can be so difficult to deal with that many organizations will simply skip it as part of their CI/CD pipeline. However, with discipline and tooling, you can implement data as a part of the entire pipeline, from the developer's laptop through the entire CI/CD process.”
“Taking data into account from square one results in a much more successful overall process.”
—Grant Fritchey
4. Considerations for platform engineering practices
When it comes to platform engineering, you have to find the lowest common denominator between teams. As Ravi Lachhman, evangelist at Harness, noted, “Platform engineering teams need to accommodate hundreds of services and a myriad of technology choices and architectures. Understanding what is normal is a challenge in and of itself.”
For example, new engineers need to know if a deployment was successful or introduced regression. The ability to aggregate and disseminate information for enterprise systems that have information such as application performance monitoring/observability technology is key, Lachhman said.
“Providing engineers access to these systems or dashboarding those systems’ feedback as judgment calls is an emerging practice.”
—Ravi Lachhman
Vishnu Vasudevan, head of product engineering and management at Opsera, explained that “modern CI/CD requires flexibility and full control over the CI/CD tools and processes for the development and operations teams. We must allow teams to construct the pipeline without worrying about the integrations involved.”
“Teams must have a 360-degree view around the pipelines and be able to reuse them across all technologies, platforms or services.”
—Vishnu Vasudevan
5. Over-communicate and find ways to connect the humans, not just the machines
“Communication has long been a challenge,” said Stephen Walters, sales engineer at Everbridge. “The purpose of agile, self-managing teams was to break down the silo walls between functions such as design, development, and testing. Likewise for DevOps to break down the barriers between teams. However, this is typically trading in one form of silo for another. The use of tools such as Scrum of Scrums tried to alleviate this issue, but has never been fully realized.”
It's true that the CI/CD pipeline has helped create horizontal silos instead of vertical ones. Communications tools such as ChatOps can help with this challenge, but even more important is instituting a culture change to embrace empathy and a more holistic understanding of solution delivery.
“Changing the mindset of people who have never used CI/CD is just as important as getting the correct skill sets into your organization.”
—Jamal Walsh, technical product owner, the Very Group
It takes a village
At the end of the day, it takes groups of people across disciplines to make CI/CD a competitive advantage. From the IT service desk to security/compliance, process reengineering, and architects, creating ways to bring those stakeholders together requires human skills in addition to technical or process-related skills.
Engineers need to become experts in their specialty, but they need the breadth of knowledge, too. With extended skills into the human side of DevOps comes the ability and willingness to think beyond one’s silo—and in this way, CI/CD can become an enabler of success.
If you'd like to learn more about the challenges and solutions for CI/CD and similar topics, see the DevOps Institute's upcoming SKILup Day schedule. Or check out The DevOps Journey SKILbook to discover how DevOps helps organizations.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
