How to scale agile and DevOps together

Agile, DevOps, continuous delivery—and all of it at scale. How do these things come together? Businesses want to do more with the technical advances that DevOps and agile have conferred, but when it comes to next steps, those who advocate “scaled agility” seem to talk past those who discuss “the DevOps enterprise.”

Are they talking about the same thing? Can scaled agile frameworks help businesses push DevOps capability to new levels of competitiveness? Or is DevOps, finally, what scaled agile frameworks such as SAFe, DAD, Nexus, and LeSS need, in order to lift those enterprise-scale practices to levels where they can truly make a difference?

Given the huge popularity of both agile and DevOps, I asked several experts what they think of the interplay between these modes of software development, and how agile and DevOps complement each other at scale. Here are the key takeaways from those discussions—things that often don't come up specifically as teams start to organize their efforts around DevOps and continuous delivery.

Talking agile vs. DevOps: Separate conversations?

The most productive way to understand the relationship of agile and DevOps is not as an intersection, but as a progression of capability. So you're not necessarily having separate conversations. You can place them along a continuum, with agile on the left, DevOps to the right of center, and continuous delivery on the far right, as the ultimate goal. That’s a progression that organizations can move through, in stages, with some teams progressing faster than others.

This is also a progression that loosely describes the history of where we are today:

• Agile software development methods grew increasingly popular from 2001 to 2008, because it freed developers from old ideas about plans and schedules that restrained creativity and/or discouraged experimentation.
• As rapid agile software releases became essential to business claims about competitiveness, pressure mounted for production teams to keep up with the changes and satisfy customers’ runtime expectations.
• By 2010, DevOps emerged as a way to integrate rapid releases with responsible deployments, and DevOps has defined IT practices for many of today’s most effective software delivery teams.

Most organizations working to apply agile methods at scale today began small. (You and your team can probably relate.) They got to the point where more teams and more projects were adopting Scrum or XP. By the time DevOps entered the discussion, experienced teams were ready to expand agile beyond the boundaries of development and QA.

If you feel that this continuum and brief history is an over-simplification, you may be right in many cases.

Improve your flow: Get teams working together

“Our goal was to put more effective working relationships in place to improve the flow of work, as well as the quality and speed of that work,” say Scott Prugh, Chief Architect and Vice President of Software Development Operations for CSG International, a company that operates customer billing systems for the telecommunications industry. Over the past few years, CSG has been exploring DevOps as a way to examine its own technology culture and organization, and to improve its techniques for managing operations.

“We found that it doesn’t matter which framework you adopt. The key thing agile did for us was helping teams manage much more of the development lifecycle."

This includes operations, infrastructure, and sysadmin roles, as well as design, architecture and quality, Prugh says. “Those responsibilities used to exist in separate organizations, and that separation wasn’t good for an optimal flow of work. Teams couldn’t learn from each other, and the separation magnified the problems with hand-offs between one specialty and the next.”

Scaled frameworks for agile offer a more cross-functional approach to the roles in the development lifecycle.

“Teams can now learn at a faster rate, so they can build and deliver to customers a lot more efficiently."

Add Ops guidance to your agile framework

Teams working towards continuous delivery through a scaled DevOps approach shouldn't wait for their agile frameworks to provide detailed guidance. “What we’ve taken to the next level is going further with those frameworks, and embedding more operations components so that teams can deliver the entire service,” Prugh says. “That has included designing and developing new functional features for customers, as well as developing operational features to make the software more efficient.”

That means these teams now own much more of the operations concern within the entire lifecycle. “Dev and Ops are much less at odds with each other in a management and organizational sense.”

Adopt a build-and-run teams concept

A general consensus has evolved among organizations that have adopted DevOps at the project level, what Amazon calls “build and run” teams. The idea is that teams that build and run their own software can improve systems faster. They’re accountable for more of the software delivery chain. Before that, accountability lay with a completely different organization. These teams also have a more general understanding of the issues that, when addressed, will make the software run well.

“At a basic level, you need to fold the components of the operational concerns into the teams that build and run the software,” says Prugh. “At CSG we call them productivity teams. You just can’t transfer accountability, assuming another group is going to take over your operational concerns.”

Move away from process thinking

Much of operations has traditionally been viewed as a process activity. While ITIL is a well-regarded process framework, Prugh’s team accrued additional benefits by folding some ITIL processes into the teams that have the engineering expertise. “You can engineer those processes so that they’re highly efficient—whether that’s brute-force automation around those processes, or changing your software so those processes are effectively built into your delivery mechanisms,” he says.

Build-and-run teams solve the problem of the agile development-to-operations handoff. Typically, there’s little continuity.

“[If] we can fold operational processes such as ITIL into the dev framework, and improve team structures, along with the skills and roles that need to be on those teams, then we’ve got much more cross-functional capability to deliver great features."

Get specific: Use DevOps to improve SAFe

While  Prugh says it doesn’t matter which scaled framework for agile you adopt, it helps to see the infusion of DevOps principles in terms of one specific framework. In this case, I’ll use the Scaled Agile Framework, or SAFe, since it’s well-known, and since it should be relatively easy for readers to see how these principles can be applied to other frameworks.

Topo Pal, Director and Platform Engineering Fellow at financial services firm Capital One, describes how DevOps principles aid SAFe in his organization. “SAFe doesn’t explicitly say you have to automate everything,” he says. “Early versions of SAFe suggested that some functions could be done in an automated way. But in my experience of DevOps you can do automation at any level, and you may not need the ‘system team’ described in the SAFe framework.”

System teams in SAFe are all about the operational aspects of a release, especially continuous integration, infrastructure, and automated builds. But SAFe does not stress that these things have to be automated, although automation can add considerable benefits. “With DevOps practices, automation adds another level of capability to SAFe,” says Pal.

“If I’m a developer, I should be able to build and run my test cases, but if I have to rely on another team, I’m faced with the same old bottlenecks that plagued waterfall methods.”

Isn’t DevOps already included in SAFe?

Yes… if you take a look at Scaled Agile Inc's SAFe diagram, you’ll see that DevOps represented in the big picture in the upper left corner, and in the accompanying article. The confusion, if there is any, has to do with where you are in the agile/DevOps discussion.

When people think about DevOps, SAFe—or any scaled agile framework, for that matter—won’t be the first thing that springs to mind, because frameworks like SAFe have more recognition on the development side of things. Currently, DevOps appears to be an afterthought in the SAFe model. Which isn’t to say that will always be the case.

“SAFe is really focused on getting the enterprise to consider how it delivers value,” says Steve Mayner, senior program consultant at SAFe Inc. “That value is critical to a company’s customer base, to its monetization model. The principles and practices are about identifying the full ecosystem, regardless of existing silos and skill sets that it takes to go from concept to cash, to get the value out the door.”

Move more people under your umbrella

In the SAFe conception, the agile release train requires roughly 50 to 150 people to deliver that value, “and that isn’t just about developers,” says Prugh. “It includes a huge number of people associated with operations, whether they’re sysadmins, security, infrastructure, help desk…all of those functions outside development have to be on board. They have to meet together regularly, to integrate, to demonstrate what they’re building iteratively."

"If you take the operations players off the train, all you’re doing is moving the bottleneck.” 

Mayner agrees: “You can develop things really fast, all the way to the point of release. But without the folks in charge of the release process, or moving things into production, or supporting the release for the user community, you only get partial benefit. Of course, SAFe didn’t grow up in the center of the DevOps conversation, but it has always been part of the core assumption that the full ecosystem has to be end-to-end.”

Important software delivery practices—such as consistency in platform configuration, from the developer machine to the test environment, to staging and production systems—can be adjusted for continuous delivery, Mayner says.

“All of these practices in a DevOps approach can be automated for change management and configuration control to allow the kind of quality that supports a continuous delivery model.” 

Next steps in the move to enterprise DevOps

Invariably, the agile conversation comes first, and it’s likely that your own organization is beyond simply having that conversation. Now, if you’re exploring DevOps, especially as a way to do more with what you’ve learned, you may be ready to apply agile principles to multiple teams, and full programs. This is a good place to be, because “development practices tend to be on the same page,” says Mayner. “Larger teams are saying ‘Wait… this isn’t going out the door unless our folks in production, support, and maintenance are also on board.’”

That’s DevOps.

Here are the five steps you should take as you begin this conversation.

1. Identify inefficiencies in your current flow. What can be automated for improved consistency and repeatability?
2. Determine what inefficiencies are the result of hand-offs from one team to the next.
3. Adopt a build-and-run team approach to your next project.
4. Whether or not you’ve begun using one of the scaled agile frameworks—SAFe, DAD, LeSS, and Nexus are among the most popular—take the time to study these diagrams and learn which of the practices described map to your own, or might help you improve your own practices at scale.
5. Consider ways that you might expand your current agile practices to include operations concerns. That might include, for example, moving configuration management into the team that was formerly just “dev.”

Expanding to a DevOps mindset is “a major result of our training sessionsn [in his SAFe consultancy,]" Mayner says. “We stress the need to have all of the ecosystem working together so that all the operation teams are participating in the release train and the continuous integration planning event.”

With larger customers who have only just begun to work in an agile manner, the progression from agile to continuous delivery might be greatly collapsed, says Mayner.

Organizations that are behind the curve, but which wake up and see how their practices need to evolve, and whose leaders educate themselves in both agile and DevOps, are now getting it. These teams don’t want to start at agile and later move to DevOps, Mayner says. "They want to start with what's state-of-the art today. They tend to jump in with both feet, with a flow-based awareness, and suddenly everybody’s on the train together.”