So you’ve adopted agile. You’re holding your stand-ups, scoring your stories, doing the ceremonies the experts recommend. You’re even beginning to see the value of the agile process. You’re showing development progress in biweekly increments, including user feedback on a continuous basis, and everyone’s excited!
But something feels off. It was great to start, but now your velocity is erratic. The defect list keeps growing. Testing takes longer and longer all the time. “How can this be?” you wonder. You’re doing all the right agile development things.
You’re not crazy. You’re suffering from a common malady I call “agile insufficiency.” The truth is, agile on its own doesn’t deliver ROI. When you improve your development process without improving deployment as well, you eventually, inevitably, face these issues.
While this comes as a surprise to many organizations, the writers of the Agile Manifesto knew it well. That’s why they included lines such as “Deliver working software frequently,” and “Working software is the primary measure of progress” and “Sponsors, developers, and users should be able to maintain a constant pace indefinitely.” They understood that software is not done in any meaningful way until it’s deployed and in the hands of customers.
That’s where DevOps comes in. DevOps is all about automating and improving your “value stream”—your process for building, constructing, and flowing features from development into production. It unlocks hidden capacity in your organization, allowing you to go fast forever without working overtime, nights, and weekends. It’s agile’s indispensable partner. I’d even argue that you can’t get ROI from agile without it.
The question is not whether to do DevOps, but how to get started. I could write a book on that topic (and many have), but the basic principles can be briefly summarized. Here they are.
1. Kill your manuals
I'm not talking about your instruction manuals, but your manual processes, which are the most common sources of error and delay. Survey your software pipeline and train yourself to watch for the word “manual.” Manual testing. Manual deployment. Manual handoffs. Any activities where teams communicate through tickets. Those are tell-tale signs of waste and good candidates for automation.
2. Find your bottlenecks
Implement instrumentation throughout your pipeline to see where code is being held up, then focus on eliminating choke points. Nothing can move through the pipeline faster than the speed at which it can go through the slowest parts. Improvements anywhere else in the pipeline will only make the bottleneck worse, since they just pile up more tasks at the choke point. The only way to make progress is to fix your biggest hold-ups.
Once you’ve automated the slowest parts of your pipeline, something else will be the slowest, and it will be time to tackle that. You’re always getting better, but there will always be room for improvement. It’s a never-ending process. Embrace it.
3. Automate your tests
This is a key enabler for really effective agile. I’ve heard it described as the gold standard, but not absolutely necessary. Wrong. Automated testing is absolutely necessary. It’s what allows you to go fast forever. People like to say they do test-driven development (TDD), but often they just mean writing the test before the code. Real TDD is automated. And effective TDD is done right now. People talk about going back to write tests later. It never happens.
If you’re going to do it (and you should), do it now. Don’t just automate UI tests, but integration, unit, acceptance. Yes, it takes longer to write a test on top of feature code, but it doesn’t slow you down in the long run. In fact, an automated regression suite helps you achieve a constant and indefinite pace, as the Agile Manifesto envisions. Trying to do agile with manual testing is a recipe for failure. Commit to automation with everything you have and protect it at all costs.
4. Build in quality
This means shifting quality control to the start of the process and baking it in throughout. Why accept a story and count the points only to create a defect? Why not just fix the problem and accept the story defect-free? It doesn’t change the amount of work you have to do. When you account for rework, it actually costs more to do it later.
Focus on improving your processes—both the agile process, where you focus on the flow of stories, and your deployment process, where you focus on getting the software to end users. Don’t fall back into waterfall habits. Worry less about making your commitments than about making work flow smoothly and consistently through your pipeline.
To that end, establish tight feedback loops with QA/QE and customers. Test within your sprint, and only accept defect-free stories. I’ve seen too many organizations pressure teams to meet every sprint commitment, only to discover in the end a bunch of defects that push their release date out further.
5. Automate your pipeline
There are two parts to this equation. The first is continuous integration (CI). That means replacing all the manual processes associated with putting source code together: building, integrating and creating a deployable artifact that you can deploy in any environment. Old-school shops may do integration once a week. They have one big integration day when everyone puts code together and works out merge conflicts, and then they kick it over to testing and go off to build the next set of features.
As integration becomes more painful, they push it off even further. Integration every week becomes every two weeks, and then even more. The longer the two code branches are apart, the more risky and costly integration becomes. CI solves this problem by testing integrated code with every change. Short-lived branches are reviewed and integrated daily. The automated process runs every time there’s a source code commit, performing security scans and static code analysis, ensuring developers are adding code that moves the project forward.
Dev shops often have a CI solution in-house, which is good. Where they can fall short is continuous delivery (CD), a process that automatically takes output from the CI, combines it with secrets (passwords and other items you shouldn’t store in source code) and puts together in an environment.
The key is to use the same automated process whenever a package goes to any environment, whether it’s dev, QA, stage or prod. A human should never go through and click configuration steps, or manually copy software into an environment, or manually stand up servers. Every CD deployment should be automated the same way, so every time you run the process it's like a dress rehearsal for live deployment.
To make this work you must get buy-in from ops. Dev teams sometimes go all-in on CI and CD, while ops continues with manual processes. This results in larger batch sizes, greater risk of error, and higher rates of failure as features stack up for deployment. Don’t let this happen to you. Everyone from dev and ops needs to be on board with automation, and the same process needs to be used for all deployments to all environments.
6. Choose metrics that drive the right behavior
Any time you establish a metric, people will try to game it. And if you measure the wrong thing, you’ll get the wrong results. Both are common problems in software development. Project managers try to measure progress with metrics that drive the wrong behaviors. Percentage done, lines of code, number of defects, number of stories—these metrics can be easily gamed to give the appearance of progress, but they don’t generate business value.
Choose metrics directly related to your business goal: getting features into the hands of customers. It really is that simple. How long does it take to get something from idea, to end user? To measure that, we propose these three metrics:
- Lead time—Time from first line of code to deployed to production. Many businesses would be shocked to find out that the actual lead time can be six months, eight months, even a year. The more you reduce that lead time—if you can bring it down to months or weeks or hours—the better for your business. The metric tells you a lot—not just about how fast you can write code, but how fast you can get out in the world.
- Deployment frequency—How frequently do you deploy? Once a quarter? Twice a year? Or 11 times a minute? More frequent deployments are a good sign that your entire DevOps pipeline is becoming faster and more reliable. The metric also encourages smaller batch sizes, which tend to produce more reliable code.
- Mean time to recover—Failures happen. They happen with waterfall, they happen with agile, they happen on-premises, and they happen in the cloud. We have to be able to both detect them and recover from them quickly. DevOps is about operating safely at speed. Sometimes you’ll miss an edge case, or a server will go down. The questions are, How fast can you recover, and how do you do it? Rollback? Fix-forward? A manual legacy release process might take a weekend to deploy, and all too often you find out on Sunday that something went wrong and you need to roll back. But if you create blue and green deployment environments, and deploy live by switching between the two, you can switch back if there’s a problem and recover in about a minute. Big difference.
Start small, think big
If this all sounds like a lot, don’t worry. There’s no harm starting small with DevOps. In fact, I recommend it. Groups I’ve worked with that commit to improving over time have been more successful than those that try to do everything at once in a big bang. Agile alone may not be sufficient for developers to go fast forever, but it takes you a long way, and you can get the rest of the way over time.
Just focus on the simplest possible task in front of you and making each commit a bit better. As long as you have all the stakeholders on board, every bit of progress is a net win. Small wins attract support for bigger ones, and pretty soon you’ll have created a virtuous cycle that generates astronomical returns.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
