In the parody rock movie This Is Spinal Tap, the band's guitarist gets himself a brand new amp that's so loud, he explains, that its volume knob goes up to 11 instead of just 10. Once you've set up your first deployment pipeline, you may be looking for ideas to make your deployments easier and more reliable. Here are 11 ideas you can use to give your pipeline that extra push.
1. Visualize
Information is power. Think of the diagnostic equipment in a hospital room, which helps nurses and doctors see a patient's status. You can set up a similar set of displays around your team to monitor the health of your deployment pipeline. Instead of displaying a patient's heart rate, your screens can show whether the most recent check-ins have successfully passed your tests. Most continuous integration (CI) servers provide some visualization that you can put up on a screen. Green is good; red means something failed. You can also use tools like BuildReactor, which can poll several different CI servers for status.
2. Static analysis
Now that you can see the status of your tests all around you, you can take the idea one step further. Why not add some visualizations that tell you about the health of the code itself? There are many tools you can run (each programming language has its own set). They can spot everything from syntax issues to potential memory leaks. Many of them can generate reports on the maintainability of your code, which look at complexity metrics, such as method, file length, and cyclomatic complexity (how many paths there are through one piece of code).
3. Stop the line
One major manufacturing innovation to come out of Toyota's production lines was the Andon cord. These cords were distributed throughout the factory and could be pulled by any worker to stop the production line if something went wrong. Instead of carrying on with their work, everyone else stopped what they were doing until the problem was solved. This seems counterintuitive at first, but Toyota found that it actually sped up their production process overall. You should adopt the same approach with your build pipeline. If a test fails, it likely means that a bug has been introduced. You'll know quickly because of the visualizations you've set up around you. When that happens, stop the line and fix that bug. No one should check in any new code until the build passes again.
4. Package once
It's tempting to check out your source code for every build step in your deployment pipeline, but don't fall into this trap. If you do this, your production deployment will have to rebuild your source code, and you can't be sure that it'll be built in the same way. Instead, package your application only once. Create your production package, called an artifact, as part of the first build in your pipeline (usually compilation or unit tests). Each subsequent step in the pipeline should depend on the previous step, with each using the same artifact. This ensures that the tests you run and the deployments to your testing environments happen with the exact same artifact that you'll deploy to production. Less variation equals less opportunity for error.
5. Trunk-based development
Many teams work on different source control branches and deploy to production from the master branch. While this technique works well for some teams (and is very common in open source software), it should be approached cautiously. When you work on branches, it's important that check-ins on those branches are also run through your pipeline. Although some CI tools have features that allow for this, often it's complicated to set up and monitor. The major disadvantage of working on separate branches is that it delays integration. The longer you delay integration, the higher the cost of integration becomes. Instead, have your whole team work on your main source control branch (called the trunk). You'll see immediately when integration work breaks your tests, and because the changes being integrated will be small, you'll be able to fix problems faster. Use a technique called feature toggling to hide work in progress from your users.
6. Go all the way
It's important when you first get started that your deployment pipeline includes deployment to at least one environment. Once you've tested this process, you can extend it to take your code all the way to production. Use the same automated deployment scripts you use in your testing environment so you have a consistent deployment process. Just because it's part of your pipeline doesn't mean it has to be triggered automatically, though. Most CI servers let you trigger a deployment manually so that you can release to customers when you're ready, safe in the knowledge that all the prerequisite tests and deployments have passed successfully.
7. Make it all look the same
Ensure that your testing or staging environments look the same as production. The fewer variations there are between the environments, the less risk there is of unexpected behavior or a failure in production. It's cheaper to find issues in your test environments (or better yet, with your automated tests) than it is to find them in production. There are often constraints that make this tricky for teams, but if you get it right, you'll appreciate it.
8. Don't just deploy to servers. Create them
Too many businesses rely on handcrafted servers that must be manually configured and constantly tweaked. No one knows exactly why they work or how to reproduce them. We call these snowflake servers, and they're dangerous because if anything happens to them, it'll be days before your system is online again. Avoid snowflakes by automating server configuration. If you need a web server running Java, MySQL, and Tomcat, write automated scripts you can run on a virtual machine to install and configure these components. You can choose from a variety of configuration management tools, such as Chef, Puppet, and Ansible, that make this easy to do. Once you've scripted your servers in this way, you can destroy and rebuild them over and over, effectively testing both your deployment and your production recovery plan. In my experience, scripting your server creation is the best way to achieve consistent environments.
9. Automate security and performance testing
Teams often experience release delays because they're waiting for security and performance testing to be performed on their systems. Why not build these checks into your deployment pipeline? There are many performance testing tools available, and you configure them to automatically place your system under load. If it takes too long to run on each check-in, you can set up a nightly build. If you're already scripting your server creation, you can spin up environments dedicated to performance testing. There are also automated security testing suites you can build into your pipeline, such as Zed Attack Proxy, a tool that can automatically test for OWASP top 10 vulnerabilities.
10. Servers as artifacts
Here's an idea: What if you didn't deploy to your production servers? What if you created them as part of your deployment pipeline instead? Imagine having a build that creates your servers, or Docker containers, alongside the build that creates your application artifact. You might then have a build step that deploys the application to the servers you've created. You can run tests against this full environment and, if all goes well, route your customers' traffic to the new environment. To do this right, you'd probably need a step in between to update some configurations to use production endpoints and data connections instead of those you use for testing.
11. Automate pipeline creation
You've learned how to script your production servers, so why not script your CI server and the configuration for your deployment pipeline? If anything happens to your CI infrastructure, you could get back up and running quickly. You could have your deployment pipeline clearly documented in automated scripts. You could share your setup with other projects and teams in your organization, or, if you fancy open source, the world. There are already some great open source scripts available for most popular CI servers to get you started.
Crank up the volume
These ideas aren't necessarily listed in order, so pick one that excites you, give it a try, and see whether it helps make your path to production clearer or more reliable. Let me know how it goes. Share your ideas and experiences on turning up the volume on your pipeline by posting with the hashtag #CrankUpYourPipeline.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
