When I was first exposed to continuous integration (CI), in a different decade, I was underwhelmed, I have to admit. Essentially we were talking about software to perform a checkout of all the code, to create a build, and then to email someone if the build broke. Test-driven development (TDD) advocates could take it further, adding unit tests to the process.
Today I know better.
If the entire loop takes less than an hour for every commit, then it will be possible to tie any broken build back to the programmer, pair, or mob that introduced the break. The same goes for unit tests.
Tying the failing tests to the person who broke them prevents the negative incentive where the programmer who creates unit tests is blamed when they fail. Under that system, when unit tests are not running under CI, the programmer who doesn't write any tests and fails to run them gets away with it, getting to "code complete" faster.
This makes CI an incredibly powerful way to maintain the integrity of the system. The challenge, though, is getting CI started, accelerating it, and keeping it running.
To get past these hurdles of CI, here's how to best do continuous integration as a service, or continuous integration as a product—and beyond.
CI as the heartbeat of the project
Calling CI the heartbeat of the project isn't wrong. If anything, it isn't worded strongly enough. After years of consulting for teams and seeing failures with TDD, refactoring, and clean code, the one thing that the failures had in common was ineffective CI.
In some cases, the team did not have CI at all; in others, it simply tolerated failures and made it the fault of the programmer who wrote the tests—instead of the programmer who broke them.
If CI is so powerful, why are so few companies using it effectively? Perhaps because setting it up takes time.
The feature beast
Because it is not customer-facing, CI is essentially invisible to product owners and business customers. That is a problem, since the time that the technical staff spends on these sorts of improvements is time not spent on new features.
Ken Pier, the senior director product quality at Socialtext and formerly with Xerox PARC, referred to this as the "feature beast." He said that unless teams can find ways to break the feature beast, the staff will invariably defer improvements.
While the team gets more features done per sprint, the deferred improvements slowly create technical debt. Agile and lean teams have spent years trying to create ways to get around this, such as the 20% of time suggested in SAFe's innovation and planning iteration. Other approaches include "just do it"—not exposing technical decisions to business customers—or reserving some percentage of work for technical staff.
Without some incredibly motivated employees doing it themselves, CI efforts are unlikely to get off the ground. Even with those employees, CI efforts might require servers and software, which means a budget. If the company won't provide developer effort to create and extend CI, it is unlikely to open the checkbook for unplanned expenses.
In this case, instead of buying, you might try renting.
Enter CI as a service
When you think about it, all CI does is check out code from version control and run some command-line utilities—one to build the software, another to perform the unit tests, another to create a virtual server, a fourth to run automated GUI tests against that server.
Each of these programs needs to report status back, in terms of abject operating-system failure (in which case, stop the testing), a pass, a fail, or, in the case of automated tooling, a large number of results.
The results need to be in a form that the CI server can consume. The tool might want to display a dashboard with system health, pass-fail results, the ability to drill down into details, historical performance, and so on.
As long as the code is available in a public repository and your team can transition the scripts, that sounds like an ideal task to run in the cloud. You can go plug-and-play with your build software—or, if you desire, stick to a vendor's pre-selected toolset—then pay a monthly fee, and you have CI as a service.
This can be as easy as CI in the cloud. Most of these modern as-a-service systems take basic cloud-based virtualization, then add a stack of tools on top to make it easy to do something specific.
For example, you might add a mobile back end, which includes all the services you need for a mobile site—including creating an account, logging in, sending messages, editing your profile, resetting passwords, seeing a directory of other users, and so on.
Beware the staffing issues
The problem, though, is the cloud setup isn't enough.
While CI as a service allows you to set up CI for a few dozen dollars a month, you still have a staffing problem: Someone has to actually configure the CI system to be effective.
If anything more than continuous build is new to the team, then setting up the system is going to be a learning experience. That means setbacks. Demos. Lessons learned. And, frankly, it means mistakes and unexpected complexity, which leads to delay.
This brings us back to the feature beast. If getting effective CI started is basically learning a new platform, such as moving from web to mobile development, then it might make sense to hire someone to get things started. The team could hire a build engineer, start a DevOps team, or, perhaps, hire someone to get the whole pipeline started.
Start effective CI as a product
When I first met Noah Sussman, he was the test team manager at Etsy, one of earliest adopters of continuous delivery. Etsy was able to succeed using a combination of feature flags, "dark" rollouts, continuous monitoring, the ability to patch and redeploy quickly, and the ability to deploy pages independently.
One core piece of the strategy was a build/deploy pipeline using Jenkins. Three years after I met Sussman, he was working as a consultant, coming into companies and setting up a CI pipeline. What impressed me about his effort was that it was tied to delivered value, not effort.
Whether it took him two weeks or two months, the real question was value for the customer. Sussman's bet was that he could reuse technologies across customers. The work would get easier for him to do, while the value was mostly based on the number of teams that were using the pipeline.
Sussman was building more than CI—what today we would call a build/deploy pipeline. His deliverables included CI, a delivery pipeline, configuration flags, and continuous monitoring, at time a time in history when only a handful of people could tie them all together.
Today, teams can set up CI pipelines as a productized service, in a week or two, for a fraction of what some consultants were charging five years ago. That setup can include understanding the build process, documenting it, moving it to a permanent server, setting up appropriate test tooling at the unit, API, and system level, defining a work process, running it long enough to build steam, training the team, and performing the handoff.
Build on the momentum
Once the team has effective CI in place, it will want to extend that to have the ability to deploy any build at the push of a button. Some teams will want a build that passes testing go to a test server automatically, while others may want a passing build to go all the way to production.
Integrating that with configuration flags, tracking requirements to deploys, and adding continuous visualized monitoring are all logical extensions the team may desire. Again, this needs to be balanced with the feature beast.
Large organizations might be able to create a central DevOps team, but smaller ones will find this expensive as a percentage of their development effort.
The time for DevOps as a service may be here, too.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
