When it comes to software delivery speed and stability, feature branches are an anti-pattern. Trunk-based development is the way to go. If that gives you pause—if you harbor fears or concerns about walking away from feature branches—that’s understandable. Let me allay your fears.
In the book Accelerate and in the State of DevOps report, Dr. Nicole Forsgren, Jez Humble, and Gene Kim have shown that consistently merging code to trunk multiple times a day delivers high performance in technology organizations. Meanwhile, long-lived feature branches negatively affect software delivery performance.
Feature branches encourage isolated work, which results in slow feedback. In large teams, merging code becomes an ordeal of conflict resolution, which can lead to unpredictable results and broken software. Ultimately, this affects reliability and slows down releases.
Trunk-based development, on the other hand, favors very fast feedback. Developers merge their code changes to trunk every day and avoid building up unreleased, untested, and unintegrated code for long periods of time.
Instead of using feature branches to represent your work in progress, put in place feature flags to dark deploy your code changes. Your new code will be deployed but not released to users until the functionality is ready. When a feature is ready for release to live users, you can use feature flags to reduce risk by gradually enabling the feature, possibly to cohorts of friendly beta-users, all the while monitoring the application (and with the option of stopping the rollout if something goes wrong).
The increased risk of isolated work
Feature branches are part of a widely adopted development process called Gitflow, in which you work on each feature in a separate branch created by one or multiple developers. Once you’ve completed the feature work, you merge the branch with the development branch, which will lead in turn to a release branch with multiple features to deploy.
This allows the developers to isolate their feature work from the common codebase, so as not to disrupt the rest of the team. For example, if a feature takes a long time to build, a team will keep it in a feature branch to avoid including unfinished functionality in the common branch.
Another pattern is to see a feature ready to be deployed, but not signed off on, in which case the team can choose to keep it unmerged until the next release. These uses seem legitimate. Unfortunately, they are the source of many dysfunctions that work against a fast-flowing development process.
A good development process enables two things for developers: You can make changes quickly, and you can easily manage the risk that those changes carry. Creating long-lived feature branches is an attempt to manage the risk of deploying incomplete features by keeping the code away from the common codebase until everything is ready. This comes at the expense of a good development experience, when the dreaded time of merge comes with the inevitable conflicts and bugs that result from it.
Why does this happen? As you work on several features in parallel in isolated branches, you deliberately create your own information silos. The longer you work on these silos, the further they will diverge. And when the time comes to merge these changes, you'll have to reconcile your conflicting understanding on all levels: business domain, application behavior, and architecture.
Tools may appear to be helping, and Git will smartly combine most code commits without conflicts. But there is no guarantee that the final result will reflect the intent of the individual changes. This may get caught in the automated pipeline of the common codebase, or otherwise make its way to production with nobody aware that a critical business logic flaw was introduced. The sneaky part is that nobody directly coded the bug—the merge created it!
Another perceived benefit of isolated branches is the ability to test changes ahead of time, as you're implementing them. But if every branch needs its dedicated test environments (integration, end-to-end, performance, etc.), this quickly gets challenging from an operational point of view. And with many diverging versions of the code at any given time, confidence in these tests will be poor. Ultimately, you'll need to repeat the tests after merging.
As merges become more painful, resulting in uncaught bugs, you might be tempted to merge less often and put rigid processes around the process. But that just starts a vicious circle of branches living longer and containing more changes, which makes everything harder and riskier. This fails both criteria for a good development process: It’s harder for developers to release changes, and the changes carry more and more risk.
Trunk-based development and feature flags
The alternative view from long-lived feature branches is that developers should be committing to the common codebase (i.e., trunk or master) at least once a day. Jez Humble and Dave Farley, the authors of Continuous Delivery, have been strong advocates of this. At DORA/Google Cloud, Nicole Forsgren has also led research that concluded that daily commits to trunk is an indicator of high performance.
The act of committing changes multiple times a day to a common branch encourages new behaviors. What was days or weeks of work developers now breaks down into smaller pieces, following the lean development approach of having small batches. Integration truly happens continuously for all changes to the application: There is only one version of the code at a given time, consistently tested and always deployable.
Does this mean all code must go live, even if the feature isn’t finished? Yes and no. Developers deploy all new code to production, yet the feature is not necessarily released until it is ready. This technique, known as a dark deployment, combined with feature flags, allows for features to be enabled without deploying new code.
Decoupling deployment from releases reduces risk, since it limits the number of variables to consider for what happens at deployment time. Teams can release new functionality on their own timeline, not right when new code goes live. Feature flags are also more than just an on/off switch; they allow for progressive rollouts.
Your team may choose to only start by enabling the feature to a test account, in order to safely test in production. You can then extend the rollout to a small percentage of users who opted in for preview features, encouraging early feedback. As you iron out issues, you gradually increase the rollout until all users have access to the new feature.
A feature flag can also act as a kill switch at any time during rollout, immediately disabling functionality across all users without deploying any code or configuration changes. This is a very useful risk management tool because it allows teams to quickly limit the impact of an unforeseen issue with a feature rollout.
Feature flags can, however, introduce complexity in both your code and testing strategy. For this reason, you should measure the number of flags and their lifespans and keep them reasonably low. Every flag should be considered as work in progress (WIP) and follow lean development principles that WIP must be limited to ensure a stable and predictable flow.
What about pull requests?
In a process that uses feature branches, developers commonly open pull requests to gather the approval of other team members for proposed changes. Code reviews have many benefits, from knowledge sharing to catching coding mistakes early on. But research shows that large change sets benefit very poorly from code reviews, or, as perfectly put by @IAmDeveloper:
If pull requests are a strict requirement for your process, using very short-lived branches (less than a day) will yield most of the benefits. By reducing the number of changes in your requests, reviewers have a much better chance of understanding what is changing and will get through approval more quickly.
Ideally, in a trunk-based development workflow, you can put other methods of reviewing code into place. A great way is to write code collaboratively, such as in pair-programming or mob-programming sessions. And since the changes already have had multiple eyes on them, they shouldn’t require a formal pull request to be merged.
How do I start?
If your team’s development process is based on long-lived feature branches, you've probably felt the pain of resolving large merge conflicts in the stressful days leading up to a release. Reducing the lifespan of branches is a great place to start. It will simplify code reviews, reduce the cognitive overload of large amounts of changes, and make conflicts less problematic as developers merge more frequently.
Next, as you break down features in several smaller code changes, you'll need to adopt a standard way of managing feature flags and rollouts. You might be tempted to build your own tool for this, since flags may appear as simple if-statements. Don’t do it. Feature flag tooling is not a differentiator for your company, and there are many tools out there already that you can plug in directly to your development process.
Yes, you'll face resistance to the idea of changing your branching strategy. But there's no better way to drive change than by showing its impact. When you start introducing trunk-based development to your team, keep measuring your key software delivery performance metrics (lead time, deployment frequency, change failure rate, and mean time to recovery) and you'll be able to demonstrate how daily commits to trunk are improving both speed and stability.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
