Decades ago, W. Edwards Deming deftly advocated that to improve business manufacturing processes, companies must select only the best suppliers and seek continuous improvement. He evangelized the benefits of eliminating the need for inspection on a mass basis by building quality into products.
Today, these same principles have manifested themselves inside every leading-edge DevOps practice on the planet—and that's no coincidence.
Today, businesses that are racing to deliver better value to their customers—and differentiate from competitors—are embracing Deming's principles within their open-source-based software development practices. As software has become the last path to differentiation in most competitive industries, practices are evolving, from artisan-based creations to those that more closely resemble high-velocity parts assembly.
As our team set out to develop the fifth annual State of the Software Supply Chain Report, we wanted to better understand the health and habits of the open-source component ecosystem. We also wanted to understand how software developers choose which open-source components to use in their own projects.
To do this, we partnered with Gene Kim from IT Revolution and Dr. Stephen Magill from MuseDev to objectively examine and document software release patterns and cybersecurity hygiene practices across 36,000 open-source software projects, 12,000 enterprise development teams, and 3.7 million open-source releases.
The results are incredibly revealing. Here's what we learned.
Velocity does not have to come at the cost of reduced security
This year's report empirically shows that velocity—and a focus on innovation—does not have to come at the cost of security. It’s important for developers to understand that secure coding practices can actually speed up their process and increase innovation.
To survive and thrive in today's application economy, the best development teams are actively embracing open-source innovation, dependency management practices, and automated tooling for open-source governance.
Exemplary open-source project initiatives benefit tremendously from higher code commit and release frequencies, and to some extent larger development teams. Excellent teams also do an outstanding job of managing their dependencies.
At the same time, the best enterprise development teams benefit from processes that support using the latest component versions while also embracing automated practices to reduce the presence of known vulnerabilities.
Going back to the practices Deming advised, these development teams are selecting the highest-quality components and ensuring that they are "building quality in" by actively selecting components without known security defects.
For organizations that tamed their supply chains, the rewards were impressive: Use of known vulnerable component releases was reduced by 55%.
Our study of over 65,000 applications also revealed that managed software supply chains are safer. We measured a 55% reduction in the use of vulnerable open-source components in companies that managed their open-source software supply chains.
Exemplary development organizations have open-source governance practices in place, track components using a software bill of materials, and often use the latest versions of component releases to reduce the presence of known security vulnerabilities.
Not all open-source software components are created equal
Choosing open-source projects should be considered an important strategic decision for enterprise development organizations. Open-source projects should be recognized as external suppliers of software whose goods should be evaluated regularly for quality and security.
You apply procurement practices and quality standards to physical assets consumed by the enterprise, and you should apply that same management rigor to external software providers as well.
To help apply evaluation criteria to open-source component suppliers, we studied five common behavioral patterns across 36,000 open-source projects. While different open-source software projects demonstrated healthy or poor performance that affected the overall quality of their releases, we identified attributes of large and small exemplars who rest within the top 3% —or 1,229—open-source project development behaviors.
To arrive at this list we examined a large number of variables including:
Do differences exist in how effectively open-source projects update their dependencies and fix vulnerabilities?
Are there exemplary teams that do this better than others?
Are components from exemplary teams more widely used than "non-exemplary" components?
What factors correlate with exemplary components?
What advice can be offered to producers of open-source components and the developers who consume them?
The resulting data was illuminating, and came with a clear answer: You can't look at all open source equally. While we identified small exemplars and large exemplars in the report, we also identified three additional groups of open-source projects: those that were laggards, those that put features first, and those that were cautious. Therefore, commercial teams should take into account what type of open-source project components they’re about to use.
Exemplary development teams differentiate through these six performance metrics.
It shouldn't be a popularity contest
It has long been believed that in the realm of open-source development, "with more eyeballs, all bugs are shallow." In other words, the more popular open-source components are, the fewer bugs or vulnerabilities they should have.
Our research into the popularity of components shows that developers should not select components based on popularity alone. While many of the best-quality open-source components are popular, not all popular open-source projects release improvements or remediate vulnerabilities frequently.
Some of the most popular open-source components may take 200 days or more, on average, to release new updates or to remediate known security vulnerabilities.
When selecting suppliers, Deming advised manufacturers to study their performance track records. How fast would new versions of parts be created, how quickly could they be delivered, and how fast would defects in those parts be remedied? He did not advise resting decisions upon the opinions of the best-known suppliers, but asked manufacturers to select suppliers based on empirical evidence of their performance, quality, and service.
While popularity tends to trend up as release speed increases (meaning exemplars are often popular), many projects with poor hygiene are popular as well. You should not choose projects based on popularity alone.
Developers need to understand MTTU and MTTR
Median time to update (MTTU) should be used as a critical metric when deciding which components to use in your software supply chain. A rapid MTTU is associated with lower security risk and is also more accessible from public sources than other metrics you might want to rely on, such as vulnerability data.
Just as traditional manufacturing supply chains intentionally select parts from approved suppliers and rely upon formalized procurement practices, enterprise development teams should adopt similar criteria for their selection of open-source components.
This practice ensures that the highest-quality parts are selected from the best and fewest suppliers. Implementing selection criteria and update practices will not only improve quality, but it can also accelerate mean time to repair when suppliers discover new defects or vulnerabilities.
When comparing MTTR with MTTU for non-security-relevant updates on a per-component basis, the study found a correlation between update behavior for security-relevant updates (MTTR) and non-security-relevant updates.
Median time to remediate (MTTR) goes hand in hand with MTTU. Exemplar components demonstrated MTTR that was 3.4 times faster, and they were 27% more likely to already be protected when new vulnerabilities were discovered.
Teams using the latest component releases can reduce the presence of vulnerable component releases in their applications by 55%. Therefore, development teams that simultaneously incorporate the latest versions of component releases and procure them from exemplary open-source projects can improve the overall quality of their applications.
Outside influences will shape software development
Developers should understand the outside influences that are taking shape. Industry groups are introducing secure development standards surrounding the use of open-source component-based development (e.g., tracking and tracing the use of open-source components and creating a software bill of materials).
At the same time, government agencies are racing to determine how to use new policies and legislation to protect citizens, businesses, and critical infrastructure. Government agencies recognize the growing importance of managing software supply chains. These practices aren’t mandatory now, but no matter what industry you’re working for, it’s likely that secure coding practices are coming, whether you like it or not.
The results of the 2019 Software Supply Chain Report opened our eyes to the true behaviors of developers when it comes to habits and health within the open-source software ecosystem.
But most importantly, we learned that exemplary software development practices that deliver high-quality and improved security at high velocity are not rare. These types of practices are present today in large numbers, and their practices serve as benchmarks for others to strive for and achieve.
As Deming stated as one of his 14 principles, it's vital to institute a vigorous program of education and self-improvement. We now have that roadmap for software developers built out, enabling them to learn, ultimately helping show people that creating safer software is indeed possible.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
