As companies increasingly adopt agile development methods, many are looking for ways to improve their application security. One of the first questions they must address is how to measure progress, experts say.
For every application-security program, there are two tracks of the problem that companies need to pursue: How to measure risk in a way that informs action, and how to use metrics to train the development staff in ways that prevent the creation of new vulnerabilities.
Here are five metrics that every company that produces software should track for better security.
1. Portion of apps covered by security
The first metric to suss out is the percentage of applications that are part of the secure-development lifecycle, said Pete Chestna, director of developer engagement at application-security firm Veracode.
Companies should start with their most critical and exposed applications but then move on to finding every application, no matter how old or seemingly insignificant. Oftentimes, companies lose track of legacy applications and forget to apply new methodologies to the old software.
"This is very much a discovery process. Talk to your operations team. So it is a long process to get through."
—Pete Chestna
He said it was best to start and not worry about having a complete list. "You have at least a partial list, and that is a good place to start."
2. Time needed to resolve vulnerabilities
Companies should also measure their average time to resolve vulnerabilities. Because eliminating the creation of vulnerabilities is likely unachievable, the development team's agility in dealing with flaws is a critical measure of the security of the application, said Zane Lackey, co-founder and chief security officer for application-management and security firm Signal Sciences.
"In any development methodology, you are always going to have vulnerabilities, so how quickly you can resolve those issues is extremely importan. You really want to see that metric go down over time."
—Zane Lackey
3. Flaw creation rate
While flaws are unavoidable, measuring the rate at which vulnerabilities are created is still important. In particular, compare the flaw-creation rate and the development team's average time to fix vulnerabilities, said Veracode's Chestna.
"If my fix rate is higher than my new-find rate, then we are going in the right direction. Yet, if it's the other way around and I'm producing more flaws than I'm fixing, then you are in trouble."
—Pete Chestna
Companies should also consider measuring the rates for different classes of vulnerabilities, such as critical, high, and low/medium, at the very least, he said.
4. Number of automated tests and tooling
Another good metric extends the first measurement of the number of applications covered by security technologies. Companies should also measure what sort of coverage and what sorts of tests each codebase is subjected to, Signal Science's Lackey said.
"Ideally, what you are looking for here is to answer whether the investments you have made in security tooling are actually helping or even being used. Are they always solved manually by hand? Because that is the only effective means."
—Zane Lackey
By putting in more automation and a tighter feedback loop, most development teams will respond more quickly to vulnerabilities and make mistakes less often, he added.
5. Application block rate
Many companies patch development problems by instead blaming—and blocking—the application, or at least a code library. Figuring out the number of applications, code libraries, and open-source components that are disallowed to developers can be a good measure of vulnerability of the development process, Lackey said—not the least because saying no too often to developers can make them less likely to listen the next time.
"You want to be very aware of when you do block things. That is real political capital that you are spending with the organization, so you only want to do it when you know that is the right decision."
—Lackey
Other security metrics that matter
The five metrics highlighted above are not, of course, the only ones. A more amorphous measurement, for example, is the number of teams that come to security proactively, which can indicate the level of engagement the developers have with security issues.
But how to scope that can be challenging. Starting out, most companies have no idea what the risk level is for their software. And even those that are chasing vulnerabilities have the tail wagging the dog—it's a very inefficient way to secure software.
Another way to think of it is as a way to pay down existing security debt and avoid incurring new debt, Chestna said.
"Companies beat themselves up about their security, but they didn't get there overnight, and they are not going to fix it overnight. Knowing how big your risk is is critical to starting on paying down your security debt."
—Chestna
Metrics to be wary of
We are an increasingly data-driven society, and measuring the progress of security programs is no different.
Companies embarking on their first foray into collecting metrics on application security need to think through what they intend to measure, said Signal Sciences' Lackey. People tend to optimize their behavior to metrics, so if a company chooses the wrong ones, it will provide incentives for the wrong behaviors.
Measuring the number of newly discovered vulnerabilities, for example, may be a reasonable metric, but it's one that needs context, he said. A large number of new bugs could mean that the development process is catching previously missed issues—a good trend—or it could mean that developers are not paying attention to security—obviously, a bad trend.
"Metrics are a powerful tool, but there are risks in the metrics themselves. You have to make sure that you are finding the best metrics for your organization."
—Lackey
It's important to keep the end goal in mind. You want to get to the point where you are at least preventing vulnerabilities from impacting your software's security, Chestna said. "And, if you get to the point where you are writing secure code to start with, that's the ultimate goal."
Image credit: Flickr
Keep learning
The future is security as code. Find out how DevSecOps gets you there with TechBeacon's Guide. Plus: See the SANS DevSecOps survey report for key insights for practitioners.
Get up to speed fast on the state of app sec testing with TechBeacon's Guide. Plus: Get Gartner's 2021 Magic Quadrant for AST.
Get a handle on the app sec tools landscape with TechBeacon's Guide to Application Security Tools 2021.
Download the free The Forrester Wave for Static Application Security Testing. Plus: Learn how a SAST-DAST combo can boost your security in this Webinar.
Understand the five reasons why API security needs access management.
Learn how to build an app sec strategy for the next decade, and spend a day in the life of an application security developer.
Build a modern app sec foundation with TechBeacon's Guide.
