When you want to know what the temperature is outside, you have two ways to do it. You can lick the tip of your finger, stick it out a window, and feel the temperature; or you can look at a thermometer placed outside your window and get an exact reading.
That analogy applies to quantifying application security risk. Here's why feelings have no place in application security, and how my new application security health calculation can provide a number that security teams can understand and take action on.
More than a feeling
Many organizations rely on a measurement called "defect density" to determine the security risk that an application's source code poses to the enterprise. Examples of its importance may be found all across the Internet.
The number is derived from static analysis and security testing (SAST) tools, which scan the source for security vulnerabilities after the code is committed to a repository. Defect density is calculated by finding how many high-severity defects exist in 10,000 lines of code.
If thresholds are wrapped around that number, then any value above 1.0 should get immediate attention, and anything under 1.0 is acceptable.
I've long been suspicious of the value of defect density as a measurement. After understanding how its value is calculated, I don't believe it provides an accurate picture of an application's overall risk.
It's like licking your finger and sticking it out the window. You have a feeling about the security of the code, but feelings have no place in application security.
Inaccuracies and avoidance
The calculation of defect density focuses on the high number of vulnerabilities that are found by SAST tooling. But what about software composition analysis tools, dynamic analysis tools, or container security analysis tools?
Or, even more important, what about low- and medium-severity vulnerabilities? Not including vulnerability information from these other pipeline security controls results in an inaccurate measurement of an application's security posture.
Because defect density depends on the number of lines of code in an application, the value can be artificially decreased by adding lines of fake code or comments. In other words, if an application has a density of anything over 1.0, developers could artificially deflate the value to be less than 1.0 by adding unused code, comments, or variables that are never used.
If you take 20,000 lines of code in an application and add another 20,000, your defect density will be significantly reduced and therefore your application will avoid the scrutiny of the security organization.
An application security health metric
I've thought long and hard about how to figure out a way to get a one-number answer that could indicate application health, a number that everyone can understand and take action on: an application security health (ASH) score.
Over the last few months, I have been working on an application security health algorithm that uses data attributes of a Common Vulnerability Enumeration (CVE). This algorithm takes into consideration not only the severity of the vulnerability, but the ease of compromise, and the likelihood that it will actually be exploited.
The result of the calculation is a simple number between 1 and 1,000. This number represents the application's security health in a similar manner that a credit score represents the risk of an individual's ability to pay back a loan to a bank.
Measurements need to be accurate, relevant, and current, which is why I created the ASH calculation. As a community, I hope we can vet the calculations, improve on them, and work toward a true measure of security health.
I'll share the source code of my ASH calculator to use on your own applications at All Day DevOps next month, and I will be online that day in the All Day DevOps Slack channel answering questions about the calculation—and getting community input.
Join me for my session, "Perception vs. Precision and Why Defect Density Sucks," on November 6, 2019, at the All Day DevOps online conference. I'll be digging deeper into the calculations of defect density and its weaknesses. Registration is free.
Keep learning
The future is security as code. Find out how DevSecOps gets you there with TechBeacon's Guide. Plus: See the SANS DevSecOps survey report for key insights for practitioners.
Get up to speed fast on the state of app sec testing with TechBeacon's Guide. Plus: Get Gartner's 2021 Magic Quadrant for AST.
Get a handle on the app sec tools landscape with TechBeacon's Guide to Application Security Tools 2021.
Download the free The Forrester Wave for Static Application Security Testing. Plus: Learn how a SAST-DAST combo can boost your security in this Webinar.
Understand the five reasons why API security needs access management.
Learn how to build an app sec strategy for the next decade, and spend a day in the life of an application security developer.
Build a modern app sec foundation with TechBeacon's Guide.
