How do you test hundreds of millions of lines of code or tens of billions of transistors? It’s very hard. But it's even harder when you're implementing cryptography.
Today's cryptography isn’t easy to understand. It often does complicated calculations on multi-thousand-bit values. And it often uses random numbers for some of those calculations.
When you generate cryptographic keys, for example, you need to generate lots of random (actually pseudorandom) bits. If a key doesn't look just like random bits, it's possible for an attacker to guess it more easily. That means that your cryptography isn't providing as much security as it could, and that's bad.
Here's why it's so complicated, plus some tips to simplify testing cryptography-enabled software.
Known vs. random values
If you're testing hardware that implements cryptography, using random values makes testing more difficult. Testing something complicated is hard, and it's made even harder if you can’t test it with lots of known input-output pairs. If your inputs change every time you do a test, as they would if you were using random numbers, testing may be close to impossible.
To work around this issue, most, perhaps all, hardware that implements cryptography has a way to bypass the output of its random number generator and use known values in testing.
Such undocumented features are surprisingly common in hardware. From May 1996 to August 1998, Dr. Dobb's Journal even had the "Undocumented Corner," a column that described some of these.
More recent examples include the rosenbridge backdoor, in which undocumented features of a microprocessor could let any application get root access for an operating system. Another high-profile issue is the use of Intel’s Visualization of Internal Signals Architecture (VISA) to monitor what other chips are doing, essentially giving a hacker a logic analyzer that he can use to probe the operation of other chips.
Such undocumented features, which you might call a "backdoor," are probably present in all hardware that implements cryptography that uses random numbers. That's essentially all of it.
It's possible to make hardware in which you can disconnect or disable the way to bypass the use of random numbers, but a clever hacker might be able to find a way to re-enable the backdoor. Or it might not get disabled before it's shipped.
This is a trade-off that may be impossible to avoid: If you want to test your hardware, you may also need to introduce a backdoor in it, and that means that it might not be as secure as you'd like it to be.
[ Special Coverage: SecureGuild Security Testing Conference 2019 ]
Things are getting worse
Suppose that you could travel through time, perhaps as H.G. Wells described in The Time Machine. If we could jump ahead to the 22nd century and talk to the engineers who work with the technologies that are popular then, it's very likely that they would be familiar with how we test hardware and software today.
Unfortunately, this would probably be because our ability to test technology hasn't kept up with our ability to build technology, and lots of our failures will probably end up as case studies in 22nd-century textbooks.
Back in the dot-com era, the company that I worked at had about one testing engineer for every three development engineers. Today, the ratio is more like one testing engineer for each development engineer. Testing has definitely gotten harder and more expensive over the past 20 years, and it doesn't look like it’s going to get easier any time soon.
Over the past few decades, the number of lines of code that are used in software has increased dramatically. When I left graduate school and got my first job in the technology industry, there were tales of legendary applications out there that used 1 million lines of code.
These might have been apocryphal. I never met anyone who worked on or with any of these monsters. It might have been the case that 1 million lines of code was as mythical as tombstones labeled "rex quondam rexque futurus,"("the once and future king"), and was just meant to scare new engineers.
The million-plus club
Today, an application with a million lines of code really isn’t that exciting. About 12 years ago, when we did our first security source-code review at a startup that I used to work for, I was somewhat surprised to learn that the source code for just one of the third-party libraries that we used to parse XML comprised over 1 million lines.
It's easy to find cases where hundreds of millions of lines of code are used today. At CES 2016, Ford Motor Co. noted that over 150 million lines of code now go into its F-150 pickup truck. Higher-end cars probably have even more, almost certainly over 200 million lines. It's just not a big deal to have millions of lines of code these days.
Hardware is getting just as complex. Before the dot-com boom, I recall being impressed with how Intel was able to manufacture a Pentium processor comprising 3.1 million transistors.
Today, integrated circuits comprising tens of billions of transistors are made. If you think about that, that’s quite amazing: Tens of billions of components, all of which work roughly like they're supposed to. It's the same sort of sufficiently advanced technology that science fiction author Arthur C. Clarke told us could be indistinguishable from magic.
Why testing software is easier
In some ways, software is easier to test than hardware is. Compiling code in debug mode makes it much easier to test, and it's relatively easy to compile in release mode once you're done debugging.
But it's still very hard to ensure that encryption is implemented correctly. Intermediate results and outputs of cryptographic algorithms often look like hundreds, or even thousands, of random bits, which makes debugging code that implements them tricky.
You never get something like "This String" as an intermediate result in cryptographic algorithms. Instead, you get something more like:
"0x7649abac8119b246cee98e9b12e9197d8964e0b149c10b7b682e6e39aaeb731c."
Ugh.
[ More from Luther Martin: Is the newest quantum breakthrough an encryption killer? ]
Ways to simplify testing
Fortunately, there’s an easy solution to this problem: Don't write your own cryptographic software! If you take a college class these days that's roughly Crypto 101, you'll be taught how cryptographic algorithms work and why they're secure. And then you'll be told to never try this on your own because the chances of doing something wrong are simply too great.
Instead, use an existing cryptographic library that's been validated to a standard such as the US government’s FIPS 140-2, "Security Requirements for Cryptographic Modules." This doesn't have to be hard or expensive. A version of OpenSSL has received FIPS 140-2 validation, for example.
Unfortunately, lots of people don’t get to that Crypto 101 class and end up trying to implement their own version of cryptographic algorithms. This is very hard to do right. Trying to do it is really not a good idea.
So beware hardware that implements cryptography because testability might have put a backdoor in it. And never, never, never implement cryptography when there's a known-good solution that you can use instead.
Keep learning
The future is security as code. Find out how DevSecOps gets you there with TechBeacon's Guide. Plus: See the SANS DevSecOps survey report for key insights for practitioners.
Get up to speed fast on the state of app sec testing with TechBeacon's Guide. Plus: Get Gartner's 2021 Magic Quadrant for AST.
Get a handle on the app sec tools landscape with TechBeacon's Guide to Application Security Tools 2021.
Download the free The Forrester Wave for Static Application Security Testing. Plus: Learn how a SAST-DAST combo can boost your security in this Webinar.
Understand the five reasons why API security needs access management.
Learn how to build an app sec strategy for the next decade, and spend a day in the life of an application security developer.
Build a modern app sec foundation with TechBeacon's Guide.
