Ever hard-coded or published a private key on your version-control system? I have. Publicly, on GitHub, no less. It's a common rookie mistake, not usually made by slightly experienced coders. However, there are scores of other ways that engineering teams are exposing their secrets (things such as API keys, SSH-keys, passwords, OAuth 2, etc.) to hackers.
Managing secrets is becoming more challenging as more companies start building microservices and using containers and cloud services more frequently. Every service requires an API key or credentials, so more software is passing credentials through a litany of services.
These articles and tools will equip you with the knowledge and ability to tackle the security challenges of managing secrets for cloud-native software.
Choosing your secrets-management strategy
Introduction to cloud-native secrets management
This three-part series gives the ultimate introduction to secrets management. George Wainblat provides technical descriptions without mentioning any specific tools or ecosystems (except Kubernetes and Docker). He explains the challenges of managing secrets in both legacy IT and cloud-native environments.
You'll learn which kinds of secrets are commonly used in classic IT versus cloud-native IT. By the end of the series, you'll understand several current practices for handling secrets, along with their pitfalls. Here are the links to Part 2, "Protecting Secrets in Use," and Part 3, "The Importance of Container Identity."
"Physical solutions like HSMs (Hardware Security Modules) and TPMs (Trusted Platform Modules) have no architectural fit in a software-defined environment. ... [In a cloud-native environment] there is an orders-of-magnitude scale increase in the number of secrets and in the interconnectivity required for them to be used / accessible during setup."
—George Wainblat
Turtles All the Way Down: Storing Secrets in the Cloud and the Data Center
Daniel Somerfield's 2015 presentation about secrets management has held up well as a deep-dive introduction to the challenges and various approaches for secrets management, along with the pros and cons of each. Somerfield also has a GitHub page full of supplementary resources for this talk. It also includes the slides (PDF).
Secret Management Design Decisions: Theory Plus an Example
This article goes deeper into the nuts and bolts of secrets management. The author, Sander Knape, looks more in-depth at the trade-offs and includes a real-world example. The specific trade-offs Knape explores include:
- Using placeholder values or encrypted values in your applications
- Adding this logic at bootstrap time or runtime
Secret Management Architectures: Finding the balance between security and complexity
Stenio Ferreira's article adds another important angle to this list: The complexity created when we build a secrets management system.
Ferreira's diagram above illustrates how the complexity of an architecture increases when you add more bells and whistles to its security systems. While it's important to minimize complexity, it's also okay to take on more complexity as your team adapts to it. Ferreira proposes a helpful list of three evolutionary stages toward comprehensive secrets management.
This gives readers a roadmap that their teams can use to get more comfortable with secrets management before trying to implement the perfect solution. Each stage adds a new open-source tool to add capabilities to the system. In Ferreira's examples, these tools include Git-crypt, Chef Vault, and Hashicorp Vault.
Best practices
Managing secrets at scale
In this talk, Alex Schoof outlines the core principles of modern secrets management, explains how these systems work, and mentions a few open-source secrets management tools.
Here are slides from a previous iteration of the same talk.
Key Management Cheat Sheet
You can usually find good advice for most security topics if you browse through the resources at the Open Web Application Security Project (OWASP). For secrets management, there's a robust cheat sheet that explains different cryptographic and key management algorithms in the first section. The second section is more aligned with the focus of this article: best practices for the key management lifecycle. It covers key generation, distribution, storage, escrow, backup, audit, and recovery.
How to keep a secret: Is your database password stored safely?
While this article is a little old and aimed at a PHP audience, its warnings are still relevant to developers today outside of the PHP ecosystem. It summarizes the good arguments laid out in the article "Environment Variables Considered Harmful for Your Secrets," and points readers to the comments on that article for a balanced view of the environment variables debate.
Intro to basic solutions
An Introduction to Managing Secrets Safely with Version Control Systems
What do you do if you've already published sensitive data in your version-control repository, or you want to check if that has ever happened? Justin Ellingwood shows you how to use Git repositories for any secret exposures and provides a few mitigation options if you have published sensitive data. He also explains how to use version-control features so that you can avoid publishing secrets in the future.
The second half of Ellingwood's post explores three options for secrets management in great detail. The three options are:
- Store encrypted secrets in a repository.
- Use the features in your configuration management system for handling secrets.
- Use an external secrets management service.
He also explores the pros and cons of each option.
Getting Secrets Out of Source Code
Could there be exposed or vulnerable secrets sitting in your source code right now? Karthik Thotta Ganesh, a security researcher at Adobe, knows it can be a challenge to find all of the secrets potentially hiding in your source code, so he and his team came up with an approach to scan their source code for secrets. This article explains how they used Gitrob and Python scripts to locate insecure secrets. Ganesh recommends that you also use truffleHog or git-secrets with Gitrob.
Cloud Security Best Practices: Finding, Securing, and Managing Secrets
Tom McLaughlin's two-part series begins by discussing how secrets leak and showing you how to find secrets embedded in your code, with examples using truffleHog and git-secrets. Part two shows three quicker, less complex solutions for secrets management that aren't full-blown external tools. The examples cover git-crypt, Amazon Web Services S3, and utilities in infrastructure-as-code (IaC) tools such as Chef, Ansible, and Puppet.
Node.js configuration and secrets management
Sergey Chikin takes a simplified approach to secrets management in his Node.js environment, since he felt that external services, IaaC tools, and consensus systems (ZooKeeper, etcd, Consul, etc.) were overkill. Whether you agree with his method or not, his is another interesting approach to secrets management for you to consider. He also shares some reasons why it's better to have developers manage secrets because ops doesn't have enough insight into an application's internal architecture.
"Most of the time, it's Ops' duty to manage the configuration, but this adds unnecessary overhead and slows down the debugging process. More often, it's developers who define the configuration options and thus they're the ones who know the proper configuration values."
—Sergey Chikin
One Line of Code that Compromises Your Server
Jack Singleton writes on the MartinFowler.com blog explaining how dangerous it is to create weak session keys during development and to then not strengthen them in production. Session keys are used for signing and/or encrypting cookies set by the application to maintain session state.
While this article looks at just one type of secret—which can be secured with a simple password management utility—it also shows you a bunch of example code that could be used to crack a weak session secret, take control of an application, and then take control of its server.
The problem with DevOps secret-management patterns
Cem Paya delves into the secrets management strategies used by each of the major IaC tools: Puppet, Ansible, and Chef. He discusses the weaknesses of Ansible's and Chef's symmetric cryptography, while also explaining why Puppet's hiera-eyaml has issues as well. He also lays out a method for improving Chef's encrypted data-bags. Overall, the article gives a good deep dive into the caveats for each of these tools, but you might also want to explore the tools yourself since the community may have implemented improvements to these issues since Paya's article was published.
Cloud and container solutions
How to keep your container secrets secure
Regardless of the tooling and orchestration you're using, Liz Rice says these four actions can help you keep your secrets secure in a containerized environment:
- Don't build secrets into the container image.
- Use volume mounts to pass secrets to a container at runtime.
- Have a plan for rotating secrets.
- Make sure your secrets are encrypted.
The article goes into detail on how to perform each of these actions.
Container Secrets Management
Aqua, a container security company, created a list of introductory articles for secrets management in container-based environments. The article covers secrets management for a wide array of ecosystems, including Docker, AWS, Kubernetes, Rancher, and OpenShift.
Kubernetes ConfigMaps and Secrets
Now that you understand the strategies for secrets management with Docker containers from that last resource, the next step is understanding how you can manage secrets when you orchestrate those containers with Kubernetes. In addition to the Kubernetes documentation, you can try out Sandeep Dinesh's Kubernetes ConfigMaps and Secrets tutorial, which takes an app with hard-coded configuration and modifies it to read from environment variables. Then he sets up Kubernetes to manage the configuration for you. In Part 2, he uses a configuration file instead of environment variables. Note that some other authors in this list don't recommend using Kubernetes to do the heavy lifting when it comes to secrets management.
Your Secrets in Cloud-Based Key Management Services
In this video presentation, Dan O'Boyle shows you how to use a cloud-based key management service (KMS). This allows users to centralize secrets along with an access control list for them. A KMS also helps users automate key rotation, which means it is often more secure than local key management practices. KMSes are essentially encryption keys as a service.
O'Boyle explores these benefits with real-world examples using Google KMS, but the tips also apply to Azure Key Vault and Amazon KMS.
Tool categories and analysis
Infrastructure Secret Management Software Overview
With all of the emerging tools to manage secrets for automated, cloud-native infrastructures, Max Timchenko decided to create an informational list with concise reviews for each of the tools in this category. The information is open to community review on GitHub and based on Timchenko's review of each tool's documentation.
The list includes:
- Ansible Vault
- Barbican
- Chef Data Bags
- Chef Vault
- Citadel
- Confidant
- Configuration Storage Systems (Consul, etcd, ZooKeeper)
- Conjur
- Crypt
- EJSON
- Keywhiz
- Knox
- Red October
- Trousseau
- Vault (HashiCorp)
The list leaves out a few options that bloggers in this list also mentioned:
Secrets and LIE-abilities: The State of Modern Secret Management
This deeper review of fewer secrets management tools by Jeff Nickoloff has a nice bulleted list of the current best practices in secrets management to kick off the article. It analyzes the secrets management capabilities of Keywhiz, Vault, Docker and SwarmKit, Mesosphere DC/OS, Rancher, and Kubernetes.
Secure Strategies For Managing Passwords, API Keys, and Other Secrets
Dane Schneider provides a useful, concise list of several strategies for secrets management, along with pros and cons, and a few examples of tools that fall under each category. You should also check out his list of common anti-patterns in secrets management.
Choosing the right secret management tools and strategies
Picking the right strategy and tools for managing secrets in your software ultimately comes down to understanding your threat model and trust boundaries. One size does not fit all.
Time, budget, team knowledge, the architecture of your apps, and several other factors will be important to consider when choosing the right path.
If you have any good advice or additional resources related to secrets management, please share them in the comments below.
Keep learning
Get up to speed on Zero Trust security with TechBeacon's Guide.
Understand why API security needs access management with this Webinar.
Learn how how privilege and policy management improves your cyber resiliency in this Webinar.
Find out why Zero Trust means rethinking your security approach.
Answer this question: Is your environment adaptive enough for Zero Trust? Get this free white paper.
