Kubernetes has come of age. Enterprise organizations across nearly all verticals, including those with strong security requirements, such as financial services, healthcare, government, and telecommunications, are deploying production applications to Kubernetes clusters.
Workloads running on Kubernetes range from serverless, microservices, and stateless applications to stateful applications with back-end databases running on or off a cluster. The project has now reached a state of maturity where there's a need to focus on how to secure Kubernetes deployments, and the community has recognized this by investing in a recently announced security audit.
The Cloud Native Computing Foundation (CNCF) has open-sourced the results of a third-party security audit of the Kubernetes project. Here are the findings—and what you can do to protect your organization's production workloads in Kubernetes.
Findings: 37 issues
Trail of Bits, one of the companies performing the audit, produced a final report, a threat model, and a white paper with recommendations for protecting Kubernetes clusters. The final report identifies 37 issues, of which five are high-severity, 17 medium-severity, eight low-severity, and seven "informational severity" issues.
Although the final report uses the term "vulnerabilities," the majority of these issues are not vulnerabilities that resulted in assigned common vulnerabilities and exposures (CVEs). Instead, most are opportunities for security hardening and enhancements, although some general issues might lead to CVEs in specific instances.
So, what actions should Kubernetes admins and developers take? The white paper includes recommendations for both. While there's space to discuss only a handful of these here, anyone who has production applications running on Kubernetes and cares about security should read the 22-page white paper.
Recommendations for cluster admins
Let's start with the most important part of the cluster to protect: the control plane. Most importantly within the control plane, the Kube API server and the etcd key-value store, which holds information about the expected state of workloads deployed on the cluster, need to be secured.
Here, basic security principles can and should be applied. These include:
- Apply the principle of least privilege by using the authentication and role-based access control capabilities.
- Encrypt all traffic to the API server.
- Encrypt the etcd data store.
Of note, the ability to encrypt secret data at rest requires Kubernetes 1.13 or later and etcd v3 or later, which brings up another security best practice: It's important to stay up to date with the most recent, stable release of Kubernetes for production deployments.
This doesn't mean always being on the leading edge—enterprise organizations can’t always move as quickly as the upstream community. But it is important to follow the upstream developments and regularly assess when a new release has a key feature or vulnerability fix.
Pay attention to networking security
Next, the audit found networking security to be weak on several fronts. There are many developments in this area that can be leveraged by the cluster admin. These include:
Network policies
Kubernetes Network Policies improve the way pod isolation policies are defined, and provide fine-grained configuration of allowable traffic between pods in different projects. They are not bidirectional and apply only to ingress traffic of pods, providing a way to define fine-grained traffic isolation policies between containers in different pods and projects.
Ingress and egress security
To better secure egress and ingress traffic for services deployed on Kubernetes, the cluster administrator should allocate a number of static IP addresses to a specific node at the host level. Admins can then configure an egress IP for each project to validate that traffic from the pod originates from the host using the pre-allocated static IP.
This allows administrators to improve the security of an external database so that only specific pods can talk to a service (the egress router), which proxies the traffic to the database.
Istio for Kubernetes
The Istio service mesh injects a container that runs as a sidecar proxy (in this case, Envoy Proxy) and forces all traffic that ingresses or egresses a pod to go through that proxy.
Istio connects individual instances of Envoy throughout a cluster using mutual transport-level security (TLS) certificates. This ensures that nothing gets in or out of the cluster without Envoy seeing it and that no traffic goes between two pods without being encrypted. It also helps to ensure that the user knows the remote endpoint they are talking to is the right one.
Multus CNI
This "meta-plugin" makes it possible to create multiple network interfaces per pod and assign a CNI to each interface created. This supports a number of use cases, but the most interesting one is the ability to separate the control plane from the data plane—another good security practice.
In this configuration, the API server connects with the pod on one network interface, and any other pod communication is managed on a separate network interface.
Protect the host systems
Next, consider how to protect Kubernetes hosts with node-host configurations and permissions.
SELinux
Linux containers share a container host and kernel. Kernel technologies such as cgroups, SELinux, and kernel namespaces provide isolation so that one container cannot access or impact other containers running on the same host or on the host itself.
SELinux provides fine-grained, mandatory access control for every process, file, and user on the Linux host and has mitigated more than one container runtime vulnerability, including CVE-2016-9962 and CVE-2019-5736.
Pod security policies
Pod security policies (PSPs) provide a cluster-level way to manage container security. They make it easier for the cluster admin to manage the use of privileged containers, host namespaces, host networking and ports, user and group IDs and SELinux profiles.
Although PSPs are still in beta in Kubernetes, the protection offered is incredibly valuable.
Secure computing mode (seccomp)
This is used to restrict the set of system calls that applications can make. It is a feature of the Linux kernel and must be enabled on the host. Once enabled, admins can use PSPs to apply seccomp profiles to pods.
Logging and auditing
This is a security best practice, but one that shouldn't be overlooked. Admins should be sure to leverage the audit capabilities built into their Linux OS, such as auditd, and add a logging stack to their Kubernetes cluster to capture all events on the cluster.
Logs should then be forwarded to an external SIEM to protect against log tampering and for analysis and alerting.
Be proactive with Kubernetes security
The security audit is good news in many ways. In completing this audit, CNCF and the Kubernetes community are reinforcing the importance of not only being reactive, but proactive in keeping Kubernetes as secure as possible. And by open-sourcing the results, the community is modeling the value of openness, and making it possible for the community to collaboratively respond to the findings.
Contributors can evaluate how they can help to improve the code, and consumers have good visibility into possible areas of concern, making it easier to determine how best to protect their deployments.
The findings call out the capabilities in Kubernetes that provide strong security, and clearly identify opportunities for improvement. When it comes to areas for improvement, it's important to note that a number of those areas can be addressed with additional configuration or tooling.
Over time, the default security posture of Kubernetes will improve, but organizations should not be afraid—there are many ways to secure Kubernetes clusters today.
Keep learning
Choose the right ESM tool for your needs. Get up to speed with the our Buyer's Guide to Enterprise Service Management Tools
What will the next generation of enterprise service management tools look like? TechBeacon's Guide to Optimizing Enterprise Service Management offers the insights.
Discover more about IT Operations Monitoring with TechBeacon's Guide.
What's the best way to get your robotic process automation project off the ground? Find out how to choose the right tools—and the right project.
Ready to advance up the IT career ladder? TechBeacon's Careers Topic Center provides expert advice you need to prepare for your next move.
