Chances are, if an organization is looking to implement unmodified, open-source Kubernetes, it's viewing it as a way to save money. That may not be the only reason to embrace vanilla Kubernetes, but it certainly is a driving factor for many.
I get it. I've been there.
Prior to my current position at Red Hat, I was a member of an enterprise team that attempted to build a platform from scratch—based on the Kubernetes project code only. It started with a company-wide initiative to improve resilience and efficiency—and, from that, a push to modernize our infrastructure and move to the cloud. After researching different options, we eventually decided to build a platform based on Kubernetes because it was the most cloud-agnostic.
While I never expected building the platform to be a simple undertaking, I also did not fully expect it to come with the challenges or expenses that resulted. We quickly learned during development that we had a lot to learn beyond just Kubernetes—despite us being a team of fairly seasoned IT professionals. Not only did it take three teams more than a year to develop the platform, but also maintenance was an ongoing battle that required serious expertise and manpower.
While building a platform based on open-source ("vanilla") Kubernetes is certainly possible, in my experience it is not for the faint of heart. What we inevitably found is that, in our case, the pros did not outweigh the cons. We had anticipated some of the challenges, but others took us by surprise.
Here's a look at five pitfalls we discovered as we built an enterprise-wide Kubernetes platform.
Choice Paralysis
To implement Kubernetes in production, you need a lot more than just Kubernetes. Organizations need to consider container runtime, monitoring and management, automation, security hardening, cloud-native storage, and much more. Just reading the list is enough to make your head spin, let alone doing it all.
The number of decisions that need to be made to build an enterprise-grade platform can be overwhelming—even for the most seasoned of IT teams. The decision making doesn't end after the platform is in place, either. New projects are being introduced constantly—ways to make development easier, faster, more secure. Teams need to constantly evolve the platform to keep pace.
This requires much more work than merely downloading the latest version of something. At minimum, it involves staying on top of researching the latest tools available, ensuring that they properly can be integrated into your platform to work with your other tools and technologies, that the new tools are refined to not open up your organization to increased vulnerabilities, and more.
All of your integrations also need to be monitored for new versions available and updated when needed. But new updates can present their own challenges. Without updating to the latest version of a technology, you are often opening your organization up to security vulnerabilities (more on security later). If you do update to the latest version, however, there are often learning curves associated with the new features and tools available.
Skill Shortages
Application platforms can be expensive. But do you know what else is expensive? Application platform developers. And to fully build and maintain an organization-wide, enterprise-grade platform you need Kubernetes "pros of pros"—and quite a few of them.
Kubernetes can be complicated. Building a complete platform around it requires knowing the ins and outs of everything touching it. Red Hat alone answers nearly 2,000 support tickets related to Kubernetes each month. If you're lucky, your organization might have the skills and manpower in house to devote to the project. However, if you're looking to expand your team to take on the development and maintenance, this can be difficult. Red Hat's 2022 Global Tech Outlook found that skill set or talent gaps were the top barriers to successful companies with their digital transformation.
Even if your team has skilled IT professionals in place, it doesn't mean they have the specific Kubernetes skills needed. Red Hat also found that 43% of developers don’t have the necessary skills for adopting containers—and that 39% are lacking resources to do so. And according to the Linux Foundation's Open Source Jobs Report, 91% of organizations that have deployed cloud-based technologies use Kubernetes—resulting in a 455% increase in the demand for Kubernetes certifications.
As Kubernetes adoption continues to increase, so does the demand for this skill set—further widening the skills gap. Organizations who decide to pursue vanilla Kubernetes may need to get creative in finding the developers they need, such as allocating time and resources to continuing Kubernetes education.
Writing Integrations
The massive amount of time that writing integrations took was something our team did not anticipate when we began developing the platform. We quickly found that it's never just one integration. Once a technology is selected, it needs to be integrated everywhere to be effective. This circles back to the skills challenge: do you have a team in place that can support writing these ongoing integrations?
Just to get started, a huge volume of code needs to be written to integrate different features together. For example, we need to integrate the company repo with the Kubernetes environment, monitoring with the company monitoring solution, logging with the alerting and metrics solution, and so on. Kubernetes is just the beginning. Everything that needs to be integrated is truly what enables organizations to make a Kubernetes platform production-grade.
These integrations also need to be fine-tuned to help ensure reliability and security while in production. Take disaster recovery, for example. If you are an enterprise organization, you can't risk having unplanned downtime. Kubernetes, as an open-source technology, does not have disaster recovery included. This means writing another integration and connecting it to the clusters and the relevant data inside to back it up. While the specific integrations can be challenging, the complexity really comes when making sure that all these individual pieces work together as a functioning whole.
Patching and Security
We quickly came to realize that patching alone could be a full-time job when it came to managing and maintaining the platform. Team members needed to constantly monitor projects for information about newly discovered vulnerabilities, patches, and updates. This can consume a significant amount of time and effort. Unless an organization has the resources to dedicate a team of people to manage this, systems can be left vulnerable to costly and wide-ranging consequences.
While at first glance it may seem that the same effort is required to apply patches both to upstream open-source code and to vendor-supported Kubernetes distribution, there can be important differences. Most upstream projects provide fixes by updating the code in the most recent version (or branch) of the project. Therefore, patching a vulnerability requires updating to the most recent version. This can add risk because that most recent version may have additional changes that are incompatible with the organization's use of the previous version—or it may include other issues that have not yet been discovered simply because the code is newer.
Vendor distributions, on the other hand, often backport vulnerability fixes to older versions— essentially isolating the upstream change from a later version that fixes a particular issue and applying it to an earlier version. This backporting provides a more stable solution for applications consuming that software—while also addressing the newly discovered vulnerability.
Getting What You Pay For
At the end of the day, Kubernetes is an incredibly powerful orchestrator. That said, it's just one piece of the puzzle. For enterprise-grade deployments, you'll need an entire stack built around Kubernetes.
If vanilla Kubernetes in production is something your team is considering, it's worth sitting down to walk through what your organization's platform will look like and what resources you have available to devote to the project.
In my own experience, the cost of "free" ended up being steep. Building a platform based on vanilla Kubernetes is possible, but there's something to be said for getting everything you need right out of the box and being able to innovate for customers on day one.
Keep learning
Choose the right ESM tool for your needs. Get up to speed with the our Buyer's Guide to Enterprise Service Management Tools
What will the next generation of enterprise service management tools look like? TechBeacon's Guide to Optimizing Enterprise Service Management offers the insights.
Discover more about IT Operations Monitoring with TechBeacon's Guide.
What's the best way to get your robotic process automation project off the ground? Find out how to choose the right tools—and the right project.
Ready to advance up the IT career ladder? TechBeacon's Careers Topic Center provides expert advice you need to prepare for your next move.
