It's not hard to find high-quality beginner tutorials if you're starting to learn Kubernetes. But where can you find more advanced learning resources once you've mastered the basics?
The official Kubernetes docs have some resources for intermediate and advanced users, but many useful pro-level tips are scattered around the web. The resources gathered below are intended to help engineers take the next step in their Kubernetes education—past the beginner's stage toward full mastery.
The topics covered include security, monitoring and logging, source code, continuous integration (CI) and continuous delivery (CD), and other areas that engineers must understand to run Kubernetes reliably in production.
Kubenetes: Looking into the guts
A few things I've learned about Kubernetes
Julia Evans is a superstar in software and operations engineering. This article includes her first impressions of the technology after trying it out and reading the source code. While it also introduces a lot of beginner concepts, the article also looks at Kubernetes from an easy-to-understand perspective.
There's also a lot of advice in this article that you won't find in other introductions, such as her suggestion to learn networking if you want to save yourself some grief. Evans' blog has a number of helpful Kubernetes resources, covering the scheduler, certificates, networks, and why it's "cool" — with helpful illustrations and expert-level advice that you won't get from your run-of-the-mill beginner tutorials.
One of Evans' famous explainer sketches.
A Tour of the Kubernetes Source Code: From kubectl to API Server
A crucial step in mastering any technology is reading the source code and seeing how it works. To read Kubernetes' source code, you'll need to learn some syntax for the Go programming language.
After that, read Brad Topol's tour of the Kubernetes source code, which includes a look at the code that runs kubectl commands as well as tools and techniques for learning the Kubernetes source code.
Level Triggering and Reconciliation in Kubernetes
When you're first learning a new technology, the starter lessons tend to be pragmatic—you don't learn all the details about how a tool does what it does. But if you're going to master a technology, you need to understand the technical philosophies that influenced the designers of the tool. Why is Kubernetes so great at self-healing?
James Bowes discovered that the edge and level-triggering concepts from electronics and systems programming are behind Kubernetes' resilient properties. His article takes a look at these concepts and how they can give you a deeper understanding of how Kubernetes works.
"For Kubernetes, thinking about the problem as a level triggered system has led to an architecture that is clean, simple, and does what the user wants in spite of the inherent problems in distributed computing."
— James Bowes
Advanced content streams
The Kubernetes Blog
The first feed any Kubernetes developer should start following is the official Kubernetes blog. Not only will it keep you updated on the latest releases of Kubernetes and various supporting tools, the blog also includes community news, security warnings, case studies, and tutorials.
Some of the best recent tutorials include:
- Developing on Kubernetes
- Zero-downtime Deployment in Kubernetes with Jenkins
- How Bitmovin is Doing Multi-Stage Canary Deployments with Kubernetes in the Cloud and On-Prem
- Dancing at the Lip of a Volcano: The Kubernetes Security Process - Explained
- Kubernetes Meets High-Performance Computing
Google Cloud Platform on Medium
When you sort the GCP blog on Medium by the tag "Kubernetes" you get a steady stream of technical content. Much of it is focused on Google's Kubernetes Engine service, but there are also plenty of product-agnostic Kubernetes tutorials and best practices articles.
Heptio on Medium
A growing number of Kubernetes startups have appeared in the last few years, and one of the most well-known is Heptio, a Kubernetes products and services company that frequently features employees as speakers at major DevOps and container conferences (including the CTO, Joe Beda, a co-creator of Kubernetes). The company also has a team of prolific writers who blog on Medium. In addition to blogging about a few Heptio releases, these experts mainly write about leading-edge Kubernetes topics, including how to secure the Kubernetes dashboard and how to ace the Certified Kubernetes Administrator (CKE) exam.
Cloud Native Computing Foundation (CNFC) YouTube Channel
Recorded conference talks can be another great way to learn about advanced topics and stay ahead of the curve. The playlists for KubeCon/CloudNativeCon are your best bet if you're looking to learn from the leading Kubernetes experts.
Advanced tips and tricks
Advanced Kubernetes Objects You Need to Know
Kubernetes beginners usually learn some basic objects like Pods, ConfigMaps, and Deployments, but Mustafa Akin wants to introduce you to some of the lesser known objects that can improve your workloads and day-to-day operations. His list includes ResourceQuota, PriorityClass, LimitRange, PodSecurityPolicy, and more. It also covers a number of admission controllers, which check Kubernetes objects before they hit the database.
Introducing Operators: Putting Operational Knowledge into Software
Kubernetes Operators are catching on since their release in late 2016. These application-specific custom controllers essentially allow you to write an application to fully manage another, which is especially helpful for hard-to-manage stateful applications such as databases. You'll want to learn how to build your own Operators as you become a more advanced Kubernetes user.
How to move from single master to multi-master in an AWS kops Kubernetes cluster
Bringing high availability to a Kubernetes cluster through multiple master nodes is a tricky maneuver, typically not attempted by beginners. But for advanced Kubernetes users, it's a useful resilience strategy to practice. This tutorial by Blender Fox shows how you can do it with kops—an enhanced Kubernetes installation—running on top of Amazon Web Services (AWS).
Cluster @%#’d – How to Recover a Broken Kubernetes Cluster
There are three types of nodes in a Kubernetes deployment: master nodes, etcd nodes, and worker nodes. This short piece on the Codefresh blog gives you concise instructions on how to tell if any of these node types are down, and what the high-availability and non-high-availability recovery steps are.
Kubernetes Cluster Federation (The Hard Way)
Kubernetes cluster federation is an almost magical technique for managing multiple Kubernetes clusters and moving workloads freely between them—using any combination of cloud providers or bare metal. The best way to teach yourself cluster federation is this tutorial by Kubernetes legend Kelsey Hightower. It's called "The Hard Way" because it's focused on doing things slowly and methodically so that you retain all the information and gain a strong understanding of Kubernetes federation on your way to building the system.
"This guide is not for people looking for a fully automated command to bring up a Kubernetes cluster federation. ... This tutorial is optimized for learning, which means taking the long route to help people understand each task required to bootstrap a Kubernetes cluster federation."
— Kelsey Hightower
Although it uses Google Cloud instead of a product-agnostic environment, you won't find many better tutorials out there. After you understand federation using Hightower's tutorial, you can move on to setting up clusters with Kubefed, which abstracts away several steps in the "hard way" process. Finally, you can solidify your knowledge by learning how to deploy a Node.js app on Kubernetes clusters across multiple cloud providers (Google, Azure, and AWS).
Implementing Advanced Scheduling Techniques with Kubernetes
This post goes deep into the inner workings of the Kubernetes scheduler, with helpful diagrams and examples. It explains how Kubernetes chooses the hosts for pods by running through the scheduler's three-part process: Node filtering, node priority calculation, and scheduling.
Provisioning Kubernetes Local Persistent Volumes
Ian Chakeres and Dhiraj Hegde describe how Salesforce Storage Cloud operations engineers prepare their bare-metal servers, provision local persistent volumes, and expose multiple storage types to stateful service owners. The article is a good example and case study, where Kubernetes is used for automated deployment, management, and scaling of these Storage Cloud systems.
Step by step guide to set up a multi-region Kubernetes app
What do you do when you're running a Kubernetes app on a public cloud in one geographic region, but your users are all over the world? If this starts to become an issue, you may need to put your app on more than one region. Francois-Guillaume Ribreau shows you how he did this for a Kubernetes app in Google Cloud Engine. Similar steps can be taken for other providers as well.
Worldwide distribution map for Ribreau's SaaS customers.
Building stuff with the Kubernetes API
When you've reached a certain level of mastery with Kubernetes and your team's needs for the tool have evolved beyond out-of-box features, it's time to start programming your own features for Kubernetes. Vladimir Vivien provides the perfect introduction for doing this in three different languages: Python, Java, and Go. Vivien walks through the different options for extending Kubernetes, from simple tools to custom API types.
Extending Kubernetes: Create Controllers for Core and Custom Resources
Thomas Stringer has an excellent tutorial that is also about extending Kubernetes—focusing on how to create controllers and custom resources. Stringer's article isn't going to help you decide whether or not you need to use controllers and custom resources, but he gives you a good page in the Kubernetes documentation that will.
Udemy: Advanced Kubernetes Usage
If low-cost online courses are one of your favorite ways to learn, then you should check out this post-beginner course at Udemy. It covers a wide range of important topics for Kubernetes, including centralized logging, authentication, authorization, package management, the job resource, scheduling, deploying on Kubernetes with Spinnaker, and federation.
Top 10 Kubernetes tips and tricks
Tips from an experienced Kubernetes user that aren't found in documentation and beginner tutorials are always helpful. Some of Timothy Josefik's tips include using pod disruption budgets, readiness and liveness probes, and labeling everything.
Kubernetes on Azure, Google, and AWS
Kubernetes Adventures on Azure
Ivan Fioravanti shares a three-part series that covers Kubernetes on Azure's infrastructure using both Linux nodes and Windows nodes. The series also shares a trick for scaling pods.
Kubernetes dashboard showing a hybrid cluster running on Azure.
Tutorial: Run Kubernetes on Amazon Web Services
The Kubernetes experts at Heptio wrote an extensive tutorial/ebook on running Kubernetes on AWS. The tutorial assumes you are already familiar with the AWS CLI and Cloud Formation. It goes into minute detail about how to create a stack from a Cloud Formation template, how to view stack and resource information, and how to manage the cluster.
Kubernetes The Hard Way
Since he works for Google, Kelsey Hightower is a great source for learning how to use Kubernetes on Google's cloud infrastructure. Although this tutorial uses Google Cloud Platform (GCP) for basic infrastructure, the core Kubernetes concepts he explores are applicable to other platforms. It's a great tutorial for beginners, but it's also useful if you want to see what using Kubernetes on GCP looks like.
CI/CD and Kubernetes
Kubernetes for dev infrastructure
Kubernetes isn't just for running production workloads. It can also be useful as a component of your development infrastructure, says Maxim Leonovich. At his company, they had a few hundred Selenium-based workflows. Running sequentially, those tests took 10 hours to complete. His mission was to make those integration tests faster.
This article explains how he used Docker and Kubernetes in his dev/test environment to spin up and tear down test environments that ran those Selenium workflows in just two or three minutes.
Dynamic Environments with Kubernetes
Dynamic environments on demand are Ian Crosby's take on the idea of immutable infrastructure, but applied one level higher. He demos this strategy using the namespaces in Kubernetes, allowing production, staging, testing, and development environments to be spun up and taken down at will.
Set Up a CI/CD Pipeline with Kubernetes
This multi-part series by Craig Martin shows you how to set up a fully containerized application stack in Kubernetes with a simple CI/CD pipeline to manage deployments. The application you deploy is a crossword puzzle example app that was built to showcase key Kubernetes and CI/CD concepts.
The architecture diagram of Martin's CI/CD pipeline, which uses minikube.
Kelsey Hightower's "Pipeline" guide
Hightower provides another excellent tutorial that shows you how to build an end-to-end deployment pipeline using Cloud Container Builder (a Google cloud commercial product), GitHub, and multiple Kubernetes clusters. By the end of the tutorial you'll be left with fine-grained access control for each environment and streamlined automated build steps targeting those environments.
Set Up a Jenkins CI/CD Pipeline with Kubernetes
This is a great post for understanding how to use Jenkins, the nearly ubiquitous CI engine, with Kubernetes. Alen Komljen shows that it's easy to set up Jenkins on a Kubernetes cluster with Helm and the Kubernetes plug-in, but it's harder to create a pipeline. There are plenty of Jenkinsfiles to use, but they can break if you try to customize them without knowing the ins and outs. Using a Jenkinsfile example, Komljen shows you how to work with them on Kubernetes.
How to Setup Scalable Jenkins on Top of a Kubernetes Cluster
A single Jenkins instance can grow very fast, which leads to redundant resource consumption and a sluggish delivery pipeline. That's why you might want to put Jenkins on a Kubernetes cluster, says Yuri Bushnev, a software developer in test for AlphaSense. Using minikube, Bushnev shows you the power of Jenkins auto-scaling on a Kubernetes cluster in this tutorial.
Kubernetes deployment
Deploying Rock Solid Applications with Kubernetes
This is the second article in a two-part series. The first article might also be useful if you've used Kubernetes only in an experimental capacity and haven't actually migrated to it. In Migrating to Kubernetes with zero downtime: The why and how, Jelmer Snoeck shows how explains why his team decided to move to Kubernetes—sharing the questions they asked. He describes the trade-offs they made and shows you how his team configured their Kubernetes cluster to achieve their goals. In Deploying Rock Solid Applications with Kubernetes, Snoeck outlines powerful Kubernetes tools for reliable deployments, including health checks, pod (anti)affinity, network policies, security policies, rescheduling, and PodDisruptionBudgets.
Kubernetes components in a high-availability configuration.
Kubernetes deployment strategies
A useful cheat sheet written by Etienne Tremel, this article gives distilled descriptions of five deployment strategies: Recreate, ramped, blue/green, canary, and a/b testing. He then lists the pros and cons of each and provides a code snippet and a link to an example using Kubernetes on GitHub.
The best architecture with Docker and Kubernetes — myth or reality?
While this article is not a tutorial, Dmitriy Paunin shows you how to think about your development workflow and software architecture in a Kubernetes and container-based environment. The article covers automated service provisioning, logging, tracing, and other topics at a strategic level.
Kubernetes tools
10 open-source Kubernetes tools for highly effective SRE and Ops Teams
There are tons of open source tools for Kubernetes all over GitHub. It's hard to be aware of everything that could potentially help you. That's why Abishek Tiwari compiled this list of some of the most popular tools for site reliability engineers (SREs) and operations engineers who use Kubernetes. The list includes concise descriptions and animated visuals for Kubectx, Cabin, Kube-ops, Kube-shell, and more.
A view of the Kube-ops dashboard.
5 Tools for Monitoring Kubernetes at Scale in Production
Another article by Craig Martin shares his top five Kubernetes monitoring tools, which are not only the ones his company uses. They are also some of the most popular tools, according to a New Stack survey, which is included in the article. The tools include Heapster, Prometheus, CAdvisor, InfluxDB, and Grafana.
50 Useful Kubernetes Tools
Stefan Thorpe has compiled a more extensive list of Kubernetes tools, conveniently categorized for easy reading. Some of the categories include tools for monitoring, security, testing, deployment, command line, service discovery, virtualization, and serverless.
Kubernetes monitoring
Monitoring in the Kubernetes era
Monitoring is crucial for keeping track of the many nodes that a Kubernetes cluster can support. Jean-Mathieu Saponaro wrote an excellent Kubernetes monitoring series on the Datadog blog. As you'd expect, they have a demo using their commercial product, but that's only in the last article of the series. The series explores how Kubernetes changes how you do monitoring, key metrics to monitor, and alerting. And it provides a concrete use case using Heapster.
Logz.io has a similar tutorial on its blog that talks about tools such as Heapster, InfluxDB, Prometheus, the ELK stack, and Grafana. The Kubernetes blog also has helpful monitoring guide that's more succinct than the other two.
Kubernetes monitoring with Prometheus in 15 minutes
Looking for a quicker start to your Kubernetes monitoring journey? This article promises to get you up and running with the popular open source monitoring framework, Prometheus, in just 15 minutes. Giancarlo Rubio harnesses one of the powerful Kubernetes Operators mentioned earlier in the article to get Prometheus set up for monitoring Kubernetes clusters in just a few minutes.
Application Logging in Kubernetes with fluentd
Rosemary Wang takes a deep dive into Kubernetes logging with the popular open-source data collection tool fluentd. She walks through the steps she took to collect the logs, isolate them, and clean them up. You can find her full sample configuration on GitHub.
The fluentd log output for nginx’s application.
Kubernetes networking
Operating a Kubernetes network
If you're diving into the topic of networking, look no further than Julia Evans' blog for some excellent, straightforward thoughts and advice for managing this tricky fundamental topic. In this post, she notes that there are plenty of resources on how to set up Kubernetes networking, but not many on how to operate that network. Evans shares tips that she has learned from operating a Kubernetes network.
Understanding Kubernetes networking
Mark Betz covers Kubernetes networking in three articles that address pods, services, and ingress. Each post is detailed and practical with several diagrams and code examples.
"I wrote these posts because I think that once you learn how each of the pieces works it all makes sense in a pretty elegant way."
— Mark Betz
Kubernetes network policy recipes
Learn how to drop/restrict traffic to applications running on Kubernetes with Ahmet Alp Balkan's awesome tutorial on GitHub. The article contains several examples of Kubernetes network policies in action. It also includes sample YAML files to leverage in your setup. If you're not familiar with Kubernetes networking policies, Balkan recommends reading his blog post about Securing Kubernetes Cluster Networking.
Kubernetes security
Kubernetes Security—Best Practice Guide
Kubernetes is a powerful platform that can also be a big problem if it is hacked because it wasn't configured properly. Just ask Tesla. That's why Simon Pirschel's GitHub repository for Kubernetes security best practices is a must-star page. In the guide, there are tips for cloud providers, installers, and general users. Each best practice comes with a helpful emoji label that indicates whether the threat level is severe, high, medium, or low.
Building Container Images Securely on Kubernetes
Jessie Frazelle is a Linux kernel and container specialist who posts about container security ideas that are ahead of the curve for most Kubernetes users. This article shows her method for building container images in Kubernetes without mounting in the docker socket or doing anything to compromise the security of their cluster. You can find the unprivileged container image builder here on GitHub.
In another article, she explores ideas for "hard multi-tenancy," which means having multiple tenants in the same cluster that don't have access to anything from other tenants. You should definitely follow her if you're looking to level up your Kubernetes knowledge.
Kubernetes ConfigMaps and Secrets
There are scores of other ways that engineering teams are exposing their secrets (things like API keys, SSH-keys, passwords, OAuth 2, etc.) to hackers. To learn more about modern secrets management, you should check out another list article on the topic and read this article, which takes an app with hard-coded configuration and modifies it to read from environment variables. Then the author, Sandeep Dinesh, sets up Kubernetes to manage the configuration for you.
In part two, he uses a configuration file instead of environment variables. Note that some experts don't recommend using Kubernetes to do the heavy lifting when it comes to secret management. For that you should look into secrets management tools.
A hacker's guide to Kubernetes security
Rory McCune has four excellent strategic lessons for securing Kubernetes, including some tips about watching ports and not letting any containers get compromised. After you've read McCune's piece, be sure check out 8 ways to bolster Kubernetes security and 5 must-do security steps for Kubernetes for more tips.
Getting credentialed
How Heptio Engineers Ace the Certified Kubernetes Administrator Exam
If you're really serious about proving your mettle as an expert Kubernetes operator, you should consider taking the Certified Kubernetes Administrator (CKE) exam. I shared this link earlier, but it's worth repeating here. The advice includes being intimately familiar with the docs, and doing loads of practice. After that, they have some useful tips for taking the exam and preparing for the practical sections.
Kubernetes Certified Administration
Walid Shaari has collected a list of online resources that will help you prepare for the CKE exam. It covers all of the exam topics, from installation to networking, security, ALM, storage, and monitoring. He even gives you a percentage on each section to indicate how much of the exam covers that topic.
Kubernetes Cookbook
If you're looking for a book on advanced Kubernetes topics for reference, check out the Kubernetes Cookbook by Michael Hausenblas and Sébastien Goasguen. It contains over 80 proven recipes and covers every major Kubernetes subtopic.
Make your own tutorials
These tutorials are meant to get you started in a few advanced topics, but it's up to you to experiment with your own unique use cases.
While it's encouraged to build practice environments that are as close to the ones you work with in production as possible, there are also a few tools, including Kubernetes playground and Let's play with Kubernetes, that provide sandbox instances that are made for setting up and experimenting with Kubernetes clusters.
If you know of any intermediate or advanced Kubernetes tutorials that really helped you and weren't on this list, share them in the comments.
Keep learning
Choose the right ESM tool for your needs. Get up to speed with the our Buyer's Guide to Enterprise Service Management Tools
What will the next generation of enterprise service management tools look like? TechBeacon's Guide to Optimizing Enterprise Service Management offers the insights.
Discover more about IT Operations Monitoring with TechBeacon's Guide.
What's the best way to get your robotic process automation project off the ground? Find out how to choose the right tools—and the right project.
Ready to advance up the IT career ladder? TechBeacon's Careers Topic Center provides expert advice you need to prepare for your next move.
