Setting up Kubernetes with Cilium and Cloudflare
Working with Docker Containers can be hard. However, there are tools which enhance the management of containers, like Kubernetes. Actually, Kubernetes is the only tool to my knowledge which acts as a management software for Docker Containers. Kubernetes is well-integrated in almost all cloud providers, like Google Cloud, Azure and AWS. As a result, it has a standardized yaml
-syntax, which is great for small developers because they can switch between The Big Three
with low effort.
tl;dr
Install everything and then apply cert-manager. ez
Install k3s
As Hagen explains in his article, we want to install k3s
with no configurations and everything disabled. He describes what components are not installed in details.
After the installation, there should be some pods running (3). Don’t be shocked if the pods are in the ContainerCreating
or Pending
state. This is because the pods can’t communicate between each other because we disabled the CNI (--flannel-backend=none
). We will later install Cilium, which will be the replacement of the Flannel CNI.
Install Helm
Helm is the package manager for Kubernetes, so you should either install it directly (follow the Helm docs) or use parts of Helm which are shipped with Cilium. We chose to install Helm directly, which is easily possible with this command:
Install Cilium
Cilium is a networking and security software for Kubernetes. Cilium is very fast, scalable and secure because it’s built upon eBPF — a revolutionary technology that can run sandboxed programs in the Linux kernel without recompiling the kernel or loading kernel modules.
We could install Cilium with Helm like shown here:
However, we wanted to install with their CLI and this is how you can do it. Firstly, install the Cilium CLI by running this code snipped:
Then you can install Cilium with your Server IP-Address:
Now we wait until Cilium says, everything is OK
or disabled
:
After a while, all pods should be Running
.
Last but not least, you can apply some resources for Cilium:
Additionally you should upgrade the cilium config. In order to do that with the proper values, first create this file in the root directory where you wanna manage the k3s cluster. Later you could also apply some hubble and prometheus related properties if you want to use Grafana or so (open the collapsed lines if you want to use our config as well).
Run this command to upgrade:
Setup Certificate Manager with Cloudflare
In order to be able to create certificates for each subdomain, it is important to apply a certificate issuer which handles certificate requests and resolves them at some provider. We chose Cloudflare as our issuer and here is the setup which you need to apply to your kubernetes cluster. For further information you can check out the cert-manager docs.
But first, we need to install the cert-manager by running the following command:
You can apply a file to the Kubernetes cluster, by running this k8s (also k3s) command:
If you want to delete the resource in the Kubernetes cluster, the command is pretty straight forward:
As you may have spotted above, we also need a secret for the API token which authenticates that this issuer is allowed to request certificates. Therefore, we create a secret with an unencrypted API Token
from Cloudflare.
Nowadays we create a token by going to your Cloudflare dashboard, then click on your profile and select the tab API Tokens
. Here you can generate a specific token for your issuer or use the Global API Key (not recommended any more). The recommended solution is to create a API token with two permissions (custom token):
- Zone - DNS - Edit
- Zone - Zone - Read
A more detailed description about the tokens, can be found in the Cloudflare docs.
After applying this secret to Kubernetes, the issuer should be ready to resolve some bad boys!
You can now use this issuer by applying this file which will hopefully create a certificate:
It usually takes around 90 seconds to authenticate the request once applied. You can check the current status of the request by running this kubernetes command. If it takes longer than 2 minutes, maybe some tips in #Troubleshooting can help you.
Example app mutanuq
Then you can use this certificate in your Ingress controller:
Setup Keel
We always wanted a clean Continuous Integration (CI) and Continuous Delivery (CD) solution for our websites. This means, that a specific commit message should trigger an automated process over GitHub, Docker Hub and our server, which in the end updates the corresponding website after about two minutes.
Keel is a robust software tool which enables this feature for Kubernetes. We used Keel for pulling new Docker Images from Docker Hub by polling every few minutes. Moreover, Keel provides a beautiful dashboard where you can control the polling as well.
In order to set up Keel with the admin dashboard, we created those files:
secret-dashboard.yaml
for the Admin Username and Password (not everyone should be able to access the dashboard)keel.yaml
for the actual k3s configs (copied and adapted from KeelHQ)
After applying both files and managing the additional certificate for keel.trueberryless.org
, the Keel dashboard works perfectly. Moreover, every Kubernetes Deployment
can opt in for automated Docker Hub Polling by setting some annotations:
Celebrate with a Coffee!
Congratulations, you’ve successfully set up Kubernetes with Cilium and Cloudflare! You deserve a coffee break. Enjoy a well-earned cup, and if you’d like to share a virtual coffee with me, feel free to support my work on Ko-fi. Thank you!
Troubleshooting
Cilium-ingress has no External-IP
Make sure that the ip-pool
includes the address specified by the annotations in the config.yaml
file.
If you don’t deploy locally but on one of The Big Three
, please check out some other documentation on why the External IP is still pending. It’s mostly their obligation to provide you with an address.
No Cloudflare Certificate Approval
There can be some problem when the certificate won’t get approved by Cloudflare.
Wrong API token
First make sure that the Cloudflare API token is correct. To make 100 percent sure, create a new one and put it (not base64 encoded) into this file:
Max auth failures reached
We once ran into the error Error: 9109: Max auth failures reached, please check your Authorization header.
. Just wait a few hours, delete the resource and apply it again:
Hopefully, you’re now good to go!