Kubernetes at Home Part 4: Installing the Cluster
Cloud & Virtualization Linux / Kubernetes / RaspberryPi

With the basic tasks to prepare the nodes completed, I was finally ready to install Kubernetes and get the cluster up and running. As I briefly mentioned in the last post, I chose to install a lightweight version of Kubernetes developed by Rancher, called K3s. I'll be going over the basics, but you can read more about K3s on their official docs . The main reason I chose K3s, like so many others do for Raspberry Pi clusters, is because it's a lightweight version of Kubernetes that's also supported on ARM CPUs and comes as a single binary that's simpler to manage. Rancher also provides installation and uninstall scripts that make it extremely easy to get started.

The first step was to run the install script provided in the K3s quick start guide on the master. In K3s terminology this is the K3s server node. 

pi@k3s-master-rpi001:~ $ curl -sfL https://get.k3s.io | sh -
[INFO]  Finding release for channel stable
[INFO]  Using v1.18.3+k3s1 as release
[INFO]  Downloading hash https://github.com/rancher/k3s/releases/download/v1.18.3+k3s1/sha256sum-arm.txt
[INFO]  Downloading binary https://github.com/rancher/k3s/releases/download/v1.18.3+k3s1/k3s-armhf
[INFO]  Verifying binary download
[INFO]  Installing k3s to /usr/local/bin/k3s
[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
[INFO]  Creating /usr/local/bin/crictl symlink to k3s
[INFO]  Creating /usr/local/bin/ctr symlink to k3s
[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
pi@k3s-master-rpi001:~ $

From the output, we can see that the installation script did a few things in addition to just downloading and installing the k3s binary. First, we see it also created symlinks for kubectl, crictl, and ctr pointing to the k3s binary. It's worth noting because when using these commands locally, they are actually part of the k3s binary and aren't standalone programs. Next, we also see that some utility scripts were created, including the uninstall scripts. Finally, we see that the script also created a systemd unit file for us, enabled it so it runs at boot, and then started it.

K3s also comes with a check-config utility which I ran next to check the installation. It provides quite a lot of output so I've grepped it. 

pi@k3s-master-rpi001:~ $ sudo k3s check-config | grep "swap|System|Necessary|Optional|Network|missing|fail"
System:
- /usr/sbin iptables v1.8.2 (nf_tables): should be older than v1.8.0 or in legacy mode (fail)
- swap: should be disabled
Generally Necessary:
Optional Features:
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: missing
- CONFIG_RT_GROUP_SCHED: missing
- Network Drivers:
      Optional (for encrypted networks):
STATUS: 1 (fail)
In the output, we see that the iptables check had failed. This is because Raspbian Buster (now called Raspberry Pi OS) defaults to using nftables instead of iptables, but K3s requires iptables. This is actually noted in the K3s installation requirements , including the steps to enable legacy iptables . We also see that swap should be disabled and some optional cgroup kernel features and scheduler options are missing.

Before moving on to adding the worker nodes, I created and ran this Ansible playbook to enable legacy iptables on all the nodes. 

# enable-legacy-iptables.yaml

I then rebooted all the nodes using the same adhoc Ansible command I used in the last post and ran the k3s check-config command again to confirm that I no longer saw the "fail" status. Next, the k3s check-config command indicated that swap "should be disabled." This didn't cause the check to fail, but I wanted to better understand why is should be disabled. After doing some research, I learned it's because that the current QoS policy in Kubernetes assumes that swap is disabled. This is stated in the Kubernetes design documents and proposals as follows:
...If swap is enabled, then resource guarantees (for pods that specify resource requirements) will not hold. For example, suppose 2 guaranteed pods have reached their memory limit. They can continue allocating memory by utilizing disk space. Eventually, if there isn’t enough swap space, processes in the pods might get killed. The node must take into account swap space explicitly for providing deterministic isolation behavior.
In short, this means that Kubernetes has no ability to control swap memory independent of physical memory and this can cause performance problems and odd behavior. While I didn't expect the small 100MiB swap file that ships with Raspberry Pi OS to be used much, rather than wait for some strange memory problem to occur, I thought it would be best to follow the recommendations to disable swap at this point. I did this on all the nodes with the following Ansible playbook: 
# disable-dphys-swapfile.yaml
After confirming that swap remains disabled after a reboot, it was time to address the last items shown missing in the k3s check-config output. Ultimately, since adding the missing kernel features were optional at this point, I decided not to make any further changes unless I found they were necessary later. Adding the missing kernel features would have also meant recompiling the kernel from source, which is not such a trivial change.

With the check-config requirements now sorted out, I again followed the K3s quick start guide and ran the install script with the K3S_URL and K3S_TOKEN environment variables to install K3s on the worker nodes and join them to the cluster. To automate this process on all the worker nodes, I did this using an Ansible playbook. The playbook below first downloads the K3S_TOKEN value from the master node where it is stored at /var/lib/rancher/k3s/server/node-token and then runs the install script on the worker nodes using the token value. 

# install-k3s-workers.yaml

After the playbook ran, I confirmed that all the nodes had joined the cluster by using the kubectl get nodes -o wide command. 

pi@k3s-master-rpi001:~ $ sudo kubectl get nodes -o wide
NAME                STATUS   ROLES    AGE    VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
k3s-master-rpi001   Ready    master   4d9h   v1.18.3+k3s1   192.168.2.111   <none>        Raspbian GNU/Linux 10 (buster)   4.19.118-v7l+    containerd://1.3.3-k3s2
k3s-worker-rpi002   Ready    <none>   43s    v1.18.3+k3s1   192.168.2.112   <none>        Raspbian GNU/Linux 10 (buster)   4.19.118-v7l+    containerd://1.3.3-k3s2
k3s-worker-rpi004   Ready    <none>   38s    v1.18.3+k3s1   192.168.2.114   <none>        Raspbian GNU/Linux 10 (buster)   4.19.118-v7+     containerd://1.3.3-k3s2
k3s-worker-rpi003   Ready    <none>   37s    v1.18.3+k3s1   192.168.2.113   <none>        Raspbian GNU/Linux 10 (buster)   4.19.118-v7+     containerd://1.3.3-k3s2

I then deployed some test pods and scaled it up to 20 replicas to verify that everything was working on all the nodes. 

pi@k3s-master-rpi001:~ $ sudo kubectl apply -f https://k8s.io/examples/controllers/nginx-deployment.yaml
deployment.apps/nginx-deployment created
pi@k3s-master-rpi001:~ $ sudo kubectl scale --replicas=20 deployment/nginx-deployment
deployment.apps/nginx-deployment scaled
pi@k3s-master-rpi001:~ $ sudo kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP            NODE                NOMINATED NODE   READINESS GATES
nginx-deployment-6b474476c4-lctzp   1/1     Running   0          2m13s   10.42.2.80    k3s-worker-rpi004              
nginx-deployment-6b474476c4-xfqvq   1/1     Running   0          2m13s   10.42.3.224   k3s-worker-rpi003              
nginx-deployment-6b474476c4-5hdq4   1/1     Running   0          2m13s   10.42.3.225   k3s-worker-rpi003              
nginx-deployment-6b474476c4-jdg5t   1/1     Running   0          2m13s   10.42.3.227   k3s-worker-rpi003              
nginx-deployment-6b474476c4-vvltg   1/1     Running   0          2m13s   10.42.3.230   k3s-worker-rpi003              
nginx-deployment-6b474476c4-5zskx   1/1     Running   0          2m13s   10.42.3.229   k3s-worker-rpi003              
nginx-deployment-6b474476c4-tcsdk   1/1     Running   0          2m13s   10.42.3.232   k3s-worker-rpi003              
nginx-deployment-6b474476c4-zsxqm   1/1     Running   0          2m13s   10.42.3.228   k3s-worker-rpi003              
nginx-deployment-6b474476c4-8mls6   1/1     Running   0          2m13s   10.42.3.231   k3s-worker-rpi003              
nginx-deployment-6b474476c4-w7j6z   1/1     Running   0          2m13s   10.42.3.226   k3s-worker-rpi003              
nginx-deployment-6b474476c4-5tfx2   1/1     Running   0          2m13s   10.42.0.82    k3s-master-rpi001              
nginx-deployment-6b474476c4-4nwz7   1/1     Running   0          2m13s   10.42.0.83    k3s-master-rpi001              
nginx-deployment-6b474476c4-f875h   1/1     Running   0          2m13s   10.42.0.84    k3s-master-rpi001              
nginx-deployment-6b474476c4-8v2c6   1/1     Running   0          2m13s   10.42.1.163   k3s-worker-rpi002              
nginx-deployment-6b474476c4-s7jd7   1/1     Running   0          2m13s   10.42.1.162   k3s-worker-rpi002              
nginx-deployment-6b474476c4-rsf46   1/1     Running   0          2m13s   10.42.1.165   k3s-worker-rpi002              
nginx-deployment-6b474476c4-mjg6d   1/1     Running   0          2m13s   10.42.1.161   k3s-worker-rpi002              
nginx-deployment-6b474476c4-97lqw   1/1     Running   0          2m13s   10.42.1.164   k3s-worker-rpi002              
nginx-deployment-6b474476c4-mc59w   1/1     Running   0          2m13s   10.42.1.167   k3s-worker-rpi002              
nginx-deployment-6b474476c4-6j62t   1/1     Running   0          2m13s   10.42.1.166   k3s-worker-rpi002

Great! Now that everything appeared to be working, I could move on to actually hosting an application in the cluster.

NEXT: Kubernetes at Home part 5: Hosting an Application

PREVIOUS: Kubernetes at Home part 3: Preparing the Nodes

Comments

comments powered by Disqus