Fall 2026 Kubernetes Upgrades
Robert F.P. Ludwick
Almost-Annual Upgrades
Every 10 months I run upgrades on my Kubernetes cluster. I haven’t blogged about my cluster setup or hardware before; I set it all up pretty shortly after I wrapped up my home lab blog posts, part 5. Maybe in the future I’ll document my initial setup, but that’s out of scope for this post.
In this upgrade I needed to upgrade from 1.33 all the way to 1.36 - hitting 1.34 and 1.35 along the way. So I read through the upgrade documentation for 1.34 to 1.36… 1.34 itself was no concern, and neither was 1.36. But 1.35? It deprecated support for containerd versions less than 2.0. And I was running 1.7 at the time.
Incepting The First Dependency - containerd
Ask any homelabber who has been in the game long enough and they’ll tell you they’ve had to go at least 1 layer deeper into dependency upgrades in order to make the upgrade they originally wanted all along. Heck, this holds true for professional technical jobs too for that matter. So in order to upgrade Kubernetes to my desired version, I had to upgrade containerd along the way. I chose to make that upgrade before doing any of the Kubernetes upgrades just to simplify my process, rather than ping-ponging back and forth between just which software I was upgrading.
I had gotten containerd 1.7 installed via the
Docker repository. After some sleuthing around, I couldn’t find an appropriate apt
repository for Raspbian/Debian on the Internet which would suit my purposes of upgrading
containerd. It looked to me like I needed to make a manual installation using their
binaries per their docs.
I was a bit concerned about the prerequisites being met on my base OS, where I was
still running Bookworm on my Raspberry
Pis (my Kubernetes cluster is running on a fleet of 8 Raspberry Pi 4s). Unfortunately
I didn’t record down where I found this information, but I came across someone stating
they had issues getting containerd 2.x running
on a Raspberry Pi Bookworm installation. I knew I needed to upgrade the base OS
too, so… why not… let’s go one level deeper.
One quick note here - I found it interesting that direct LTS upgrades from 1.7 to 2.3 were supported but that intermediate upgrades weren’t necessarily so, per containerd’s docs. So I decided to go all the way to 2.3.
Incepting The Second Dependency - Raspbian
I’ve done this song and dance before. My fleet was on Raspbian Bookworm when I began this exercise, but when I first set this cluster up I set it up on Bullseye. Raspberry Pi major OS upgrades aren’t officially supported, but you can still do it if you’re careful. So I pulled up this guide to make my upgrade to Trixie. I’m a glutton for punishment, which is why I do on-device upgrades rather than flash a new installation of Raspbian.
Before I get into what happened when I upgraded my first node, here is effectively the full script I ran on a per-node basis to upgrade the OS… and, simultaneously, containerd. I’ll get to that shortly. I also ran this on my four Raspberry Pis which are not in my Kubernetes cluster.
sudo apt update
# START k8s node only
sudo apt install -y kubernetes-cni=1.6.0-1.1
sudo apt-mark hold kubernetes-cni
# END k8s node only
sudo apt full-upgrade -y # on first k8s node, don't -y
# START k8s node only
# cordon node
# END k8s node only
sudo reboot
# START Pihole only
sudo pihole -up
# END Pihole only
sudo vim /etc/apt/sources.list /etc/apt/sources.list.d/*
# START non-k8s only
apt-key list | grep -A4 "trusted.gpg$"
sudo apt-key export <last8> | sudo gpg --dearmor -o /tmp/raspi.gpg
file /tmp/raspi.gpg
sudo apt-key del <last8>
sudo mv /tmp/raspi.gpg /etc/apt/trusted.gpg.d/
# END non-k8s only
sudo apt update
sudo apt purge raspberrypi-ui-mods
sudo apt autoremove -y # on first k8s node, don't -y
sudo apt full-upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confnew" --purge --auto-remove # on first k8s node, don't -y
sudo sync
sudo reboot
# START k8s node only
sudo service kubelet stop
# END k8s node only
lsb_release -c
sudo apt autoremove -y # on first k8s node, don't -y
# START k8s node only
sudo apt remove -y --allow-change-held-packages containerd.io
wget -O /tmp/containerd-2.3.3-linux-arm64.tar.gz https://github.com/containerd/containerd/releases/download/v2.3.3/containerd-2.3.3-linux-arm64.tar.gz
sudo tar -xvzf /tmp/containerd-2.3.3-linux-arm64.tar.gz -C /usr/local
sudo wget -O /usr/lib/systemd/system/containerd.service https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
wget -O /tmp/runc.arm64 https://github.com/opencontainers/runc/releases/download/v1.5.1/runc.arm64
sudo install -m 755 /tmp/runc.arm64 /usr/local/sbin/runc
wget -O /tmp/cni-plugins-linux-arm64-v1.9.1.tgz https://github.com/containernetworking/plugins/releases/download/v1.9.1/cni-plugins-linux-arm64-v1.9.1.tgz
sudo mkdir -p /opt/cni/bin
sudo tar -xvzf /tmp/cni-plugins-linux-arm64-v1.9.1.tgz -C /opt/cni/bin
sudo systemctl daemon-reload
sudo systemctl unmask containerd
sudo systemctl enable --now containerd
sudo service kubelet start
# uncordon node
# END k8s node only
“Simultaneous” Upgrades
As you see above, my OS upgrade script also includes the containerd upgrade. Why?
Well, when I upgraded the OS on the first node, when it rebooted into Trixie…
it bogged down under massively high load. To the point where I could barely even
SSH in and run ls commands. When I managed to run top the most intensive processes
were all containerd.
Rather than try very slowly to dump out log files and journals, I just shut down
kubelet and containerd. Node returned to responding quickly. Now I could
have gone through those log files and tried to figure things out, but I had a hunch
the problem was Trixie and containerd 1.7 in combination. So I did it live, I shipped
it, I yeeted myself to containerd 2.3 as part of the OS upgrade.
It worked. When I brought back up kubelet and the 2.3 version of containerd
services, everything worked correctly. The node was reporting to the control plane
as healthy, the daemon sets came online correctly… everything was green. So I
uncordoned the node and let some workloads run on it. All was well. Phew! So then
I ran the upgrade on a second node with the full script above. All was well! Great,
about to move on to node 3…
Concourse Issues
As I said, I’m a glutton for punishment. I run CI at home, via Concourse,
on my Kubernetes cluster. Again, this is also a blog post for another day. Well,
after I upgraded my second node, I noticed that when one of my Concourse workers
was scheduled onto one of the newly-upgraded nodes, it was in a CrashLoopBackoff.
Sigh. Looked at the logs, and while I didn’t record their output directly, they
were complaining about iptables issues. I immediately
knew it was related to the OS upgrade, but how?
I came upon the root cause in this Concourse GH Issue 9449.
The baked-in support for iptables in the concourse container image was legacy.
If you read that issue, you’ll see how I resolved my problem. I’ll quickly
summarize - I extended the concourse image and removed the legacy iptables packages
in favor of the modern ones. It worked on both the nodes still o Bookworm and the
two which had been upgraded to Trixie.
After this point, I was able to run the OS & containerd upgrades on the six remaining Kubernetes hosts.
Kubernetes Version Upgrade, Finally
Thankfully, the Kubernetes upgrades went much more smoothly, as they have in the past. The script below is the pattern I used to go from 1.33 all the way to 1.36; it shows the upgrade from 1.35 to 1.36.
# 1.36
# upgrade kubectl version on laptop to 1.36
sudo sed -i "s/35/36/g" /etc/apt/sources.list.d/kubernetes.list
sudo apt update
sudo apt-mark unhold kubeadm
sudo apt-get install -y kubeadm=1.36.3-1.1
kubeadm version
# check the version output
# START control plane only
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.36.3
# END control plane only
sudo kubeadm upgrade node
kubectl drain kube-<name> --ignore-daemonsets
# wait for longhorn replicas to copy off of kube-<node>
sudo apt-mark unhold kubelet kubectl kubernetes-cni
sudo apt-get install -y \
kubectl=1.36.3-1.1 \
kubelet=1.36.3-1.1 \
kubernetes-cni=1.9.1-1.1
sudo apt-mark hold kubeadm kubelet kubectl kubernetes-cni
sudo systemctl daemon-reload
sudo systemctl restart kubelet
sudo apt autoremove -y
kubectl uncordon kube-<name>
You’ll notice in that script above a note about Longhorn replicas. I run Longhorn to get local block storage for my Kubernetes nodes. I have only a few volumes, and the biggest one is Prometheus - at the time of my upgrades it was 164Gb in size. I run two replicas per volume, and during Kubernetes node maintenance I need to allow Longhorn replicas to copy off of the node under maintenance to make sure I maintain two replicas, and also to allow Longhorn to try to get a volume on the same node as the workload it is supporting.
So there ends up being a bit of thrash of Longhorn volumes during rolling upgrades across my fleet. With workloads always shifting around during upgrades, Longhorn volumes end up having to copy on occasion, and for 164Gb volumes, that takes time and a lot of I/O on the hosts.
Stretch Goal: Longhorn
During my Kubernetes node upgrades, I would keep a watch on my Longhorn volumes in its UI. At some point I looked at the UI footer and noted… that I was several Longhorn versions behind… Ugh… Well, in for a penny, in for a pound. Stretch goal unlocked: upgrade Longhorn versions too!
I was on 1.7 and needed to upgrade all
the way to 1.12. Typically, Longhorn upgrades
are easy if lengthy - it’s a simple helm upgrade to the new version. Going to
1.8 was easy, as was 1.9.
But 1.10 failed to fully come up; multiple
pods throwing errors. I downgraded the Helm chart back to 1.9.2 and everything was
stable again. Upon further reading, I had erroneously missed this note in 1.9
and in 1.10
about an API version deprecation. My Longhorn installation began pre 1.3, so I had
the old v1beta1 APIs. Thankfully, their migration script helped me remove those
references and I was able to upgrade to 1.10. And then eventually all the way to
1.12.
Longhorn upgrades can get lengthy for me. Because replicas sometimes need to copy between nodes during these upgrades, I have to wait until they have all completed before I can upgrade the volumes’ engine image and eventually remove the old engine image once all references have been dropped.
This Round Is Done
I’m calling this upgrade round as complete. I need to upgrade Prometheus as it’s several versions behind, but otherwise I’m in good shape once again.
Actually, speaking of Prometheus, one byproduct of that particularly large volume is that I found some metrics emissions from Home Assistant that I didn’t need in Prometheus and definitely didn’t need in HA any longer, so I’m pruning some data as a positive side effect.