apt-mark hold not set on kubernetes installation

[Todd B]

Not a bug in Ubuntu per se, but the Ubuntu installer behavior creates bugs in the Kubernetes cluster over time.

When upgrading a minor release of Kubernetes (K8s), it's essential to run post-installation upgrade scripts on the cluster in order to bring it up-to-date with the most recent kubeadm version installed by Ubuntu. The way that it stands today, upgrading Ubuntu 18.04 LTS to keep up with patches will also automatically upgrade Kubernetes without the operator knowing that they need to take further action to bring the cluster in line with the newest minor version of K8s. If you look at the official K8s upgrade instructions for Ubuntu at https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-15/, you'll see that they are expecting a hold to be placed on the package upgrade and that the operator must release it using `apt-mark unhold kubeadm`. In my particular case, I installed K8s version 1.13.4 earlier this year on a new 18.04 LTS server and through regular patching it was automatically upgraded to 1.15.0 without my knowledge so I never upgraded the cluster to match. The result is that the pods in the cluster became unavailable at an alarming rate and my application was sending back lots of 504 Gateway Timeout errors to the browser as the result of the disconnected pod. I was able to upgrade the cluster to 1.14.0, then to 1.15.0, and that fixed the problem completely. I've now added holds for kubeadm, kubelet, and kubectl to my own installation instructions, but it would be nice if Ubuntu did this automatically on installation so that folks don't get into this pickle in the first place.

Thanks.

ubuntu@kubernetes-master:~$ lsb_release -rd
Description: Ubuntu 18.04.2 LTS
Release: 18.04

[Todd B (spam-buiten)]

Oh, additionally... Though I've never marked any package for hold until now, I'm assuming that `apt-get upgrade` would warn that it's skipping a new package that's on hold, thereby allowing me to see that a new version is available that I can chose to upgrade when I'm ready. If not, that's ok, because I can manually check available package versions if I need to. Putting a hold on the package at initial installation would be a life-saver and would have saved me many, many hours tracking down this issue