Hi. I'm using EKS with a self-managed nodepool, set up via TF.
Over the last 72 hours or so, there has been a high incidence (but not 100%) of new instances registering themselves with k8s, but not honouring the --register-with-taint or --label-node flags.
These instances are booted with the following user-data:
[[[
#!/bin/bash
set -e
B64_CLUSTER_CA=
API_SERVER_URL=
/etc/eks/bootstrap.sh serverless-dev-europe-1 --container-runtime containerd --kubelet-extra-args '--register-with-taints=foo-node=true:NoSchedule --node-labels=foo-node=true' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL
]]]
On inspection of a broken node, kubelet is running from snap with those command-line arguments.
My *suspicion* is that the user-data script is racing - perhaps kubelet is coming up once without the appropriate arguments. This'd cause the node to be registered, then the process is replaced but by that point the node resource is created in k8s and isn't updated. (Certainly, restarting the kubelet process doenst' address the issue.)
I see that https://bugs.launchpad.net/cloud-images/+bug/2012689 is mentioned as a recent fix; I'm wondering whether this is related.
The problem here is two "snap set" commands, one after the other, in bootstrap.sh.
The first gives kubelet all the information to contact the API server and register itself, but not the extra args. This restarts kubelet and begins a race.
The second adds the extra-args and restarts kubelet.
--register-
If the first restart of kubelet gets as far as registering the node, those taints / labels will *never* apply.
The fix is to merge the two `kubelet set` commands into one.
A more principled fix would be to not start kubelet at all until it's configured properly.