Comment 2 for bug 1838692

Revision history for this message
David Sullivan (dsullivanwr) wrote :

I believe this issue stems from the addition of the kubelet-fs
https://opendev.org/starlingx/config/commit/e74ef5f7c4c71464347142932615e1545884133c

Digging into the issue kubelet fails to start because the config is missing.
localhost kubelet[103405]: info F0731 20:52:58.914248 103405 server.go:189] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kubelet/config.yaml: no such file or directory

As part of the kubelet-fs we mount the fs to /var/lib/kubelet. Unfortunately kubeadm reset will unmount this path. This means on replays the following will happen:
kubeadm reset -> unmounts kubelet-lv from /var/lib/kubelet
kubeadm init -> writes config to /var/lib/kubelet
kubeadm init -> starts kubelet
kubelet service -> mounts kubelet-lv to /var/lib/kubelet
kubelet service -> reads in config.yaml

In the case where the first run of kubeadm init failed, the kubelet-lv will be empty and the kubelet service will fail. If the first kubeadm init passed the contents of /var/lib/kubelet will be stale.

Probably a quick solution would be to remount kubelet-lv to /var/lib/kubelet between kubeadm reset and kubeadm init

See also
https://github.com/kubernetes/kubernetes/blob/v1.13.5/cmd/kubeadm/app/cmd/reset.go#L157