We found that all the worker nodes on the cloud were status NotReady. I tried restarting one, which fixed it, using 'snap restart kubelet'. Here's the output after:
ubuntu@juju-1b688f-somecloud-k8s-5:/var/log/juju$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
juju-1b688f-somecloud-k8s-10 NotReady <none> 29d v1.13.5
juju-1b688f-somecloud-k8s-11 Ready <none> 29d v1.13.6
juju-1b688f-somecloud-k8s-12 NotReady <none> 29d v1.13.5
juju-1b688f-somecloud-k8s-7 NotReady <none> 29d v1.13.5
juju-1b688f-somecloud-k8s-8 NotReady <none> 29d v1.13.5
juju-1b688f-somecloud-k8s-9 NotReady <none> 29d v1.13.5
Restarted all the kubelets:
ubuntu@juju-1b688f-somecloud-k8s-5:/var/log/juju$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
juju-1b688f-somecloud-k8s-10 Ready <none> 29d v1.13.6
juju-1b688f-somecloud-k8s-11 Ready <none> 29d v1.13.6
juju-1b688f-somecloud-k8s-12 Ready <none> 29d v1.13.6
juju-1b688f-somecloud-k8s-7 Ready <none> 29d v1.13.6
juju-1b688f-somecloud-k8s-8 Ready <none> 29d v1.13.6
juju-1b688f-somecloud-k8s-9 Ready <none> 29d v1.13.6
Before the restart, the logs were full of this:
May 29 03:50:53 juju-1b688f-somecloud-k8s-10 kubelet.daemon[30866]: I0529 03:50:53.872978 30866 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 29 03:51:03 juju-1b688f-somecloud-k8s-10 kubelet.daemon[30866]: I0529 03:51:03.889760 30866 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 29 03:51:13 juju-1b688f-somecloud-k8s-10 kubelet.daemon[30866]: I0529 03:51:13.907342 30866 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 29 03:51:23 juju-1b688f-somecloud-k8s-10 kubelet.daemon[30866]: I0529 03:51:23.924960 30866 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
May 29 03:51:33 juju-1b688f-somecloud-k8s-10 kubelet.daemon[30866]: I0529 03:51:33.944682 30866 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Restarted kubelet:
May 29 03:51:40 juju-1b688f-somecloud-k8s-10 systemd[1]: Stopping Service for snap application kubelet.daemon...
May 29 03:51:40 juju-1b688f-somecloud-k8s-10 systemd[1]: Stopped Service for snap application kubelet.daemon.
May 29 03:51:40 juju-1b688f-somecloud-k8s-10 systemd[1]: Started Service for snap application kubelet.daemon.
May 29 03:51:40 juju-1b688f-somecloud-k8s-10 kubelet.daemon[21661]: I0529 03:51:40.371776 21661 controller.go:101] kubelet config controller: starting controller
May 29 03:51:40 juju-1b688f-somecloud-k8s-10 kubelet.daemon[21661]: I0529 03:51:40.371996 21661 controller.go:267] kubelet config controller: ensuring filesystem is set up correctly
May 29 03:51:40 juju-1b688f-somecloud-k8s-10 kubelet.daemon[21661]: I0529 03:51:40.372017 21661 fsstore.go:59] kubelet config controller: initializing config checkpoints directory "/root/cdk/kubelet/dynamic-conf
May 29 03:51:40 juju-1b688f-somecloud-k8s-10 kubelet.daemon[21661]: I0529 03:51:40.378500 21661 server.go:407] Version: v1.13.6
May 29 03:51:40 juju-1b688f-somecloud-k8s-10 kubelet.daemon[21661]: W0529 03:51:40.378650 21661 plugins.go:118] WARNING: openstack built-in cloud provider is now deprecated. Please use 'external' cloud provider¤
May 29 03:51:41 juju-1b688f-somecloud-k8s-10 kubelet.daemon[21661]: I0529 03:51:41.407589 21661 controller.go:207] kubelet config controller: local source is assigned, will not start remote config source informe
May 29 03:51:41 juju-1b688f-somecloud-k8s-10 kubelet.daemon[21661]: I0529 03:51:41.407780 21661 controller.go:197] kubelet config controller: starting status sync loop
May 29 03:51:41 juju-1b688f-somecloud-k8s-10 kubelet.daemon[21661]: I0529 03:51:41.407809 21661 status.go:145] kubelet config controller: updating Node.Status.Config
May 29 03:51:41 juju-1b688f-somecloud-k8s-10 kubelet.daemon[21661]: I0529 03:51:41.407809 21661 controller.go:226] kubelet config controller: starting Node informer
May 29 03:51:41 juju-1b688f-somecloud-k8s-10 kubelet.daemon[21661]: I0529 03:51:41.407828 21661 controller.go:231] kubelet config controller: starting Kubelet config sync loop
May 29 03:51:41 juju-1b688f-somecloud-k8s-10 kubelet.daemon[21661]: I0529 03:51:41.426329 21661 watch.go:89] kubelet config controller: initial Node watch event
... and so on.
The snap refresh policy is mon5, I can't see anything in 'snap changes' other than the service restart, but I do know that there was a recent release in snaps and we just hit past a last Monday in the month.
