Kubernetes workers constantly in and out of maintenance

[Hans van den Bogert]

Since a full cluster restart I am experiencing the following behaviour, please see the asciinema since this says more than a 1000 words

https://asciinema.org/a/lglCBQvT1ngisNUcHaS8IX1Oe

juju debug-log does not seem to give any indication why this is happening, though added the last ~20k lines anyway.

[George Kraft (cynerva)]

In case we lose the asciinema, the symptom is that kubernetes-worker units are repeatedly bouncing leader-settings-changed hooks with status message "Joining snap cohort." appearing. kubernetes-worker charm revision 692, cloud is MAAS.

[George Kraft (cynerva)]

Can you provide debug-log for kubernetes-worker-86 as well? That's the leader unit and I think we'll need to see what it is doing.

> Since a full cluster restart

If I understand correctly, you rebooted the hosts?

[Hans van den Bogert (hbogert)]

> If I understand correctly, you rebooted the hosts?

Yes, the physical nodes were rather harshly rebooted, only on Kubernetes level were the nodes drained.

Added logs for kubernetes-worker-86.

[Hans van den Bogert (hbogert)]

for archival purposes

[Cory Johns (johnsca)]

This seems to be due to the interplay between the k8s-master treating the cohort update of all snaps as a single group[1] and both the k8s-master[2] and k8s-worker[3] using a workaround for the fact that the initial snap install is not deferred until a set of cohort keys is decided upon (which, in turn, stems from trying to minimize the already large impact the refactor to add cohort support had). This also touches on the cohort update delay discussed in the team daily on Aug. 21 and should perhaps be handled together.

Specifically, the k8s-master determines if new cohort keys need to be created based on whether a new rev of any of the master or worker snaps is available. This runs into trouble when the master asks if a refresh is available for a snap that it doesn't have installed (i.e., kubelet); since it's not installed, it won't see a refresh being available and won't generate a new cohort key, whereas the workers will see the available refresh due to the workaround and will cycle between themselves trying to get it installed. The k8s-master's decision to create a new cohort key should instead be driven by the current vs recorded revision for each snap and how long the current revision has been available vs the refresh timer config.

A workaround to recover from this state is to run the following:

juju run --unit kubernetes-master/leader -- 'leader-set cohort_keys=; hooks/leader-settings-changed'

[Hans van den Bogert (hbogert)]

#6 worked. Fun fact, my cluster idle wattage reduced from 150w to 100w.

[Cory Johns (johnsca)]

Forgot to add the code reference links in my previous comment:

[1]: https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/master/reactive/kubernetes_master.py#L424-L428

[2]: https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/master/reactive/kubernetes_master.py#L442-L450

[3]: https://github.com/charmed-kubernetes/charm-kubernetes-worker/blob/master/reactive/kubernetes_worker.py#L291-L299

[Kevin W Monroe (kwmonroe)]

Master PR for review:
https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/125

[Cory Johns (johnsca)]

Small addition: https://github.com/charmed-kubernetes/charm-kubernetes-master/pull/129