Removal of leader unit may result in loss of keystone-policy configmap

[Paul Goins]

I recently had to redeploy a kubernetes-master unit's host system in an environment which is using K8s keystone integration.

While in the process of redeploying the host in question, it was found that keystone authentication had broken. I did a "kubectl rollout restart -n kube-system deployment.apps/k8s-keystone-auth", and saw that the newly created pods had errors, e.g.:

  E0419 18:09:33.449632 1 main.go:62] failed to get configmap k8s-auth-policy: configmaps "k8s-auth-policy" not found

I believe this got deleted during teardown of the kubernetes-master unit. I did take a quick look at code, and it looks like if this happened to be the leader node, it may have went ahead and removed that configmap. I'm not 100% sure though.

Also, upon finally re-adding the unit, this configmap did not get recreated. I ultimately pulled the keystone-policy out and manually applied it in order to resolve the issue.

[George Kraft (cynerva)]

Looks like this will happen if the kubernetes-master leader unit is removed, and leadership is handed back to another kubernetes-master that once had leadership in the past.

The dying leader removes the keystone policy when it departs from the keystone-credentials relation[1].

The newly elected leader only creates a new policy if the kubernetes-master.keystone-policy-handled flag isn't set[2]. If the unit had been a leader once in the past, then the flag is already set, so it won't recreate the policy.

[1]: https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/d0ba96c4347b143fdde040dd0b41d5e8d84f943d/reactive/kubernetes_master.py#L3008
[2]: https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/d0ba96c4347b143fdde040dd0b41d5e8d84f943d/reactive/kubernetes_master.py#L2978