Removal of leader unit may result in loss of keystone-policy configmap
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Kubernetes Control Plane Charm |
Triaged
|
Medium
|
Unassigned |
Bug Description
I recently had to redeploy a kubernetes-master unit's host system in an environment which is using K8s keystone integration.
While in the process of redeploying the host in question, it was found that keystone authentication had broken. I did a "kubectl rollout restart -n kube-system deployment.
E0419 18:09:33.449632 1 main.go:62] failed to get configmap k8s-auth-policy: configmaps "k8s-auth-policy" not found
I believe this got deleted during teardown of the kubernetes-master unit. I did take a quick look at code, and it looks like if this happened to be the leader node, it may have went ahead and removed that configmap. I'm not 100% sure though.
Also, upon finally re-adding the unit, this configmap did not get recreated. I ultimately pulled the keystone-policy out and manually applied it in order to resolve the issue.
Looks like this will happen if the kubernetes-master leader unit is removed, and leadership is handed back to another kubernetes-master that once had leadership in the past.
The dying leader removes the keystone policy when it departs from the keystone- credentials relation[1].
The newly elected leader only creates a new policy if the kubernetes- master. keystone- policy- handled flag isn't set[2]. If the unit had been a leader once in the past, then the flag is already set, so it won't recreate the policy.
[1]: https:/ /github. com/charmed- kubernetes/ charm-kubernete s-master/ blob/d0ba96c434 7b143fdde040dd0 b41d5e8d84f943d /reactive/ kubernetes_ master. py#L3008 /github. com/charmed- kubernetes/ charm-kubernete s-master/ blob/d0ba96c434 7b143fdde040dd0 b41d5e8d84f943d /reactive/ kubernetes_ master. py#L2978
[2]: https:/