Comment 1 for bug 1941763

On AWS, the kubernetes-master charm is stuck because kubectl calls are failing, but only sometimes:

# kubectl get po
error: You must be logged in to the server (Unauthorized)
# kubectl get po
No resources found in default namespace.

This is happening because /root/.kube/config is pointing to the IP of kubeapi-load-balancer, which is distributing traffic between multiple kubernetes-master units. The two units do not agree on what the admin token is:

$ juju ssh kubernetes-master/0 sudo cat /root/.kube/config | grep token
    token: admin::RAO...c8r
$ juju ssh kubernetes-master/1 sudo cat /root/.kube/config | grep token
    token: admin::e2z...jYz

The auth webhook reads the local /root/.kube/config to determine what the admin token is, so, if the request from kubernetes-master/0 lands on kubernetes-master/1 or vice versa, the request fails.

The two kubernetes-master units are unable to progress to the point where they eventually agree on what the token is. They try and fail, repeatedly, to create the admin token as a Kubernetes secret:

unit-kubernetes-master-1: 13:09:43 INFO unit.kubernetes-master/1.juju-log WARN: Unable to create secret for admin
unit-kubernetes-master-0: 13:10:20 INFO unit.kubernetes-master/0.juju-log WARN: Unable to create secret for admin
...
unit-kubernetes-master-0: 18:03:11 INFO unit.kubernetes-master/0.juju-log WARN: Unable to create secret for admin
unit-kubernetes-master-1: 18:18:43 INFO unit.kubernetes-master/1.juju-log WARN: Unable to create secret for admin

In the past, this wasn't a problem because kubernetes-master kubectl requests went straight to the local IP, not kubeapi-load-balancer. So the request would always land on the local unit, where the local admin token is guaranteed to work.

We'll need to either revert kubernetes-master back to using its local IP, or fix the admin token handling to be less reliant on a successful connection to the Kubernetes API.