k8s workers are failling to connect to k8s master during kube-api-endpoint-changed hook

[Alexander Balderson]

Deploying a bare metal k8s with ha cluster for kubernets master. The deployment sets the vip for k8s master, and the kubernetes-master-loadbalancer-ip. It is also using vault, with auto generated certificates. During the deployment, the k8s workers are able to connect to the master over the vip for a while, and then suddenly they start getting connection refused, and unauthorized against the loadbalancer-ip.

The error came to light when the test_dns_provider also got an unauthorized trying to connect to the loadbalancer-ip.

Is it possible that the ip moved hosts, and the generated certificate from vault was no longer valid?

The logs for the run can be found:
https://oil-jenkins.canonical.com/artifacts/84ea5313-8012-4c51-8f15-b53b359d6bbe/index.html
and the crashdump at:
https://oil-jenkins.canonical.com/artifacts/84ea5313-8012-4c51-8f15-b53b359d6bbe/generated/generated/kubernetes/juju-crashdump-kubernetes-2021-06-21-10.29.01.tar.gz

Some snippets from the k8s worker logs:

2021-06-21 10:18:20 WARNING unit.kubernetes-worker/0.kube-api-endpoint-relation-changed logger.go:60 error: You must be logged in to the server (Unauthorized)
2021-06-21 10:18:21 WARNING unit.kubernetes-worker/0.kube-api-endpoint-relation-changed logger.go:60 error: You must be logged in to the server (Unauthorized)

2021-06-21 10:12:26 WARNING unit.kubernetes-worker/0.kube-api-endpoint-relation-changed logger.go:60 The connection to the server 10.246.64.82:6443 was refused - did you specify the right host or port?
2021-06-21 10:12:27 WARNING unit.kubernetes-worker/0.kube-api-endpoint-relation-changed logger.go:60 The connection to the server 10.246.64.82:6443 was refused - did you specify the right host or port?

and test_dns_provider with:

ERROR failed to open kubernetes client: unable to determine legacy status for namespace kubernetes-k8s: Get "https://10.246.64.82:6443/api/v1/namespaces/kubernetes-k8s": Forbidden