ha-cluster-vip not correctly written to kubeconfig

[Jake Nabasny]

== SUMMARY ==

When either the "ha-cluster-vip" or "loadbalancer-ips" options are set in the kubeapi-load-balancer charm, it writes the VIP to the kubeconfigs on the kubernetes-worker units with brackets "[]" around it (e.g. https://[10.100.0.99]:6443). This causes a critical problem because the kube-proxy cannot parse this URL and fails to start. This leads to the kubelet failing and receiving on error in the juju status: "Waiting for kubelet to start". The related error from the kube-proxy snap is:

kube-proxy.daemon[77861]: W0424 20:53:28.522475 77861 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get "https://['10.100.0.99']:6443/api/v1/nodes?fieldSelector=metadata.name%3Djuju-df0e12-2&limit=500&resourceVersion=0": dial tcp: lookup '10.100.0.99': no such host

== EXAMPLE BUNDLE CONFIGURATION ==

    options:
      ha-cluster-vip: "10.100.0.99"
      loadbalancer-ips: "10.100.0.99"

== WORKAROUND ==

Update the proxy-extra-args parameter on the kubernetes-control-plane and kubernetes-worker charms with:

juju config kubernetes-control-plane proxy-extra-args=master=https://10.100.0.99:6443
juju config kubernetes-worker proxy-extra-args=master=https://10.100.0.99:6443

Then reboot the worker units.

[Navdeep (navdeep-bjn)]

This seems to be a retrogression since the same setting used to work with charmed k8s 1.21 and is causing an issue when I deploy with the 1.24 channel

[Adam Dyess (addyess)]

It doesn't seem possible to reproduce this with 1.28/edge channel charms.

can you provide the charm revisions used from `juju status`

[Navdeep (navdeep-bjn)]

App Version Status Scale Charm Channel Rev Exposed Message

kubeapi-load-balancer 1.18.0 active 3 kubeapi-load-balancer 1.24/stable 27 yes Loadbalancer ready.

The kubeconfig files on the worker nodes also have the same square brackets in the server line

$ sudo grep http /root/cdk/ -R
/root/cdk/kubeproxyconfig: server: https://[10.100.0.99]:6443
/root/cdk/kubeproxyconfig.new: server: https://[10.100.0.99]:6443
/root/cdk/kubeconfig: server: https://[10.100.0.99]:6443
/root/cdk/kubeconfig.new: server: https://[10.100.0.99]:6443

[Adam Dyess (addyess)]

I've reproduced now!

It's vital to use the following relations

kubeapi-load-balancer:lb-consumers kubernetes-control-plane:loadbalancer-external loadbalancer regular
kubeapi-load-balancer:lb-consumers kubernetes-control-plane:loadbalancer-internal loadbalancer regular

[Adam Dyess (addyess)]

Possible solution:
https://github.com/charmed-kubernetes/charm-kubeapi-load-balancer/pull/20

Looking for a review

[Navdeep (navdeep-bjn)]

Adam,

I am using the following relations in my bundle file. do I need any other relations?

- - kubernetes-control-plane:loadbalancer-external
  - kubeapi-load-balancer:lb-consumers
- - kubernetes-control-plane:loadbalancer-internal
  - kubeapi-load-balancer:lb-consumers

[George Kraft (cynerva)]

Navdeep, those should be the only relations you need between kubeapi-load-balancer and kubernetes-control-plane.

You can download the reference bundle to see what other relations may be expected for a given release of Charmed Kubernetes. For example:

juju download charmed-kubernetes --channel 1.27/stable
unzip charmed-kubernetes*.bundle -d charmed-kubernetes
cat charmed-kubernetes/bundle.yaml

[Adam Dyess (addyess)]

Demonstration of functionality in ck-1.24, 1.25, and 1.26
https://paste.ubuntu.com/p/Cp7NtPHnrb/

[Navdeep (navdeep-bjn)]

I verified the fix in one of our test clusters. Please let me now once it is available in the stable branch.

[Francesco De Simone (fdesi)]

Hi, is there an ETA for when this fix will be released?

[George Kraft (cynerva)]

This fix is currently available in the 1.27/candidate charm channel. We're still working to get this promoted to 1.27/stable but we're having some issues with our test suite. It's hard to provide an ETA given the nature of these issues so the best I can offer is: hopefully within the next couple of days.