[Master, API LB] VIP hostname does not get propagated to kubeconfig

[Pedro Guimarães]

Looking into how kubernetes cluster VIP / hostname gets propagated over from keepalived down to kubernetes-master, we can see:

  kubeapi-load-balancer:
    charm: cs:~containers/kubeapi-load-balancer
    expose: true
    options:
      extra_sans: *kubeapi-lb-url
  keepalived:
    charm: cs:~containers/keepalived
    options:
      virtual_ip: *kubeapi-lb-vip
      vip_hostname: *kubeapi-lb-url
      port: 443
  kubernetes-master:
    charm: cs:~containers/kubernetes-master
    num_units: 2
    options:
      extra_sans: *kubeapi-lb-url

In that configuration, as described on: https://jaas.ai/u/containers/keepalived, we will get keepalived & kube-api-loadbalancer correctly configured to work with both VIP & hostname. However, kubernetes-master is not aware of cluster fqdn, since it did not receive any information except extra_sans.

Therefore, kubeconfig will have:
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <CERTIFICATE>
    server: https://<IP>:443
. . . .

Looking into how kubernetes-master builds up kubeconfig file: https://github.com/charmed-kubernetes/charm-kubernetes-master/blob/8b365975b12d09984a99fc8852f04d9b2bdf4256/reactive/kubernetes_master.py#L1100

We can see that actually only IPs are passed around to build_kubeconfig, which creates the file itself.

The issue is that, on configuration described at https://jaas.ai/u/containers/keepalived, we define extra_sans only for VIP hostname, not for VIP itself. Then, when using the kubeconfig generated, we see an error:

$ kubectl get nodes
Unable to connect to the server: x509: certificate is valid for <LIST OF IPS, BUT NO VIP>

That happens because our kubeconfig reaches out kube-apiserver using the VIP on its http request, but extra_sans was configured only for hostname.

I believe the most appropriate way is to have a vip_hostname option on both kubernetes-master & kube_api_loadbalancer.

[Pedro Guimarães (pguimaraes)]

If I set extra_sans on both kubernetes-master & api loadbalancer to have:

extra_sans: <VIP> <HOSTNAME>

Then it works.
I believe an alternative is to fix keepalived README to advise to include both on extra_sans parameter.

For me, this is a work-around. I still think we should ditch VIP if we have a hostname configured for it. That will spare deployments where we have, for example, OpenStack provider and the VIP changes every time since ports get created and destroyed on-demand.

[George Kraft (cynerva)]

Can we see your bundle, or output from `juju status --format yaml`? We'll need to see what kubernetes-master:loadbalancer is related to.

[Felipe Reyes (freyes)]

@cynerva, this is the bundle I use to reproduce the issue https://paste.ubuntu.com/p/tjSgkmQ7SZ/ , I ended up with this bundle following the set of commands described in keepalived charm's readme.

[Felipe Reyes (freyes)]

ubuntu@juju-04930e-5:~$ export KUBECONFIG=config
ubuntu@juju-04930e-5:~$ kubectl get pods
Unable to connect to the server: x509: certificate is valid for 192.168.10.211, 192.168.10.211, not 192.168.10.34
ubuntu@juju-04930e-5:~$ grep server config
    server: https://192.168.10.34:443
ubuntu@juju-04930e-5:~$ sed -i s/192.168.10.34/loadbalancer.lan/g config
ubuntu@juju-04930e-5:~$ grep server config
    server: https://loadbalancer.lan:443
ubuntu@juju-04930e-5:~$ host loadbalancer.lan
loadbalancer.lan has address 192.168.10.34
ubuntu@juju-04930e-5:~$ kubectl get pods
No resources found in default namespace.
ubuntu@juju-04930e-5:~$ kubectl get componentstatuses
Unable to connect to the server: x509: certificate is valid for 192.168.10.211, 192.168.10.211, not 192.168.10.34
ubuntu@juju-04930e-5:~$ grep server config
    server: https://192.168.10.34:443
ubuntu@juju-04930e-5:~$ sed -i s/192.168.10.34/loadbalancer.lan/g config
ubuntu@juju-04930e-5:~$ kubectl get componentstatuses
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-2 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}

[George Kraft (cynerva)]

Thanks for the details. Indeed, it looks like the keepalived charm sends the VIP, not the hostname, over the loadbalancer relation: https://github.com/juju-solutions/charm-keepalived/blob/b1facc0f540d98f9cf4c8ccfea4c8abd9ce796f1/src/reactive/keepalived.py#L100

[Pedro Guimarães (pguimaraes)]

Hi @cynerva, hacluster also has the same effect and we orientation is to use hacluster instead of keepalived.

When we look for OpenStack bundles, each openstack service has a os-*-hostname option (where * is public, private or admin).
That is set on the charms themselves. Maybe, that should be the approach for k8s charms as well.

[George Kraft (cynerva)]

@pguimaraes Can you provide more details on the issue with hacluster?

Were you using hacluster with or without kubeapi-load-balancer?

Did you configure ha-cluster-vip, ha-cluster-dns, or both?

Are you able to work around it with extra_sans?