Comment 0 for bug 1897115

juju version: 2.8.3-bionic-amd64
provider: vSphere

Charmed Kubernetes deployment doesn't settle and is not usable with "waiting: Waiting to retry addon deployment" in kubernetes-master. The root cause is Juju believes that /32 addresses from an vxlan interface of flannel in kubernetes-master units are the ones other services should use instead of the "main" /26 addresses the units have. By the nature of /32 in IPv4, nobody can reach to it.

kubernetes-master has the "main" and flannel/32 address:

$ juju run --unit kubernetes-master/leader -- ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
ens192 UP 133.XX.XX.XX/26 fe80::XXXX:XXXX:XXXX:XXXX/64
flannel.1 UNKNOWN 10.1.75.0/32 fe80::YYYY:YYYY:YYYY:YYYY/64

network-get returns 10.1.75.0/32 which is not reachable from other units:

$ juju run --unit kubernetes-master/leader -- network-get kube-api-endpoint --ingress-address
10.1.75.0

Other services such as kubeapi-loadbalancer writes the following config, then Nginx cannot reach to the backend servers.

$ cat /etc/nginx/sites-enabled/apilb
upstream target_service {
  server 10.1.75.0:6443;
  server 10.1.8.0:6443;
}

And there is no way to specify which subnet to be used explicitly because network spaces are not supported in vSphere provider as far as I'm concerned.

$ juju spaces
cannot list spaces: spaces not supported (not supported)
ERROR cannot list spaces: spaces not supported (not supported)

Here is the full output of network-get, and it doesn't contain the expected /26 address at all.

$ juju run --unit kubernetes-master/leader -- network-get kube-api-endpoint
bind-addresses:
- macaddress: fa:73:41:53:bd:2c
  interfacename: flannel.1
  addresses:
  - hostname: ""
    address: 10.1.75.0
    cidr: 10.1.75.0/32
egress-subnets:
- 10.1.75.0/32
ingress-addresses:
- 10.1.75.0

FWIW, "primary-network" is already set in model-config.