Kubernetes Multicloud controller Passing wrong IP to external Kubernetes

[Dominik Fleischmann]

I'm using a kubernetes controller on microk8s on which I'm adding an additional k8s. When deploying workloads on the external kubernetes the deployment stays in allocating. While the controller.yaml in .local/share/juju shows this api-endpoint:
 api-endpoints: ['172.21.248.27:17070']

the agent.conf file in th operator pod of the deployed workload (in external k8s) shows this one:
 apiaddresses:
 - 10.152.183.83:17070

This is the deployment of the controller:
NAME READY STATUS RESTARTS AGE
pod/controller-0 2/2 Running 1 25m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/controller-service LoadBalancer 10.152.183.83 172.21.248.27 17070:31851/TCP 25m

NAME READY AGE
statefulset.apps/controller 1/1 25m

Steps to reproduce this:

First machine:
sudo snap install microk8s --classic
microk8s.enable dns storage metallb
juju bootstrap microk8s lk8s --config controller-service-type=loadbalancer

Second machine:
sudo snap install microk8s --classic
microk8s.enable dns storage metallb
microk8s.config

First machine:
cat <external-k8s-config> | juju add-k8s ek8s
juju add-model test ek8s
juju deploy cs:~dominik.f/bundle/hf-k8s-bundle-0

[Ian Booth (wallyworld)]

The lack of modelling of controller shadow addresses is a current Juju limitation affecting not just k8s controllers. The same would issue apply if the controller were deployed in a lxd container on say aws, or behind a nat etc.

This is a significant feature we'll need to add to the juju roadmap, since any solution will probably involve additions to network spaces modelling, need to account for in cluster vs out of cluster addressing etc etc.

[Ian Booth (wallyworld)]

Something which may work...

Maybe create the loadbalancer first, and then bootstrap with a config for controller-external-ips set to the address of the load balancer. It's an extra manual step but might work??

For reference, here's a post describing the available bootstrap options for k8s controllers

https://discourse.juju.is/t/new-features-and-changes-in-juju-2-7/2268

[Joseph Phillips (manadart)]

The addresses are correctly discovered by the CAAS provider, and are stored in the cloudservice record for the controller/API server.

I ran up a controller on microk8s with metallb to verify this.

The issue would appear to be the boolean passed to apiHostPortsForCAAS. We *always* do a public scope match for clients, and a local-cloud match for agents.

If we returned *all* of the addresses to agents, then they would connect with one they could access.

I would simply order them preferring local-cloud, which would mean agents attempt to connect those first, and then fall back to the public scope.

We don't yet have a good way to lasso external subnets in spaces (such as shadow IPs), but if/when that was modelled correctly and we instituted space support for CAAS, then enforcing the address range to use could be done with the juju-mgmt-space config option.

But for now, it seems like an easy win to give the agents the info they might need.

[John A Meinel (jameinel)]

I thought for CMR we return public IP addresses for agents based on the same issue. (so if a model isn't in the same space as the controller, then we use the public address instead of the private.)
That feels like it would be true everywhere (on AWS between 2 different VPC, you need to use the public address just the same, 172.* won't be routable.)

[Ian Booth (wallyworld)]

@jam, there's no logic yet for handing out different agent api addresses depending on the context. We do it for CMR but the code paths are totally different; it's in the context of populating relation data, and for a cross model relation, we simply punt on always using the public address; we don't / can't check yet to see if a cloud local address will do as we either don't model it or don't interpret it if we do.

[Ian Booth (wallyworld)]

Joe had an excellent suggestion which is easy to implement for now even without space support.

https://github.com/juju/juju/pull/11398

So we should have something working for 2.8