incorrect controller address for cross model consumers

[Ian Booth]

For a cross model relation, the consuming controller queries the offering controller to find out the IP addresses it should connect to in order to access the offer. There's 2 APIs available:

// APIHostPortsForAgents returns the collection of API addresses that should
// be used by agents.
// If the controller model is CAAS type, the return will be the controller
// k8s service addresses in cloud service.
// If there is no management network space configured for the controller,
// or if the space is misconfigured, the return will be the same as
// APIHostPortsForClients.
// Otherwise the returned addresses will correspond with the management net space.
// If there is no document at all, we simply fall back to APIHostPortsForClients.
func (st *State) APIHostPortsForAgents() ([]network.SpaceHostPorts, error)

or

// APIHostPortsForClients returns the collection of *all* known API addresses.
func (st *State) APIHostPortsForClients() ([]network.SpaceHostPorts, error)

The API being used is APIHostPortsForAgents(). As well as filtering on the mgmt space, this API on k8s also adds in the local FQDN of the controller service.

For k8s offers, the effect is that the consuming side tries to resolve the FQDN address and fails, causing workers to bounce. On k8s, there's no space support so the only difference between the agent and client addresses is the FQDN. On machine controllers, this has the possibility to break CMR if the consuming controller can't reach addresses on the offering controller mgmt space.

Where offer and consumer are in different controllers, in the absence of extended network modelling which Juju does not yet support, it should default to choosing the most public addresses available, which means APIHostPortsForClients(). ForCMR where the models are in the same controller, we can still use the agent addresses.

[John A Meinel (jameinel)]

For cross model interactions across controllers, I think it is correct to model the connection to the remote controller as a "Client" connection, not as an "Agent" connection. (APIHostPortsForAgents is very much for things that are in the "same" location as the controller, and thus expected that they could route to the controllers at `juju-ha-space`.

If you consider a CMR that had one controller in GCE and one controller in AWS, I would expect that the agents running in GCE would access the controller in an HA space that was local to GCE, and the agents in AWS would access the controller in the HA space that was local to AWS, but that the GCE controller should reach the AWS controller as a normal 'client' connection.

[John A Meinel (jameinel)]

That said, if we have a list of addresses to connect to, and one of them doesn't resolve, that shouldn't prevent us from using the others. So we might have a different error handling problem where we are treating a "this address isn't for us to use" as a fatal error.

[Joseph Phillips (manadart)]

See apiserver/facades/controller/crosscontroller/crosscontroller.go

We register with this func:

func() ([]string, string, error) {
    return common.StateControllerInfo(st)
}

In that method we call the common apiAddresses function, which in turn uses APIHostPortsForAgents.

We should contrive a function that calls APIHostPortsForClients since we specifically know that this is a cross-controller API.

[Joseph Phillips (manadart)]

https://github.com/juju/juju/pull/17594