Fan network's subnets not allowed to connect to database

[Pedro Guimarães]

Hi,

Full juju status: https://pastebin.canonical.com/p/DP96fkRz9y/
Bundle: https://pastebin.canonical.com/p/2cvJcqxS6p/

I am currently running a mysql/keystone/horizon/vault deployment to provide ldap connection on top of a private cloud.

I am using fan-networking for this deployment.

Mysql runs on top of LXCs whereas keystone / vault / horizon may run on top of lxcs or directly on VMs. However, units running on VMs directly cannot connect to the mysql cluster running on top of LXC + FAN net.

These routers get stuck in:

keystone-mysql-router/1 waiting - (config-changed) MySQL Router not yet bootstrapped

With following flags are present in the unit above:
charm.installed
charm.is-subordinate
charm.mysqlrouter.bootstrap-attempted <<<<<<---------
charms.openstack.do-default-charm.installed
charms.openstack.do-default-config.changed
charms.openstack.do-default-update-status
charms.openstack.do-default-upgrade-charm
db-router.available
db-router.connected
endpoint.shared-db.changed
endpoint.shared-db.changed.egress-subnets
endpoint.shared-db.changed.ingress-address
endpoint.shared-db.changed.private-address
endpoint.shared-db.joined
keystone-mysql-router-installed
shared-db.available

Looking into mysql router code, this is the line it gets stuck:

https://github.com/openstack/charm-mysql-router/blob/master/src/lib/charm/openstack/mysql_router.py#L411-L466

And indeed, the cmd generated above uses a fan-networking IP:
sudo /usr/bin/mysqlrouter --user mysql --name keystone-mysql-router --bootstrap mysqlrouteruser:<PASSWORD>@252.153.13.224 --directory /var/lib/mysql/keystone-mysql-router --conf-use-sockets --conf-bind-address 127.0.0.1 --report-host <HOST_IP> --conf-base-port 3306 --disable-rest --force

full command + output:
https://pastebin.canonical.com/p/z6B27qpF8v/

Whereas the relation between mysql cluster and that same unit shows:
https://pastebin.canonical.com/p/ftzFsfrvjp/

Which shows ingress-address seen by mysql-innodb-cluster is actually the host's IP and not fan-networking IP.

That actually exposes another issue, in which mysql-innodb-cluster cannot be used outside of Juju relations given there is no way to manually edit the access list.

Mysql cluster charm should expose an option to add subnets to this access list, e.g.:

$ juju config mysql-innodb-cluster access-list="252.0.0.0/8,10.0.0.0/16"

[Pedro Guimarães (pguimaraes)]

If I replace --report-host by its FAN IP, I get:

# sudo /usr/bin/mysqlrouter --user mysql --name keystone-mysql-router --bootstrap mysqlrouteruser:<PASSWORD>@252.153.13.224 --directory /var/lib/mysql/keystone-mysql-router --conf-use-sockets --conf-bind-address 127.0.0.1 --report-host 252.153.11.1 --conf-base-port 3306 --disable-rest --force
Error: Unable to connect to the metadata server: Error connecting to MySQL server at 252.153.13.224:0: Access denied for user 'mysqlrouteruser'@'252.153.11.1' (using password: YES) (1045)

Seems mysql cluster is not accepting clients coming from different subnets.

[Pedro Guimarães (pguimaraes)]

This issue is a blocker for any deployment running fan-networking or if we are running mysql-router across different subnets.

[Pedro Guimarães (pguimaraes)]

Raising as field-high, only work-around currently available is to drop fan-networking and colocate all workloads on the same VMs.

[Billy Olsen (billy-olsen)]

The charms in the OpenStack charms collection are not tested with Fan Networking enabled within Juju and this is outside the bounds of what is known to work and have been tested. I will evaluate this bug from here forward as a wishlist item/feature request to support Fan Networking in the charm.

Generally speaking, the charms use the juju network spaces to influence which interface the charm will configure for communication methods.

In the case of fan networking, juju does not allow moving the overlay fan network to a different space and considers it in the same space as the underlay network. As such, the overlay and underlay networks are part of the same logical space.

Attempts to move it result in:

$ juju move-to-space database 252.0.0.0/8
ERROR cannot update space "database": subnet "252.0.0.0/8" is a fan overlay of "10.5.0.0/16" and cannot be moved; move the underlay instead

More analysis later, but the code uses the network-get --primary-address <relation> which does not return secondary interfaces (such as fan-networking) here. Additionally, none of the network-get calls (aside from a bare network-get --format yaml <relation>) will return the secondary interface. In this scenario, since Juju is aware that there are containers that are using fan networking and I believe Juju should return the appropriate interface to use across this relation.

[Billy Olsen (billy-olsen)]

Removing field-high designation. I understand the need for this, but as previously stated this is being evaluated as a wishlist item/feature request.