Document need for db-router binding

[Seth Tanner]

some mysql-router units bootstrap, but not all of them

Model Controller Cloud/Region Version SLA Timestamp
openstack trc-default TRC/default 2.7.6 unsupported 10:34:50-07:00

App Version Status Scale Charm Store Rev OS Notes
ceph-mon 15.2.1 active 3 ceph-mon jujucharms 48 ubuntu
ceph-osd 15.2.1 active 6 ceph-osd jujucharms 303 ubuntu
ceph-radosgw 15.2.1 waiting 1 ceph-radosgw jujucharms 288 ubuntu
cinder 16.0.0 waiting 1 cinder jujucharms 303 ubuntu
cinder-ceph 16.0.0 active 1 cinder-ceph jujucharms 256 ubuntu
cinder-mysql-router 8.0.20 waiting 1 mysql-router jujucharms 0 ubuntu
dashboard-mysql-router 8.0.20 active 1 mysql-router jujucharms 0 ubuntu
glance 20.0.0 waiting 1 glance jujucharms 297 ubuntu
glance-mysql-router 8.0.20 waiting 1 mysql-router jujucharms 0 ubuntu
keystone 17.0.0 waiting 1 keystone jujucharms 314 ubuntu
keystone-mysql-router 8.0.20 waiting 1 mysql-router jujucharms 0 ubuntu
mysql-innodb-cluster 8.0.20 active 3 mysql-innodb-cluster jujucharms 0 ubuntu
mysql-router waiting 0 mysql-router jujucharms 0 ubuntu
neutron-api 16.0.0 waiting 1 neutron-api jujucharms 286 ubuntu
neutron-api-plugin-ovn 16.0.0 waiting 1 neutron-api-plugin-ovn jujucharms 0 ubuntu
neutron-mysql-router 8.0.20 waiting 1 mysql-router jujucharms 0 ubuntu
nova-cloud-controller 21.0.0 waiting 1 nova-cloud-controller jujucharms 345 ubuntu
nova-compute 21.0.0 active 5 nova-compute jujucharms 316 ubuntu
nova-mysql-router 8.0.20 waiting 1 mysql-router jujucharms 0 ubuntu
ntp 3.5 active 5 ntp jujucharms 39 ubuntu
openstack-dashboard 18.3.2 waiting 1 openstack-dashboard jujucharms 304 ubuntu
ovn-central 20.03.0 active 3 ovn-central jujucharms 0 ubuntu
ovn-chassis 20.03.0 active 5 ovn-chassis jujucharms 1 ubuntu
placement 3.0.0 waiting 1 placement jujucharms 11 ubuntu
placement-mysql-router 8.0.20 waiting 1 mysql-router jujucharms 0 ubuntu
rabbitmq-server 3.8.2 active 1 rabbitmq-server jujucharms 102 ubuntu
vault 1.1.1 active 1 vault jujucharms 39 ubuntu
vault-mysql-router 8.0.20 active 1 mysql-router jujucharms 0 ubuntu

Unit Workload Agent Machine Public address Ports Message
ceph-mon/0 active idle 0 10.100.190.1 Unit is ready and clustered
ceph-mon/2* active idle 2 10.100.190.3 Unit is ready and clustered
ceph-mon/3 active idle 14 10.100.190.2 Unit is ready and clustered
ceph-osd/0 active idle 3 10.100.190.7 Unit is ready (12 OSD)
ceph-osd/1 active idle 4 10.100.190.10 Unit is ready (12 OSD)
ceph-osd/2 active idle 5 10.100.190.8 Unit is ready (12 OSD)
ceph-osd/3* active idle 6 10.100.190.6 Unit is ready (12 OSD)
ceph-osd/4 active idle 7 10.100.190.9 Unit is ready (12 OSD)
ceph-osd/5 active idle 8 10.100.190.11 Unit is ready (12 OSD)
ceph-radosgw/0* waiting idle 12/lxd/0 10.100.113.11 80/tcp Incomplete relations: identity
cinder/0* waiting idle 12/lxd/1 10.100.113.0 8776/tcp Incomplete relations: database, identity
  cinder-ceph/0* active idle 10.100.113.0 Unit is ready
  cinder-mysql-router/0* waiting idle 10.100.113.0 MySQL Router not yet bootstrapped
glance/0* waiting idle 12/lxd/2 10.100.113.7 9292/tcp Incomplete relations: identity, database
  glance-mysql-router/0* waiting idle 10.100.113.7 MySQL Router not yet bootstrapped
keystone/0* blocked idle 13/lxd/0 10.100.113.8 5000/tcp Incomplete relations: database, Allowed_units list provided but this unit not present
  keystone-mysql-router/0* waiting idle 10.100.113.8 MySQL Router not yet bootstrapped
mysql-innodb-cluster/0* active idle 9/lxd/0 10.100.33.19 Unit is ready: Mode: R/O
mysql-innodb-cluster/1 active idle 10/lxd/0 10.100.33.7 Unit is ready: Mode: R/W
mysql-innodb-cluster/2 active idle 11/lxd/0 10.100.33.0 Unit is ready: Mode: R/O
neutron-api/0* waiting idle 12/lxd/3 10.100.113.13 9696/tcp Incomplete relations: identity, database
  neutron-api-plugin-ovn/0* waiting idle 10.100.113.13 'neutron-plugin' incomplete
  neutron-mysql-router/0* waiting idle 10.100.113.13 MySQL Router not yet bootstrapped
nova-cloud-controller/0* waiting idle 12/lxd/4 10.100.113.9 8774/tcp,8775/tcp Incomplete relations: database, identity
  nova-mysql-router/0* waiting idle 10.100.113.9 MySQL Router not yet bootstrapped
nova-compute/0 active idle 9 10.100.113.3 Unit is ready
  ntp/2 active idle 10.100.113.3 123/udp chrony: Ready
  ovn-chassis/2 active idle 10.100.113.3 Unit is ready
nova-compute/1 active idle 10 10.100.113.4 Unit is ready
  ntp/1 active idle 10.100.113.4 123/udp chrony: Ready
  ovn-chassis/1 active idle 10.100.113.4 Unit is ready
nova-compute/2 active idle 11 10.100.113.2 Unit is ready
  ntp/0* active idle 10.100.113.2 123/udp chrony: Ready
  ovn-chassis/0* active idle 10.100.113.2 Unit is ready
nova-compute/3* active idle 12 10.100.112.201 Unit is ready
  ntp/3 active idle 10.100.112.201 123/udp chrony: Ready
  ovn-chassis/3 active idle 10.100.112.201 Unit is ready
nova-compute/4 active idle 13 10.100.113.1 Unit is ready
  ntp/4 active idle 10.100.113.1 123/udp chrony: Ready
  ovn-chassis/4 active idle 10.100.113.1 Unit is ready
openstack-dashboard/0* waiting idle 13/lxd/1 10.100.113.14 80/tcp,443/tcp Incomplete relations: identity
  dashboard-mysql-router/0* active idle 10.100.113.14 Unit is ready
ovn-central/0* active idle 9/lxd/1 10.100.113.12 6641/tcp,6642/tcp Unit is ready (leader: ovnnb_db, ovnsb_db northd: active)
ovn-central/1 active idle 10/lxd/1 10.100.113.5 6641/tcp,6642/tcp Unit is ready
ovn-central/2 active idle 11/lxd/1 10.100.112.255 6641/tcp,6642/tcp Unit is ready
placement/0* waiting idle 13/lxd/2 10.100.113.6 8778/tcp 'shared-db' incomplete, 'identity-service' incomplete
  placement-mysql-router/0* waiting idle 10.100.113.6 MySQL Router not yet bootstrapped
rabbitmq-server/0* active idle 13/lxd/3 10.100.33.21 5672/tcp Unit is ready
vault/0* active idle 13/lxd/4 10.100.113.10 8200/tcp Unit is ready (active: true, mlock: disabled)
  vault-mysql-router/0* active idle 10.100.113.10 Unit is ready

[David Ames (thedac)]

Seth,

I am anxious to get to the bottom of this bug but I think I am going to need more information.

Can I see the bundle used to deploy?
Is this on MAAS or some other cloud substrate?
Do the machines have multiple interfaces?
Unfortunately, the juju crash dump attached is truncated and most of what I need is missing. Can you re-run a crash dump taking care to let it complete before uploading?

The biggest hint comes in the juju unit logs of the mysql-router instances that failed:

Failed to bootstrap mysqlrouter: Error: Unable to connect to the metadata server: Error connecting to MySQL server at 10.100.33.19:0: Access denied for user 'mysqlrouteruser'@'10.100.32.255' (using password: YES) (1045)

This generally says to me that the address the mysql-innodb-cluster received over the relationship is not the one mysql-router is actually using to communicate with the cluster.

If there are multiple interfaces on the machines this most likely means we need to "bind" the relation. A snippet of the bundle might look like this:

cinder-mysql-router:
  bindings:
   "": <DEFAULT SPACE>
   db-router: <SPACE FOR DB COMMS>

mysql-innodb-cluster:
  bindings:
   "": <DEFAULT SPACE>
   db-router: <SPACE FOR DB COMMS>

For now marking the bug incomplete for more information.

[David Ames (thedac)]

Seth,

If by chance "10.100.32.255" is cinder's VIP, this might be this bug:

https://bugs.launchpad.net/charm-mysql-router/+bug/1861523

[Seth Tanner (sjtanner)]

David,
I've already destroyed and attempted to redeploy a few more times, but I can do it again and get back to the state when I reported the bug.
That said I will add the bundle, it is a very lightly modified version of the stable bundle.
This is on MAAS
Yes the machines have multiple interfaces
in order for mysql-router to properly bootstrap none of the machines with the subordinate charm can have an interface that is on the same subnet as mysql-innodb-cluster

I will try binding db-router and report back.

[Seth Tanner (sjtanner)]

here is the spaces overlay that I'm using

[Seth Tanner (sjtanner)]

David,
The bindings for db-router did the trick.
I have attached the updated, working overlay.
It looks like this may not be a bug after all, but I don't recall seeing this in any of the documentation. I would be happy to help write some documentation as it relates to the 20.05 and the mysql-router charm.

[David Ames (thedac)]

Seth,

That is good news. I agree this has become a documentation bug.

For future reference, your original bindings looked like this:

  mysql-innodb-cluster:
    bindings:
      shared-db: *os-internal-api
      cluster: *os-internal-api

If they look like this it should work

  mysql-innodb-cluster:
    bindings:
      db-router: *os-internal-api
      cluster: *os-internal-api

Percona-cluster used to use shared-db directly. Mysql-innodb-cluster uses db-router. mysql-router uses shared-db to the client application. So the binding for the server side needs to be db-router.

I have a think on how we can make this clearer in docs.

[Seth Tanner (sjtanner)]

Thanks David, this all makes sense.

[Peter Matulis (petermatulis)]

I'll update the README for the mysql-router charm accordingly.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-mysql-router/+/811013

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-mysql-router/+/811013
Committed: https://opendev.org/openstack/charm-mysql-router/commit/9cc7f6f13a66eb7015c535c08f6ba31ef46cd92e
Submitter: "Zuul (22348)"
Branch: master

commit 9cc7f6f13a66eb7015c535c08f6ba31ef46cd92e
Author: Peter Matulis <email address hidden>
Date: Sat Sep 25 15:15:44 2021 -0400

    Review README

    Small review of README

      Add Documentation section

      Minor changes

      Fix a bug by adding an admonishment

    Also synced tox/pip boilerplate in order to include:
    * https://github.com/openstack-charmers/release-tools/pull/156
    * https://github.com/openstack-charmers/release-tools/pull/157

    Closes-Bug: #1881171
    Change-Id: I405e8f2692be76695ac3f57a7618deae98e45225
    Co-authored-by: Aurelien Lourot <email address hidden>