Document need for db-router binding
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MySQL InnoDB Cluster Charm |
Invalid
|
Medium
|
Unassigned | ||
MySQL Router Charm |
Fix Released
|
Medium
|
Peter Matulis |
Bug Description
some mysql-router units bootstrap, but not all of them
Model Controller Cloud/Region Version SLA Timestamp
openstack trc-default TRC/default 2.7.6 unsupported 10:34:50-07:00
App Version Status Scale Charm Store Rev OS Notes
ceph-mon 15.2.1 active 3 ceph-mon jujucharms 48 ubuntu
ceph-osd 15.2.1 active 6 ceph-osd jujucharms 303 ubuntu
ceph-radosgw 15.2.1 waiting 1 ceph-radosgw jujucharms 288 ubuntu
cinder 16.0.0 waiting 1 cinder jujucharms 303 ubuntu
cinder-ceph 16.0.0 active 1 cinder-ceph jujucharms 256 ubuntu
cinder-mysql-router 8.0.20 waiting 1 mysql-router jujucharms 0 ubuntu
dashboard-
glance 20.0.0 waiting 1 glance jujucharms 297 ubuntu
glance-mysql-router 8.0.20 waiting 1 mysql-router jujucharms 0 ubuntu
keystone 17.0.0 waiting 1 keystone jujucharms 314 ubuntu
keystone-
mysql-innodb-
mysql-router waiting 0 mysql-router jujucharms 0 ubuntu
neutron-api 16.0.0 waiting 1 neutron-api jujucharms 286 ubuntu
neutron-
neutron-
nova-cloud-
nova-compute 21.0.0 active 5 nova-compute jujucharms 316 ubuntu
nova-mysql-router 8.0.20 waiting 1 mysql-router jujucharms 0 ubuntu
ntp 3.5 active 5 ntp jujucharms 39 ubuntu
openstack-dashboard 18.3.2 waiting 1 openstack-dashboard jujucharms 304 ubuntu
ovn-central 20.03.0 active 3 ovn-central jujucharms 0 ubuntu
ovn-chassis 20.03.0 active 5 ovn-chassis jujucharms 1 ubuntu
placement 3.0.0 waiting 1 placement jujucharms 11 ubuntu
placement-
rabbitmq-server 3.8.2 active 1 rabbitmq-server jujucharms 102 ubuntu
vault 1.1.1 active 1 vault jujucharms 39 ubuntu
vault-mysql-router 8.0.20 active 1 mysql-router jujucharms 0 ubuntu
Unit Workload Agent Machine Public address Ports Message
ceph-mon/0 active idle 0 10.100.190.1 Unit is ready and clustered
ceph-mon/2* active idle 2 10.100.190.3 Unit is ready and clustered
ceph-mon/3 active idle 14 10.100.190.2 Unit is ready and clustered
ceph-osd/0 active idle 3 10.100.190.7 Unit is ready (12 OSD)
ceph-osd/1 active idle 4 10.100.190.10 Unit is ready (12 OSD)
ceph-osd/2 active idle 5 10.100.190.8 Unit is ready (12 OSD)
ceph-osd/3* active idle 6 10.100.190.6 Unit is ready (12 OSD)
ceph-osd/4 active idle 7 10.100.190.9 Unit is ready (12 OSD)
ceph-osd/5 active idle 8 10.100.190.11 Unit is ready (12 OSD)
ceph-radosgw/0* waiting idle 12/lxd/0 10.100.113.11 80/tcp Incomplete relations: identity
cinder/0* waiting idle 12/lxd/1 10.100.113.0 8776/tcp Incomplete relations: database, identity
cinder-ceph/0* active idle 10.100.113.0 Unit is ready
cinder-
glance/0* waiting idle 12/lxd/2 10.100.113.7 9292/tcp Incomplete relations: identity, database
glance-
keystone/0* blocked idle 13/lxd/0 10.100.113.8 5000/tcp Incomplete relations: database, Allowed_units list provided but this unit not present
keystone-
mysql-innodb-
mysql-innodb-
mysql-innodb-
neutron-api/0* waiting idle 12/lxd/3 10.100.113.13 9696/tcp Incomplete relations: identity, database
neutron-
neutron-
nova-cloud-
nova-
nova-compute/0 active idle 9 10.100.113.3 Unit is ready
ntp/2 active idle 10.100.113.3 123/udp chrony: Ready
ovn-chassis/2 active idle 10.100.113.3 Unit is ready
nova-compute/1 active idle 10 10.100.113.4 Unit is ready
ntp/1 active idle 10.100.113.4 123/udp chrony: Ready
ovn-chassis/1 active idle 10.100.113.4 Unit is ready
nova-compute/2 active idle 11 10.100.113.2 Unit is ready
ntp/0* active idle 10.100.113.2 123/udp chrony: Ready
ovn-chassis/0* active idle 10.100.113.2 Unit is ready
nova-compute/3* active idle 12 10.100.112.201 Unit is ready
ntp/3 active idle 10.100.112.201 123/udp chrony: Ready
ovn-chassis/3 active idle 10.100.112.201 Unit is ready
nova-compute/4 active idle 13 10.100.113.1 Unit is ready
ntp/4 active idle 10.100.113.1 123/udp chrony: Ready
ovn-chassis/4 active idle 10.100.113.1 Unit is ready
openstack-
dashboard-
ovn-central/0* active idle 9/lxd/1 10.100.113.12 6641/tcp,6642/tcp Unit is ready (leader: ovnnb_db, ovnsb_db northd: active)
ovn-central/1 active idle 10/lxd/1 10.100.113.5 6641/tcp,6642/tcp Unit is ready
ovn-central/2 active idle 11/lxd/1 10.100.112.255 6641/tcp,6642/tcp Unit is ready
placement/0* waiting idle 13/lxd/2 10.100.113.6 8778/tcp 'shared-db' incomplete, 'identity-service' incomplete
placement-
rabbitmq-server/0* active idle 13/lxd/3 10.100.33.21 5672/tcp Unit is ready
vault/0* active idle 13/lxd/4 10.100.113.10 8200/tcp Unit is ready (active: true, mlock: disabled)
vault-
Changed in charm-mysql-router: | |
milestone: | 20.08 → none |
Changed in charm-mysql-innodb-cluster: | |
milestone: | 20.08 → none |
Changed in charm-mysql-router: | |
assignee: | nobody → Peter Matulis (petermatulis) |
status: | Triaged → In Progress |
Changed in charm-mysql-router: | |
milestone: | none → 21.10 |
Changed in charm-mysql-router: | |
status: | Fix Committed → Fix Released |
Seth,
I am anxious to get to the bottom of this bug but I think I am going to need more information.
Can I see the bundle used to deploy?
Is this on MAAS or some other cloud substrate?
Do the machines have multiple interfaces?
Unfortunately, the juju crash dump attached is truncated and most of what I need is missing. Can you re-run a crash dump taking care to let it complete before uploading?
The biggest hint comes in the juju unit logs of the mysql-router instances that failed:
Failed to bootstrap mysqlrouter: Error: Unable to connect to the metadata server: Error connecting to MySQL server at 10.100.33.19:0: Access denied for user 'mysqlrouteruse r'@'10. 100.32. 255' (using password: YES) (1045)
This generally says to me that the address the mysql-innodb- cluster received over the relationship is not the one mysql-router is actually using to communicate with the cluster.
If there are multiple interfaces on the machines this most likely means we need to "bind" the relation. A snippet of the bundle might look like this:
cinder- mysql-router:
bindings:
"": <DEFAULT SPACE>
db-router: <SPACE FOR DB COMMS>
mysql-innodb- cluster:
bindings:
"": <DEFAULT SPACE>
db-router: <SPACE FOR DB COMMS>
For now marking the bug incomplete for more information.