charm needs to handle database not being available (was: mysql-innodb-cluster:certificates in a bundle is known to cause a race condition)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MySQL InnoDB Cluster Charm |
Invalid
|
Undecided
|
Unassigned | ||
OpenStack Bundles |
Invalid
|
Undecided
|
Unassigned | ||
OpenStack Octavia Charm |
Triaged
|
High
|
Unassigned |
Bug Description
The current openstack-base bundle has the following relation.
- - vault:certificates
- mysql-innodb-
It can restart mysqld as soon as Vault's CA is available. And if that happens during init_db of a OpenStack API charm, the charm can be in an unrecoverable error. It may not happen with openstack-base out of the box, but by adding octavia on top for example it's easy to trigger the error.
mysql-innodb-
unit-mysql-
unit-octavia-0: 06:14:56 DEBUG unit.octavia/
unit-octavia-0: 06:14:56 ERROR unit.octavia/
Traceback (most recent call last):
File "/var/lib/
bus.
File "/var/lib/
_invoke(
File "/var/lib/
handler.
File "/var/lib/
self.
File "/var/lib/
octavia_
File "/var/lib/
subprocess.
File "/usr/lib/
raise CalledProcessEr
subprocess.
mysqld was started around 6:15.
$ juju run -a mysql-innodb- cluster -- 'ps aux | grep mysql[d]' cluster/ 0 cluster/ 1 cluster/ 2
- Stdout: |
mysql 81833 22.5 8.8 5639528 991860 ? Ssl 06:15 14:42 /usr/sbin/mysqld
UnitId: mysql-innodb-
- Stdout: |
mysql 78404 3.9 6.0 4320160 678576 ? Ssl 06:16 2:26 /usr/sbin/mysqld
UnitId: mysql-innodb-
- Stdout: |
mysql 80191 3.8 6.0 4254608 680776 ? Ssl 06:17 2:19 /usr/sbin/mysqld
UnitId: mysql-innodb-
And "Forcing close of thread 19995 user: 'octavia'."
2022-08- 09T06:14: 52.758658Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30- 0ubuntu0. 20.04.2) . 09T06:14: 55.992274Z 0 [System] [MY-011504] [Repl] Plugin group_replication reported: 'Group membership changed: This member has left the group.' 09T06:14: 58.001073Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 19995 user: 'octavia'. 09T06:15: 49.034626Z 0 [Warning] [MY-011630] [Repl] Plugin group_replication reported: 'Due to a plugin error, some transactions were unable to be certified and will now rollback.' 09T06:15: 49.034839Z 19995 [ERROR] [MY-011615] [Repl] Plugin group_replication reported: 'Error while waiting for conflict detection procedure to finish on session 19995' 09T06:15: 49.034866Z 19995 [ERROR] [MY-010207] [Repl] Run function 'before_commit' in plugin 'group_replication' failed 09T06:15: 49.038577Z 0 [System] [MY-011651] [Repl] Plugin group_replication reported: 'Plugin 'group_replication' has been stopped.' 09T06:15: 50.284934Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30- 0ubuntu0. 20.04.2) (Ubuntu).
2022-08-
2022-08-
2022-08-
2022-08-
2022-08-
2022-08-
2022-08-