featureset001 - mysql fails to start with Can't open shared library '/usr/auth_ed25519.so' (errno: 0, cannot open shared object file: No such file or directory)

[Pooja Jadhav]

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/60c0d26/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

2021-05-18 21:07:09 | 2021-05-18 21:07:09.053141 | fa163e29-eba5-5433-5170-000000006ec6 | TIMING | tripleo_container_manage : Check podman create status | overcloud-controller-0 | 0:44:58.623329 | 639.58s
2021-05-18 21:07:09 | 2021-05-18 21:07:09.094865 | fa163e29-eba5-5433-5170-000000006ec8 | TASK | Check containers status
2021-05-18 21:12:27 | [ERROR]: Container(s) which failed to be created by podman_container module:
2021-05-18 21:12:27 | ['mysql_wait_bundle']
2021-05-18 21:12:27 | [ERROR]: Container(s) which did not finish after 300 minutes:
2021-05-18 21:12:27 | ['mysql_wait_bundle']
2021-05-18 21:12:27 | 2021-05-18 21:12:27.039300 | fa163e29-eba5-5433-5170-000000006ec8 | FATAL | Check containers status | overcloud-controller-0 | error={"changed": false, "msg": "Failed container(s): ['mysql_wait_bundle'], check logs in /var/log/containers/stdouts/"}
2021-05-18 21:12:27 | 2021-05-18 21:12:27.042573 | fa163e29-eba5-5433-5170-000000006ec8 | TIMING | tripleo_container_manage : Check containers status | overcloud-controller-0 | 0:50:16.612736 | 317.95s
2021-05-18 21:12:27 |

https://logserver.rdoproject.org/65/33765/2/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/41f4bca/logs/undercloud/var/log/containers/mysql/mariadb.log.txt.gz

2021-05-18 19:43:50 0 [ERROR] mysqld: Can't open shared library '/usr/auth_ed25519.so' (errno: 0, cannot open shared object file: No such file or directory)
2021-05-18 19:43:50 0 [ERROR] Couldn't load plugins from 'auth_ed25519.so'.
2021-05-18 19:43:53 0 [Note] InnoDB: Using Linux native AIO
2021-05-18 19:43:53 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-05-18 19:43:53 0 [Note] InnoDB: Uses event mutexes
2021-05-18 19:43:53 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-05-18 19:43:53 0 [Note] InnoDB: Number of pools: 1
2021-05-18 19:43:53 0 [Note] InnoDB: Using SSE2 crc32 instructions
2021-05-18 19:43:53 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
2021-05-18 19:43:53 0 [Note] InnoDB: Completed initialization of buffer pool
2021-05-18 19:43:53 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-05-18 19:43:54 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2021-05-18 19:43:54 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-05-18 19:43:54 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-05-18 19:43:54 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-05-18 19:43:54 0 [Note] InnoDB: Waiting for purge to start
2021-05-18 19:43:54 0 [Note] InnoDB: 10.3.27 started; log sequence number 1625443; transaction id 20
2021-05-18 19:43:54 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-05-18 19:43:54 0 [Note] InnoDB: Buffer pool(s) load completed at 210518 19:43:54
2021-05-18 19:43:54 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-05-18 19:43:54 0 [Note] /usr/libexec/mysqld: ready for connections.
Version: '10.3.27-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 0 MariaDB Server
2021-05-18 19:43:54 0 [Note] /usr/libexec/mysqld (initiated by: root[root] @ localhost []): Normal shutdown
2021-05-18 19:43:54 0 [Note] Event Scheduler: Purging the queue. 0 events
2021-05-18 19:43:54 0 [Note] InnoDB: FTS optimize thread exiting.
2021-05-18 19:43:54 0 [Note] InnoDB: Starting shutdown...
2021-05-18 19:43:54 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2021-05-18 19:43:54 0 [Note] InnoDB: Buffer pool(s) dump completed at 210518 19:43:54
2021-05-18 19:43:56 0 [Note] InnoDB: Shutdown completed; log sequence number 1625452; transaction id 23
2021-05-18 19:43:56 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-05-18 19:43:56 0 [Note] /usr/libexec/mysqld: Shutdown complete

2021-05-18 19:43:57 0 [Note] InnoDB: Using Linux native AIO
2021-05-18 19:43:57 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-05-18 19:43:57 0 [Note] InnoDB: Uses event mutexes
2021-05-18 19:43:57 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-05-18 19:43:57 0 [Note] InnoDB: Number of pools: 1
2021-05-18 19:43:57 0 [Note] InnoDB: Using SSE2 crc32 instructions
2021-05-18 19:43:57 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
2021-05-18 19:43:57 0 [Note] InnoDB: Completed initialization of buffer pool
2021-05-18 19:43:57 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-05-18 19:43:57 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2021-05-18 19:43:57 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-05-18 19:43:57 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-05-18 19:43:57 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-05-18 19:43:57 0 [Note] InnoDB: 10.3.27 started; log sequence number 1625452; transaction id 20
2021-05-18 19:43:57 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-05-18 19:43:57 0 [Note] InnoDB: Buffer pool(s) load completed at 210518 19:43:57
2021-05-18 19:43:57 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-05-18 19:43:57 0 [Note] /usr/libexec/mysqld: ready for connections.
Version: '10.3.27-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 0 MariaDB Server
2021-05-18 19:43:58 0 [Note] /usr/libexec/mysqld (initiated by: root[root] @ localhost []): Normal shutdown
2021-05-18 19:43:58 0 [Note] Event Scheduler: Purging the queue. 0 events
2021-05-18 19:43:58 0 [Note] InnoDB: FTS optimize thread exiting.
2021-05-18 19:43:58 0 [Note] InnoDB: Starting shutdown...
2021-05-18 19:43:58 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2021-05-18 19:43:58 0 [Note] InnoDB: Buffer pool(s) dump completed at 210518 19:43:58
2021-05-18 19:43:59 0 [Note] InnoDB: Shutdown completed; log sequence number 1625461; transaction id 21
2021-05-18 19:43:59 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-05-18 19:43:59 0 [Note] /usr/libexec/mysqld: Shutdown complete

2021-05-18 19:44:05 0 [Note] InnoDB: Using Linux native AIO
2021-05-18 19:44:05 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-05-18 19:44:05 0 [Note] InnoDB: Uses event mutexes
2021-05-18 19:44:05 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-05-18 19:44:05 0 [Note] InnoDB: Number of pools: 1
2021-05-18 19:44:05 0 [Note] InnoDB: Using SSE2 crc32 instructions
2021-05-18 19:44:05 0 [Note] InnoDB: Initializing buffer pool, total size = 1G, instances = 8, chunk size = 128M
2021-05-18 19:44:05 0 [Note] InnoDB: Completed initialization of buffer pool
2021-05-18 19:44:05 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-05-18 19:44:05 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2021-05-18 19:44:05 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-05-18 19:44:05 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-05-18 19:44:05 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-05-18 19:44:05 0 [Note] InnoDB: Waiting for purge to start
2021-05-18 19:44:05 0 [Note] InnoDB: 10.3.27 started; log sequence number 1625461; transaction id 20
2021-05-18 19:44:05 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-05-18 19:44:05 0 [Note] InnoDB: Buffer pool(s) load completed at 210518 19:44:05
2021-05-18 19:44:05 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-05-18 19:44:05 0 [Note] Server socket created on IP: '192.168.24.1'.
2021-05-18 19:44:05 0 [Note] /usr/libexec/mysqld: ready for connections.
Version: '10.3.27-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server

[Rabi Mishra (rabi)]

The issue is with overcloud/controller mysql_wait_bundle and it's a duplicate of https://bugs.launchpad.net/tripleo/+bug/1928869