CS9 - Standalone: mysql_wait_bundle fails with "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'"

[Douglas Viroel]

Standalone jobs have this intermittent failure, where mysql_wait_bundle containet fails to start. The followin errors are shown[1][2]:

Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0]
Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'

NOTE: this happens more often in FIPS enabled jobs, considering this change [3], but the issue was seen in non-FIPS jobs too.

[1] https://2496c8b3721efd3eb4ab-5b369dc32177e920b38f658289f2b193.ssl.cf5.rackcdn.com/824479/18/check/tripleo-ci-centos-9-standalone/69a6eda/logs/undercloud/home/zuul/standalone_deploy.log
[2] https://1deec6edd105464d9773-b3be29898e9936ff543653f1540e5e4c.ssl.cf1.rackcdn.com/824479/18/check/tripleo-ci-centos-9-scenario012-standalone/98ede58/logs/undercloud/home/zuul/standalone_deploy.log
[3] https://review.opendev.org/c/openstack/tripleo-ci/+/824479/

[Luca Miccini (lmiccini2)]

the issue is:

Mar 11 20:30:15.866 standalone.localdomain pacemaker-controld [25187] (log_executor_event) error: Result of start operation for galera-bundle-podman-0 on standalone: Timed Out after 20s (Resource agent did not complete in time) | CIB update 88, graph action confirmed; call=28 key=galera-bundle-podman-0_start_0

Bundle: galera-bundle
  Podman: image=149.202.181.113:5001/tripleomastercentos9/openstack-mariadb:pcmklatest network=host options="--user=root --log-driver=k8s-file --log-opt path=/var/log/containers/stdouts/galera-bundle.log -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" promoted-max=1 replicas=1 run-command="/bin/bash /usr/local/bin/kolla_start"
  Network: control-port=3123
  Storage Mapping:
   options=ro source-dir=/var/lib/kolla/config_files/mysql.json target-dir=/var/lib/kolla/config_files/config.json (mysql-cfg-files)
   options=ro source-dir=/var/lib/config-data/puppet-generated/mysql/ target-dir=/var/lib/kolla/config_files/src (mysql-cfg-data)
   options=ro source-dir=/etc/hosts target-dir=/etc/hosts (mysql-hosts)
   options=ro source-dir=/etc/localtime target-dir=/etc/localtime (mysql-localtime)
   options=rw,z source-dir=/var/lib/mysql target-dir=/var/lib/mysql (mysql-lib)
   options=rw,z source-dir=/var/log/mariadb target-dir=/var/log/mariadb (mysql-log-mariadb)
   options=rw,z source-dir=/var/log/containers/mysql target-dir=/var/log/mysql (mysql-log)
   options=rw source-dir=/dev/log target-dir=/dev/log (mysql-dev-log)
  Resource: galera (class=ocf provider=heartbeat type=galera)
   Attributes: additional_parameters=--open-files-limit=16384 cluster_host_map=standalone:standalone.ctlplane.localdomain enable_creation=true log=/var/log/mysql/mysqld.log wsrep_cluster_address=gcomm://standalone.ctlplane.localdomain
   Meta Attrs: container-attribute-target=host master-max=1 ordered=true
   Operations: demote interval=0s timeout=120s (galera-demote-interval-0s)
               monitor interval=20s timeout=30s (galera-monitor-interval-20s)
               monitor interval=10s role=Promoted timeout=30s (galera-monitor-interval-10s)
               monitor interval=30s role=Unpromoted timeout=30s (galera-monitor-interval-30s)
               promote interval=0s on-fail=block timeout=300s (galera-promote-interval-0s)
               start interval=0s timeout=120s (galera-start-interval-0s)
               stop interval=0s timeout=120s (galera-stop-interval-0s)