Could not prefetch mysql_user provider 'mysql'

[Ananya Banerjee]

overcloud stack fails with
Notice: /Stage[main]/Mysql::Server::Installdb/File[/var/log/mariadb/mariadb.log]/ensure: created
Info: Class[Mysql::Server::Installdb]: Unscheduling all events on Class[Mysql::Server::Installdb]
Error: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0]
Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]/returns: change from 'notrun' to ['0'] failed: '/usr/bin/clustercheck >/dev/null' returned 1 instead of one of [0]
Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)

logs:
https://080d890aa67c3ac6e825-db7bec567acf22c68dd5010e0a0fcd60.ssl.cf1.rackcdn.com/843159/9/check/tripleo-ci-centos-8-9-multinode-mixed-os/263219b/logs/subnode-1/var/log/extra/podman/containers/mysql_wait_bundle/stdout.log

Affected jobs:
tripleo-ci-centos-8-9-multinode-mixed-os

This is blocking wallaby 8 gate

[Marios Andreou (marios-b)]

please avoid filing bugs when you only have one example of the error.

Looking at the job history it looks good [1]. The error in description was from the run on 843159 [2] and it looks like it happened once.

I don't think this is a gate blocker - at least not yet ;) - so moving status and importance for now.

We can revisit if we see this happen again/in more cases or at least more than once.

Please feel free to move back if I have missed something or my analysis here is wrong

[1] https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-9-multinode-mixed-os
[2] https://review.opendev.org/c/openstack/tripleo-heat-templates/+/843159/