Comment 4 for bug 1875352

I think it happens because pacemaker and libqb are different on the host vs inside the container.

host:

May 04 12:53:23 Installed: corosynclib-2.4.5-4.el7.x86_64
May 04 12:53:23 Installed: corosync-2.4.5-4.el7.x86_64
May 04 12:53:23 Installed: pacemaker-libs-1.1.21-4.el7.x86_64
May 04 12:53:24 Installed: pacemaker-cli-1.1.21-4.el7.x86_64
May 04 12:53:24 Installed: pacemaker-cluster-libs-1.1.21-4.el7.x86_64
May 04 12:53:25 Installed: pacemaker-1.1.21-4.el7.x86_64
May 04 12:53:23 Installed: libqb-1.0.1-9.el7.x86_64

container:

()[mysql@e6473ba41b87 /]$
pacemaker-1.1.20-5.el7_7.2.x86_64
pacemaker-libs-1.1.20-5.el7_7.2.x86_64
pacemaker-remote-1.1.20-5.el7_7.2.x86_64
pacemaker-cli-1.1.20-5.el7_7.2.x86_64
puppet-pacemaker-0.8.1-0.20200203133231.83d23b3.el7.noarch
pacemaker-cluster-libs-1.1.20-5.el7_7.2.x86_64
libqb-1.0.1-7.el7.x86_64

and this results in:

https://logserver.rdoproject.org/19/26919/6/check/tripleo-ceph-integration-centos-7-scenario001-standalone-train/f326712/logs/undercloud/var/log/containers/stdouts/mysql_init_bundle.log.txt.gz

020-05-04T13:09:52.199888272+00:00 stderr F Error: /Stage[main]/Pacemaker::Stonith/Pacemaker::Property[Disable STONITH]/Pcmk_property[property--stonith-enabled]: Could not evaluate: backup_cib: Running: pcs cluster cib /var/lib/pacemaker/cib/puppet-cib-backup20200504-10-v9mh2h failed with code: 1 -> Error: unable to get cib
2020-05-04T13:09:52.199888272+00:00 stderr F . Either the cluster was not running or the versions of pcmk/pcs between host and container are not matching
2020-05-04T13:09:53.195928655+00:00 stderr F Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Property[galera-role-standalone]/Pcmk_property[property-standalone-galera-role]: Could not evaluate: backup_cib: Running: pcs cluster cib /var/lib/pacemaker/cib/puppet-cib-backup20200504-10-14wsb19 failed with code: 1 -> Error: unable to get cib
2020-05-04T13:09:53.195928655+00:00 stderr F . Either the cluster was not running or the versions of pcmk/pcs between host and container are not matching
2020-05-04T13:09:53.197083097+00:00 stdout F Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]: Dependency Pcmk_property[property-standalone-galera-role] has failures: true
2020-05-04T13:09:53.197287929+00:00 stderr F Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]: Skipping because of failed dependencies
2020-05-04T13:09:53.198072623+00:00 stderr F Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Ocf[galera]/Pcmk_resource[galera]: Skipping because of failed dependencies
2020-05-04T13:09:53.198588812+00:00 stderr F Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]: Skipping because of failed dependencies
2020-05-04T13:09:53.213205898+00:00 stdout F Info: Creating state file /var/lib/puppet/state/state.yaml
2020-05-04T13:09:53.217818031+00:00 stderr F Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
2020-05-04T13:09:53.219131213+00:00 stdout F Changes:
2020-05-04T13:09:53.219131213+00:00 stdout F Total: 6
2020-05-04T13:09:53.219131213+00:00 stdout F Events:
2020-05-04T13:09:53.219131213+00:00 stdout F Failure: 2
2020-05-04T13:09:53.219131213+00:00 stdout F Success: 6
2020-05-04T13:09:53.219131213+00:00 stdout F Total: 8
2020-05-04T13:09:53.219131213+00:00 stdout F Resources:
2020-05-04T13:09:53.219131213+00:00 stdout F Failed: 2
2020-05-04T13:09:53.219131213+00:00 stdout F Changed: 6
2020-05-04T13:09:53.219131213+00:00 stdout F Skipped: 72
2020-05-04T13:09:53.219131213+00:00 stdout F Out of sync: 8
2020-05-04T13:09:53.219131213+00:00 stdout F Total: 82
2020-05-04T13:09:53.219131213+00:00 stdout F Time:
2020-05-04T13:09:53.219131213+00:00 stdout F File line: 0.00
2020-05-04T13:09:53.219131213+00:00 stdout F File: 0.03
2020-05-04T13:09:53.219131213+00:00 stdout F Pcmk property: 1.99
2020-05-04T13:09:53.219131213+00:00 stdout F Last run: 1588597793
2020-05-04T13:09:53.219131213+00:00 stdout F Config retrieval: 4.75
2020-05-04T13:09:53.219131213+00:00 stdout F Total: 2.43
2020-05-04T13:09:53.219131213+00:00 stdout F Version:
2020-05-04T13:09:53.219131213+00:00 stdout F Config: 1588597786
2020-05-04T13:09:53.219131213+00:00 stdout F Puppet: 5.5.10
2020-05-04T13:09:53.527062554+00:00 stderr F + rc=1
2020-05-04T13:09:53.527062554+00:00 stderr F + set -e
2020-05-04T13:09:53.527062554+00:00 stderr F + set +ux

so until the containers are rebuilt with the same rpms as the base image we could hit this issue.

We're trying to solve this via https://bugs.launchpad.net/tripleo/+bug/1863442.