centos-7; failed with code: 1 -> Error: unable to get cib 2020-04-28 16:16:39 | . Either the cluster was not running or the versions of pcmk/pcs between host and container are not matching

[wes hayutin]

All branches of TripleO are failing this atm.

2020-04-28 16:16:39 | Error: /Stage[main]/Pacemaker::Stonith/Pacemaker::Property[Disable STONITH]/Pcmk_property[property--stonith-enabled]: Could not evaluate: backup_cib: Running: pcs cluster cib /var/lib/pacemaker/cib/puppet-cib-backup20200428-9-1jijw4n failed with code: 1 -> Error: unable to get cib
2020-04-28 16:16:39 | . Either the cluster was not running or the versions of pcmk/pcs between host and container are not matching
2020-04-28 16:16:39 | Error: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Pacemaker::Property[haproxy-role-overcloud-controller-2]/Pcmk_property[property-overcloud-controller-2-haproxy-role]: Could not evaluate: backup_cib: Running: pcs cluster cib /var/lib/pacemaker/cib/puppet-cib-backup20200428-9-1fkd444 failed with code: 1 -> Error: unable to get cib
2020-04-28 16:16:39 | . Either the cluster was not running or the versions of pcmk/pcs between host and container are not matching
2020-04-28 16:16:39 | Error: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Pacemaker::Property[haproxy-role-overcloud-controller-1]/Pcmk_property[property-overcloud-controller-1-haproxy-role]: Could not evaluate: backup_cib: Running: pcs cluster cib /var/lib/pacemaker/cib/puppet-cib-backup20200428-9-19gmakk failed with code: 1 -> Error: unable to get cib
2020-04-28 16:16:39 | . Either the cluster was not running or the versions of pcmk/pcs between host and container are not matching
2020-04-28 16:16:39 | Error: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Pacemaker::Property[haproxy-role-overcloud-controller-0]/Pcmk_property[property-overcloud-controller-0-haproxy-role]: Could not evaluate: backup_cib: Running: pcs cluster cib /var/lib/pacemaker/cib/puppet-cib-backup20200428-9-1dirb5x failed with code: 1 -> Error: unable to get cib
2020-04-28 16:16:39 | . Either the cluster was not running or the versions of pcmk/pcs between host and container are not matching
2020-04-28 16:16:39 | Error: /Stage[main]/Tripleo::Profile::Pacemaker::Haproxy_bundle/Tripleo::Pacemaker::Haproxy_with_vip[haproxy_and_control_vip]/Pacemaker::Resource::Ip[control_vip]/Pcmk_resource[ip-192.168.24.18]: Could not evaluate: backup_cib: Running: pcs cluster cib /var/lib/pacemaker/cib/puppet-cib-backup20200428-9-4n1qel failed with code: 1 -> Error: unable to get cib
2020-04-28 16:16:39 | . Either the cluster was not running or the versions of pcmk/pcs between host and container are not matching
2020-04-28 16:16:39 | Error:

https://logserver.rdoproject.org/33/718433/5/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train-branch/d7269ea/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

Link to failing jobs
https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train-branch&job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch&job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch&job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001

Latest pass was today..

https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train-branch&job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-stein-branch&job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch&job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001&result=success

Perhaps a CentOS update?

[wes hayutin (weshayutin)]

https://opendev.org/openstack/puppet-pacemaker/src/branch/master/lib/puppet/provider/pcmk_common.rb#L63-L78

[wes hayutin (weshayutin)]

This is CentOS-7 only...

CentOS-8 is fine..
https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset001&job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001

[wes hayutin (weshayutin)]

pacemaker-1.1.21-4.el7.x86_64.rpm 2020-04-03 21:03 478K
http://mirror.centos.org/centos/7/os/x86_64/Packages/?C=M;O=D

###################################################################

On train.. containers we have:
pr 26 15:28:29 Installed: pacemaker-libs-1.1.20-5.el7_7.2.x86_64
Apr 26 15:28:31 Installed: pacemaker-cli-1.1.20-5.el7_7.2.x86_64
Apr 26 15:28:32 Installed: pacemaker-cluster-libs-1.1.20-5.el7_7.2.x86_64
Apr 26 15:28:33 Installed: pacemaker-1.1.20-5.el7_7.2.x86_64
Apr 26 15:28:35 Installed: pcs-0.9.167-3.el7.centos.1.x86_64
Apr 26 15:28:37 Installed: pacemaker-remote-1.1.20-5.el7_7.2.x86_64
Apr 26 15:28:39 Installed: rabbitmq-server-3.6.16-1.el7.noarch
Apr 28 14:50:43 Updated: puppet-swift-15.4.1-0.20200427173913.1795fc0.el7.noarch
Apr 28 14:50:43 Updated: puppet-pacemaker-0.8.1-0.20200428131415.d501b27.el7.noarch
LOG:
https://logserver.rdoproject.org/33/718433/5/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train-branch/d7269ea/logs/undercloud/var/log/extra/podman/containers/rabbitmq/log/yum.log.txt.gz

#############################################################################

On the bm node:
pacemaker-1.1.21-4.el7.x86_64
pacemaker-cli-1.1.21-4.el7.x86_64
pacemaker-cluster-libs-1.1.21-4.el7.x86_64
pacemaker-libs-1.1.21-4.el7.x86_64
pacemaker-remote-1.1.21-4.el7.x86_64

https://logserver.rdoproject.org/33/718433/5/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train-branch/d7269ea/logs/overcloud-controller-0/var/log/extra/rpm-list.txt.gz

#############################################################################

[wes hayutin (weshayutin)]

kicking off the train - queens pipelines to get the container rpm versions and bm node versions aligned