Pike upgrade CI jobs failing due to "openstack-cinder-volume is not running anywhere and so cannot be restarted"

[Jose Luis Franco]

Verifying the CI status page, it can be seen that the CI job gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike is failing recently.

When checking the job logs, the error seem to be on the upgrade step:

https://logs.rdoproject.org/43/522043/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z081f531e693a49d6a99eab9e64660a70/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz#_2017-11-22_01_57_28

In the detailed failure log: https://logs.rdoproject.org/43/522043/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z081f531e693a49d6a99eab9e64660a70/undercloud/home/jenkins/failed_upgrade.log.txt.gz we can see the displayed error:

   Failed Actions:
    * rabbitmq_stop_0 on upstream-centos-7-2-node-rdo-cloud-tripleo-48185-20744 '\''unknown error'\'' (1): call=51, status=complete, exitreason='\''none'\'',
        last-rc-change='\''Wed Nov 22 01:56:41 2017'\'', queued=0ms, exec=270ms

    Daemon Status:
      corosync: active/enabled
      pacemaker: active/enabled
    + grep openstack-cinder-volume
      pcsd: active/enabled'
    + for service in '$SERVICES_TO_RESTART'
    + echo 'Restarting openstack-cinder-volume...'
    + pcs resource restart --wait=600 openstack-cinder-volume
    Error: Error performing operation: No such device or address
    openstack-cinder-volume is not running anywhere and so cannot be restarted

[Jose Luis Franco (jfrancoa)]

Journal logs:

Nov 22 01:56:40 upstream-centos-7-2-node-rdo-cloud-tripleo-48185-20744 rabbitmq-cluster(rabbitmq)[155874]: ERROR: Unexpected return code from '/usr/sbin/rabbitmqctl cluster_status' exit code: 1
Nov 22 01:56:40 upstream-centos-7-2-node-rdo-cloud-tripleo-48185-20744 crmd[110807]: error: Failed to receive meta-data for ocf:heartbeat:galera
Nov 22 01:56:40 upstream-centos-7-2-node-rdo-cloud-tripleo-48185-20744 crmd[110807]: error: No metadata for ocf::heartbeat:galera

https://logs.rdoproject.org/43/522043/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z081f531e693a49d6a99eab9e64660a70/subnode-2/var/log/journal.txt.gz#_Nov_22_01_56_41

[Jose Luis Franco (jfrancoa)]

The error doesn't seem to appear any more. Closing the bug.