Pike upgrade CI jobs failing due to "openstack-cinder-volume is not running anywhere and so cannot be restarted"

Bug #1733846 reported by Jose Luis Franco on 2017-11-22
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Medium
Unassigned

Bug Description

Verifying the CI status page, it can be seen that the CI job gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike is failing recently.

When checking the job logs, the error seem to be on the upgrade step:

https://logs.rdoproject.org/43/522043/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z081f531e693a49d6a99eab9e64660a70/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz#_2017-11-22_01_57_28

In the detailed failure log: https://logs.rdoproject.org/43/522043/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z081f531e693a49d6a99eab9e64660a70/undercloud/home/jenkins/failed_upgrade.log.txt.gz we can see the displayed error:

   Failed Actions:
    * rabbitmq_stop_0 on upstream-centos-7-2-node-rdo-cloud-tripleo-48185-20744 '\''unknown error'\'' (1): call=51, status=complete, exitreason='\''none'\'',
        last-rc-change='\''Wed Nov 22 01:56:41 2017'\'', queued=0ms, exec=270ms

    Daemon Status:
      corosync: active/enabled
      pacemaker: active/enabled
    + grep openstack-cinder-volume
      pcsd: active/enabled'
    + for service in '$SERVICES_TO_RESTART'
    + echo 'Restarting openstack-cinder-volume...'
    + pcs resource restart --wait=600 openstack-cinder-volume
    Error: Error performing operation: No such device or address
    openstack-cinder-volume is not running anywhere and so cannot be restarted

Jose Luis Franco (jfrancoa) wrote :

Journal logs:

Nov 22 01:56:40 upstream-centos-7-2-node-rdo-cloud-tripleo-48185-20744 rabbitmq-cluster(rabbitmq)[155874]: ERROR: Unexpected return code from '/usr/sbin/rabbitmqctl cluster_status' exit code: 1
Nov 22 01:56:40 upstream-centos-7-2-node-rdo-cloud-tripleo-48185-20744 crmd[110807]: error: Failed to receive meta-data for ocf:heartbeat:galera
Nov 22 01:56:40 upstream-centos-7-2-node-rdo-cloud-tripleo-48185-20744 crmd[110807]: error: No metadata for ocf::heartbeat:galera

https://logs.rdoproject.org/43/522043/2/openstack-check/gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike/Z081f531e693a49d6a99eab9e64660a70/subnode-2/var/log/journal.txt.gz#_Nov_22_01_56_41

Changed in tripleo:
milestone: none → queens-3
importance: Undecided → Medium
tags: added: upgrade
Changed in tripleo:
status: New → Triaged
Changed in tripleo:
milestone: queens-3 → queens-rc1
Changed in tripleo:
milestone: queens-rc1 → rocky-1
Changed in tripleo:
milestone: rocky-1 → rocky-2
Jose Luis Franco (jfrancoa) wrote :

The error doesn't seem to appear any more. Closing the bug.

Changed in tripleo:
status: Triaged → Won't Fix
milestone: rocky-2 → none
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers