CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid backend was found

Bug #1729253 reported by Cédric Jeanneret deactivated on 2017-11-01
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
tripleo
Critical
Martin André

Bug Description

Dear Stackers,

Apparently, there's an issue for the aforementioned scenario in the CI. It fails with the following stacktrace:

2017-11-01 07:54:17 | ==============================
2017-11-01 07:54:17 | Failed 1 tests - output below:
2017-11-01 07:54:17 | ==============================
2017-11-01 07:54:17 |
2017-11-01 07:54:17 | tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern[compute,id-557cd2c2-4eb8-4dce-98be-f86765ff311b,image,volume]
2017-11-01 07:54:17 | ------------------------------------------------------------------------------------------------------------------------------------------------------
2017-11-01 07:54:17 |
2017-11-01 07:54:17 | Captured traceback:
2017-11-01 07:54:17 | ~~~~~~~~~~~~~~~~~~~
2017-11-01 07:54:17 | Traceback (most recent call last):
2017-11-01 07:54:17 | File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
2017-11-01 07:54:17 | return f(self, *func_args, **func_kwargs)
2017-11-01 07:54:17 | File "/usr/lib/python2.7/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 100, in test_volume_boot_pattern
2017-11-01 07:54:17 | volume_origin = self._create_volume_from_image()
2017-11-01 07:54:17 | File "/usr/lib/python2.7/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 43, in _create_volume_from_image
2017-11-01 07:54:17 | return self.create_volume(name=vol_name, imageRef=img_uuid)
2017-11-01 07:54:17 | File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 234, in create_volume
2017-11-01 07:54:17 | volume['id'], 'available')
2017-11-01 07:54:17 | File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 204, in wait_for_volume_resource_status
2017-11-01 07:54:17 | resource_name=resource_name, resource_id=resource_id)
2017-11-01 07:54:17 | tempest.exceptions.VolumeResourceBuildErrorException: volume d59a4620-f7dc-476c-bb62-6279f4e24b5c failed to build and is in ERROR status

An example:
http://logs.openstack.org/74/513274/11/check/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-container/4d603a9/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-11-01_07_54_17

(original review raising that issue: https://review.openstack.org/#/c/513274/)

Thank you!

Alex Schultz (alex-schultz) wrote :

It looks like cinder is missing the backends.

http://logs.openstack.org/18/516418/1/gate/legacy-tripleo-ci-centos-7-containers-multinode/bd8a18e/logs/subnode-2/var/log/containers/cinder/cinder-scheduler.log.txt.gz#_2017-11-01_12_45_44_266

2017-11-01 12:45:44.266 7 ERROR cinder.scheduler.flows.create_volume [req-87efaadc-0de0-472c-9459-246af22c12c2 e93848cd3a6b4eda808b5d6eddc247e9 70bc340995e14c97917ea43ec0757958 - default default] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. No weighed backends available: NoValidBackend: No valid backend was found. No weighed backends available

Changed in tripleo:
status: New → Triaged
milestone: none → queens-2
importance: Undecided → Critical
tags: added: alert ci containers
summary: - CI issue: legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-
- container fails with volume
+ CI issue: Failed to run cinder task ScheduleCreateVolumeTask, No valid
+ backend was found
Changed in tripleo:
assignee: nobody → Alan Bishop (alan-bishop)
Changed in tripleo:
status: Triaged → In Progress
Alan Bishop (alan-bishop) wrote :

cinder-volume is down:

http://logs.openstack.org/18/516418/1/gate/legacy-tripleo-ci-centos-7-containers-multinode/bd8a18e/logs/subnode-2/var/log/cinder/volume.log.txt.gz#_2017-11-01_12_42_49_271

CI jobs are not containerizing cinder-volume (and cinder-backup) when run under pacemaker, and this may be causing a race condition in db sync.

Fix proposed in https://review.openstack.org/517038

Changed in tripleo:
assignee: Alan Bishop (alan-bishop) → Martin André (mandre)
wes hayutin (weshayutin) wrote :

adding promotion blocker because the job is failing in the upstream gates jobs.

tags: added: promotion-blocker

Reviewed: https://review.openstack.org/517038
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=17d34888958c892cbf8fda6ef89e323ed1075076
Submitter: Zuul
Branch: master

commit 17d34888958c892cbf8fda6ef89e323ed1075076
Author: Alan Bishop <email address hidden>
Date: Wed Nov 1 12:14:57 2017 -0400

    Containerize cinder services under pacemaker in CI

    This patch fixes a timing issue with cinder's db sync when
    cinder-manage is containerized but cinder-volume is not.

    Cinder services were containerized in [1], and this patch updates
    the CI jobs so cinder-volume and cinder-backup with pacemaker are
    also containerized.

    [1] https://review.openstack.org/479001

    Change-Id: Ic20af8a9bb24c4d21d1fd71bc65b001aa9c09c7c
    Closes-Bug: #1729253
    Closes-Bug: #1729339

Changed in tripleo:
status: In Progress → Fix Released

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b2 development milestone.

Change abandoned by Martin André (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/518856

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers