pike, volume failed to build in error status. list index out of range in cinder

[wes hayutin]

http://logs.openstack.org/85/564285/20/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/ef7b33c/logs/subnode-2/var/log/extra/errors.txt.gz

http://logs.openstack.org/85/564285/20/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/ef7b33c/logs/subnode-2/var/log/cinder/volume.log.txt.gz#_2018-06-20_02_17_09_291

2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service [req-f011b9be-8976-4c96-a88b-144446ffde94 - - - - -] Error starting thread.: IndexError: list index out of range
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service Traceback (most recent call last):
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 721, in run_service
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service service.start()
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/service.py", line 217, in start
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service service_id=Service.service_id)
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 441, in init_host
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service self.driver.init_capabilities()
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 704, in init_capabilities
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service stats = self.get_volume_stats(True)
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 483, in get_volume_stats
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service self._update_volume_stats()
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 466, in _update_volume_stats
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service free_capacity, total_capacity = self._get_pool_stats()
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 419, in _get_pool_stats
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service if pool['name'] == pool_name][0]['stats']
2018-06-20 02:17:09.291 ERROR /var/log/cinder/volume.log: 153072 ERROR oslo_service.service IndexError: list index out of range

Results in:
http://logs.openstack.org/85/564285/20/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/ef7b33c/logs/stackviz/#/stdin/test-details/tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
    return f(self, *func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 100, in test_volume_boot_pattern
    volume_origin = self._create_volume_from_image()
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 43, in _create_volume_from_image
    return self.create_volume(name=vol_name, imageRef=img_uuid)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 234, in create_volume
    volume['id'], 'available')
  File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 204, in wait_for_volume_resource_status
    resource_name=resource_name, resource_id=resource_id)
tempest.exceptions.VolumeResourceBuildErrorException: volume ef78d00c-e1d4-4cbe-ad81-c6e710f8273d failed to build and is in ERROR status

[wes hayutin (weshayutin)]

this looks ok
http://logs.openstack.org/85/564285/20/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/ef7b33c/logs/subnode-2/var/log/cinder/cinder-api.log.txt.gz#_2018-06-20_02_23_47_009

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/576851

[Quique Llorente (quiquell)]

Testing patch: https://review.openstack.org/#/c/576866/

[Giulio Fidente (gfidente)]

I think this is a duplicate of https://bugs.launchpad.net/tripleo/+bug/1777762

[Quique Llorente (quiquell)]

Looking at https://review.openstack.org/#/c/576866/ test patch:
- Previous failing scenearions 001 and 004 are passing now.
- We may have problems with upgrades, the added upgrades jobs is failing.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/576851
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=7d2415a8a8c94bbd9043edd957e7edb37629bc7c
Submitter: Zuul
Branch: master

commit 7d2415a8a8c94bbd9043edd957e7edb37629bc7c
Author: Wes Hayutin <email address hidden>
Date: Wed Jun 20 09:19:07 2018 -0400

    set the ceph container for pike

    A patch was made that change how the containers
    were pulled in [1]. A default to luminous was
    set in defaults that needs to be overriden
    in the release config.

    [1] https://github.com/openstack/tripleo-quickstart-extras/commit/b3794ff03ab841bbbc15a339e35a7ec02b193111

    Closes-Bug: #1777759
    Related-Bug: #1777762
    Change-Id: I82e422f30d63b2fe8808e585e2f0eed273c96b2a

[OpenStack Infra (hudson-openstack)]

Change abandoned by Quique Llorente (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/576866

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart 2.1.1 release.