Description
===========
Cinder volume failed to build and went to ERROR state - No valid backend was found ( Stderr: ' Volume group "cinder-volumes" not found\n)
Error Snippet
=============
https://9323961f9b936186ab18-8b23fb57107c099fc623f82f25f9ddb3.ssl.cf1.rackcdn.com/719441/1/gate/tripleo-ci-centos-8-standalone/d3002ff/job-output.txt
~~~
2020-04-15 01:15:58.537228 | primary | TASK [os_tempest : Execute tempest tests] **************************************
2020-04-15 01:15:58.537249 | primary | Wednesday 15 April 2020 01:15:58 +0000 (0:00:00.072) 1:25:23.938 *******
2020-04-15 01:29:47.889427 | primary | fatal: [undercloud]: FAILED! => {
2020-04-15 01:29:47.889530 | primary | "changed": false,
2020-04-15 01:29:47.889565 | primary | "cmd": "set -e\nif [ -d /openstack/venvs/tempest-untagged/bin ];\nthen\n. /openstack/venvs/tempest-untagged/bin/activate\nfi\ntempest run --concurrency 2 --blacklist-file /home/zuul/tempest/etc/tempest_blacklist.txt --whitelist-file /home/zuul/tempest/etc/tempest_whitelist.txt > /var/log/tempest/tempest_run.log\n",
2020-04-15 01:29:47.889593 | primary | "delta": "0:13:48.239156",
2020-04-15 01:29:47.889616 | primary | "end": "2020-04-15 01:29:47.335787",
2020-04-15 01:29:47.889642 | primary | "rc": 1,
2020-04-15 01:29:47.889663 | primary | "start": "2020-04-15 01:15:59.096631"
2020-04-15 01:29:47.889700 | primary | }
~~~
https://9323961f9b936186ab18-8b23fb57107c099fc623f82f25f9ddb3.ssl.cf1.rackcdn.com/719441/1/gate/tripleo-ci-centos-8-standalone/d3002ff/logs/undercloud/var/log/tempest/tempest_run.log
~~~
{0} tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario [34.963675s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
return f(*func_args, **func_kwargs)
File "/usr/lib/python3.6/site-packages/tempest/scenario/test_minimum_basic.py", line 118, in test_minimum_basic_scenario
volume = self.create_volume()
File "/usr/lib/python3.6/site-packages/tempest/scenario/manager.py", line 327, in create_volume
volume['id'], 'available')
File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 210, in wait_for_volume_resource_status
resource_name=resource_name, resource_id=resource_id)
tempest.exceptions.VolumeResourceBuildErrorException: volume b975e166-9c64-4adb-93bd-2735821bd1fb failed to build and is in ERROR status
~~~
https://9323961f9b936186ab18-8b23fb57107c099fc623f82f25f9ddb3.ssl.cf1.rackcdn.com/719441/1/gate/tripleo-ci-centos-8-standalone/d3002ff/logs/undercloud/var/log/containers/cinder/cinder-scheduler.log
~~~
6c6f2c96a667478abe6fbe2d77871a2b - default default] Setting Volume b975e166-9c64-4adb-93bd-2735821bd1fb to error due to: No valid backend was found. No weighed backends available error_out /usr/lib/python3.6/site-packages/cinder/volume/flows/common.py:83
~~~
https://9323961f9b936186ab18-8b23fb57107c099fc623f82f25f9ddb3.ssl.cf1.rackcdn.com/719441/1/gate/tripleo-ci-centos-8-standalone/d3002ff/logs/undercloud/var/log/containers/cinder/cinder-volume.log
~~~
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager Traceback (most recent call last):
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 467, in _init_host
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager self.driver.check_for_setup_error()
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/drivers/lvm.py", line 319, in check_for_setup_error
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager self.configuration.lvm_suppress_fd_warnings))
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/brick/local_dev/lvm.py", line 107, in __init__
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager if self._vg_exists() is False:
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/brick/local_dev/lvm.py", line 140, in _vg_exists
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager run_as_root=True)
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in _execute
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager result = self.__execute(*args, **kwargs)
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 126, in execute
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager return processutils.execute(*cmd, **kwargs)
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager cmd=sanitized_cmd)
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager Command: env LC_ALL=C vgs --noheadings -o name cinder-volumes
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager Exit code: 5
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager Stdout: ''
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager Stderr: ' Volume group "cinder-volumes" not found\n Cannot process volume group cinder-volumes\n'
2020-04-15 01:11:30.059 68 ERROR cinder.volume.manager
~~~
Another example:-
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bac/719438/4/gate/tripleo-ci-centos-8-scenario002-standalone/baccdd9/logs/undercloud/var/log/tempest/tempest_run.log
barbican_tempest_plugin.tests.scenario.test_volume_encryption.VolumeEncryptionTest.test_encrypted_cinder_volumes_cryptsetup [46.582934s] ... FAILED
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
return f(*func_args, **func_kwargs)
File "/usr/lib/python3.6/site-packages/barbican_tempest_plugin/tests/scenario/test_volume_encryption.py", line 121, in test_encrypted_cinder_volumes_cryptsetup
volume_type='cryptsetup')
File "/usr/lib/python3.6/site-packages/barbican_tempest_plugin/tests/scenario/test_volume_encryption.py", line 55, in create_encrypted_volume
return self.create_volume(volume_type=volume_type['name'])
File "/usr/lib/python3.6/site-packages/barbican_tempest_plugin/tests/scenario/manager.py", line 236, in create_volume
volume['id'], 'available')
File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 210, in wait_for_volume_resource_status
resource_name=resource_name, resource_id=resource_id)
tempest.exceptions.VolumeResourceBuildErrorException: volume 81ce11e9-c98f-48fc-a576-fda0dcb58713 failed to build and is in ERROR status
https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bac/719438/4/gate/tripleo-ci-centos-8-scenario002-standalone/baccdd9/logs/undercloud/var/log/containers/cinder/cinder-scheduler.log
~~~
42569dbea34c47fa8043148e84e9d0d4 - default default] Setting Volume 81ce11e9-c98f-48fc-a576-fda0dcb58713 to error due to: No valid backend was found. No weighed backends available error_out /usr/lib/python3.6/site-packages/cinder/volume/flows/common.py:83
~~~
~~~
Stderr: ' Volume group "cinder-volumes" not found\n Cannot process volume group cinder-volumes\n'
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager Traceback (most recent call last):
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 467, in _init_host
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager self.driver.check_for_setup_error()
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/volume/drivers/lvm.py", line 319, in check_for_setup_error
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager self.configuration.lvm_suppress_fd_warnings))
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/brick/local_dev/lvm.py", line 107, in __init__
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager if self._vg_exists() is False:
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/brick/local_dev/lvm.py", line 140, in _vg_exists
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager run_as_root=True)
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/os_brick/executor.py", line 52, in _execute
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager result = self.__execute(*args, **kwargs)
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 126, in execute
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager return processutils.execute(*cmd, **kwargs)
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager cmd=sanitized_cmd)
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager Command: env LC_ALL=C vgs --noheadings -o name cinder-volumes
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager Exit code: 5
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager Stdout: ''
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager Stderr: ' Volume group "cinder-volumes" not found\n Cannot process volume group cinder-volumes\n'
2020-04-14 23:04:56.541 65 ERROR cinder.volume.manager
~~~
Investigation so far:-
Cinder error:-
2020-04-15 01:11:30.059 68 ERROR cinder. volume. manager Stderr: ' Volume group "cinder-volumes" not found\n Cannot process volume group cinder-volumes\n'
Good:-
https:/ /e8dabbe3d00824 f512de- e25d1f8bae8601a ed7ec89510f7333 bc.ssl. cf2.rackcdn. com/718545/ 19/check/ tripleo- ci-centos- 8-standalone/ ab81b23/ logs/undercloud /home/zuul/ ansible. log
~~~ ******* ******* ******* ******* *******
2020-04-14 02:06:41,931 p=17519 u=root | TASK [cinder create LVM volume group] *******
2020-04-14 02:06:41,932 p=17519 u=root | Tuesday 14 April 2020 02:06:41 +0000 (0:00:00.333) 0:01:17.941 *********
2020-04-14 02:06:42,599 p=17519 u=root | changed: [standalone]
~~~
https:/ /e8dabbe3d00824 f512de- e25d1f8bae8601a ed7ec89510f7333 bc.ssl. cf2.rackcdn. com/718545/ 19/check/ tripleo- ci-centos- 8-standalone/ ab81b23/ logs/undercloud /var/log/ extra/lvm. txt
~~~ volumes- pool cinder-volumes twi-aotz-- <9.54g 0.00 10.58
VG #PV #LV #SN Attr VSize VFree
cinder-volumes 1 1 0 wz--n- <10.04g 484.00m
PV VG Fmt Attr PSize PFree
/dev/loop2 cinder-volumes lvm2 a-- <10.04g 484.00m
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
cinder-
~~~
https:/ /e8dabbe3d00824 f512de- e25d1f8bae8601a ed7ec89510f7333 bc.ssl. cf2.rackcdn. com/718545/ 19/check/ tripleo- ci-centos- 8-standalone/ ab81b23/ logs/undercloud /var/log/ journal. txt localdomain ansible- command[ 20051]: Invoked with creates=/dev/loop2 executable= /bin/bash _raw_params=if ! losetup /dev/loop2; then
losetup /dev/loop2 /var/lib/ cinder/ cinder- volumes
fi
if ! pvdisplay | grep cinder-volumes; then
pvcreate /dev/loop2
fi
if ! vgdisplay | grep cinder-volumes; then
vgcreate cinder-volumes /dev/loop2
fi
_uses_ shell=True warn=True stdin_add_ newline= True strip_empty_ ends=True argv=None chdir=None removes=None stdin=None localdomain kernel: loop: module loaded localdomain systemd[1]: Created slice system- lvm2\x2dpvscan. slice. localdomain systemd[1]: Starting LVM event activation on device 7:2... localdomain lvm[20065]: pvscan[20065] PV /dev/loop2 not used.
~~~
Apr 14 02:06:42 standalone.
Apr 14 02:06:42 standalone.
Apr 14 02:06:42 standalone.
Apr 14 02:06:42 standalone.
Apr 14 02:06:42 standalone.
~~~
Bad:-
https:/ /storage. gra.cloud. ovh.net/ v1/AUTH_ dcaab5e32b234d5 6b626f72581e364 4c/zuul_ opendev_ logs_3ea/ 715006/ 8/gate/ tripleo- ci-centos- 8-standalone/ 3ea0819/ logs/undercloud /home/zuul/ ansible. log
~~~ ******* ******* *****.. .
2020-04-14 12:43:13,479 p=28349 u=root | TASK [cinder create LVM volume group] *******