Grenade-multinode jobs error "Timed out waiting for volume cinder_grenade_vol2 status to be available"

[Slawek Kaplonski]

Since few days all (at least neutron) grenade-multinode jobs are failing with error like:

2020-10-24 09:23:03.342 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:_wait_for_volume_update:64 : echo 'Volume cinder_grenade_vol2 status is detaching and not yet available, waiting...'
2020-10-24 09:23:03.342 | Volume cinder_grenade_vol2 status is detaching and not yet available, waiting...
2020-10-24 09:23:03.344 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:_wait_for_volume_update:65 : sleep 1
2020-10-24 09:23:04.348 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:_wait_for_volume_update:66 : timeleft=0
2020-10-24 09:23:04.350 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:_wait_for_volume_update:67 : [[ 0 == 0 ]]
2020-10-24 09:23:04.352 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:_wait_for_volume_update:68 : die 68 'Timed out waiting for volume cinder_grenade_vol2 status to be available'
2020-10-24 09:23:04.354 | + /opt/stack/new/devstack/functions-common:die:198 : local exitcode=0
2020-10-24 09:23:04.356 | [Call Trace]
2020-10-24 09:23:04.356 | /opt/stack/new/grenade/projects/70_cinder/resources.sh:284:verify
2020-10-24 09:23:04.356 | /opt/stack/new/grenade/projects/70_cinder/resources.sh:228:_wait_for_volume_update
2020-10-24 09:23:04.356 | /opt/stack/new/grenade/projects/70_cinder/resources.sh:68:die
2020-10-24 09:23:04.360 | [ERROR] /opt/stack/new/grenade/projects/70_cinder/resources.sh:68 Timed out waiting for volume cinder_grenade_vol2 status to be available
2020-10-24 09:23:05.362 | 1 die /opt/stack/new/devstack/functions-common
2020-10-24 09:23:05.362 | 68 _wait_for_volume_update /opt/stack/new/grenade/projects/70_cinder/resources.sh
2020-10-24 09:23:05.362 | 228 verify /opt/stack/new/grenade/projects/70_cinder/resources.sh
2020-10-24 09:23:05.362 | 284 main /opt/stack/new/grenade/projects/70_cinder/resources.sh
2020-10-24 09:23:05.362 | Exit code: 1

Example of such failed job: https://zuul.opendev.org/t/openstack/build/74ec9d634a564141afd7f07680689175/logs

Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Timed%20out%20waiting%20for%20volume%20cinder_grenade_vol2%20status%20to%20be%20available%5C%22

[Brian Rosmaita (brian-rosmaita)]

melwitt thinks the reason behind this is the "Parameterized generics cannot be used with class" error that shows up in the nova compute log in the log URL you gave:

http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018289.html

She seems to have isolated the problem to an incompatibility between python 3.6 and the version of libvirt-python currently allowed in upper-constraints:

http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018287.html

Also, melwitt has a patch up to change the u-c, but there seems to be some controversy:

https://review.opendev.org/#/c/759552/

[Brian Rosmaita (brian-rosmaita)]

Nova took a different approach to fixing the problem:
https://review.opendev.org/#/c/759831/

Let's see if that fixes this error.

[Brian Rosmaita (brian-rosmaita)]

Closing as a duplicate of the nova bug.