nova_volume_attach adds volume detach cleanup twice

[Abdallah Yasin]

Cinder Tempest test_encrypted_cinder_volumes_cryptsetup randomly fails with {"code": 400, "message": "Invalid volume: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be \'in-use\' and attach_status must be \'attached\' to detach. (HTTP 400) (Request-ID: req-8b4f3ddf-2bda-46ad-ad2b-71be67413523)"}. This is not the only test that fails with the same error, sometimes other tests fails with the same error.

This seems to be handled in https://bugs.launchpad.net/tempest/+bug/1722577, so after further investigation, it turned out that the issue is in the nova_volume_attach function, where the cleanup is actually added twice for a one attach command, once in self.servers_client.attach_volume found in [1], and then added another time in the nova_volume_attach function itself found at [2].

The mentioned bug is only handled in [1] but not [2].

Further Logs can be found at [3].

[1] https://github.com/openstack/tempest/blob/b2c04d4c502725f50ba3ce879911ad9610476b25/tempest/api/compute/base.py#L578

[2] https://github.com/openstack/tempest/blob/2c3da54a82e8844ea575c93576324855793738e1/tempest/scenario/manager.py#L736

[3] http://13.74.249.42/437/Cinder-tgtadm-Daily/

[Lee Yarwood (lyarwood)]

Yup another hit below:

https://1b330c50dd371d8d6f8e-c67b93b429db84769e417a4684ceffa6.ssl.cf2.rackcdn.com/757111/9/check/nova-multi-cell/fce0115/testr_results.html

I've pushed a change to drop the cleanup from nova_volume_attach below:

https://review.opendev.org/c/openstack/tempest/+/767515

[Ghanshyam Mann (ghanshyammann)]

These are two different code path, one is for API test
- https://github.com/openstack/tempest/blob/b2c04d4c502725f50ba3ce879911ad9610476b25/tempest/api/compute/base.py#L578

and the below one is for scenario tests that do not use the above method and call service client directly

https://github.com/openstack/tempest/blob/2c3da54a82e8844ea575c93576324855793738e1/tempest/scenario/manager.py#L724

So there should not be race between these. Also in nova_volume_attach() cleanup part, it is adding nova_volume_detach() as cleanup method which does wait for the volume to be detached so not sure what it is causing race here.

[Lee Yarwood (lyarwood)]

Both tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_cryptsetup and tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_scenario already make their own calls to nova_volume_detach thus the removal in my change of the additional cleanup call in nova_volume_attach.

I'm not sure what you're point is regarding the api tests, c#0 confirms that the issue is already resolved there.

[Martin Kopec (mkopec)]

https://review.opendev.org/c/openstack/tempest/+/767515

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 26.1.0 release.