test_attach_scsi_disk_with_config_drive intermittently fails at detaching volume
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tempest |
Fix Released
|
Undecided
|
Balazs Gibizer |
Bug Description
2020-12-01 07:23:50.118692 | controller | Captured traceback:
2020-12-01 07:23:50.118702 | controller | ~~~~~~~~~~~~~~~~~~~
2020-12-01 07:23:50.118712 | controller | Traceback (most recent call last):
2020-12-01 07:23:50.118730 | controller |
2020-12-01 07:23:50.118742 | controller | File "/opt/stack/
2020-12-01 07:23:50.118752 | controller | self.assertEqual(0, len(volume_
2020-12-01 07:23:50.118779 | controller |
2020-12-01 07:23:50.118807 | controller | File "/opt/stack/
2020-12-01 07:23:50.118825 | controller | self.assertThat
2020-12-01 07:23:50.118844 | controller |
2020-12-01 07:23:50.118861 | controller | File "/opt/stack/
2020-12-01 07:23:50.118871 | controller | raise mismatch_error
2020-12-01 07:23:50.118886 | controller |
2020-12-01 07:23:50.118895 | controller | testtools.
The volume reaches available state but nova still returns that the attachment exists[2]:
2020-12-01 07:23:50.122191 | controller | 2020-12-01 06:51:27,540 116889 INFO [tempest.
2020-12-01 07:23:50.122201 | controller | 2020-12-01 06:51:27,614 116889 INFO [tempest.
2020-12-01 07:23:50.122211 | controller | 2020-12-01 06:51:27,614 116889 DEBUG [tempest.
2020-12-01 07:23:50.122221 | controller | Body: None
2020-12-01 07:23:50.122230 | controller | Response - Headers: {'date': 'Tue, 01 Dec 2020 06:51:27 GMT', 'se
2020-12-01 07:23:50.122240 | controller | rver': 'Apache/2.4.41 (Ubuntu)', 'content-length': '197', 'content-type': 'application/json', 'openstack-
2020-12-01 07:23:50.122250 | controller | Body: b'{"volumeAttac
The nova code first delete the attachment in cinder and then delete the BDM from the nova DB[1]. So I think what we see here is a race condition in the test.
Here is a logstash signature[3]
[1] novacode: https:/
[2] example failure: https:/
[3] http://
Changed in tempest: | |
status: | Fix Committed → Fix Released |
As discussed the test is using the servers_client to lookup the active volume attachments of an instance via the n-api os-volume_ attachments API. There's a small window for a race when using this API to determine if a volume is attached as we only delete the block_device_ mapping record in Nova *after* we have marked the volume as detached (and thus `available`) in c-api. The test should be *waiting* until there are zero volume attachments listed for the instance via n-api.