swap volume test intermittently fails waiting for old volume to go to in-use after swap

[Matt Riedemann]

This is a failure from testing this change:

https://review.openstack.org/#/c/533285/

A run of this test failed here:

http://logs.openstack.org/89/532689/5/check/nova-multiattach/094e46b/job-output.txt.gz#_2018-01-23_12_43_58_792416

But the swap volume operation completed successfully in the nova-compute service:

http://logs.openstack.org/89/532689/5/check/nova-multiattach/094e46b/logs/screen-n-cpu.txt.gz#_Jan_23_11_19_22_009270

And the attachment for the volume and the instance was deleted here:

http://logs.openstack.org/89/532689/5/check/nova-multiattach/094e46b/logs/screen-n-cpu.txt.gz#_Jan_23_11_19_20_451671

Corresponding request in cinder-api logs:

http://logs.openstack.org/89/532689/5/check/nova-multiattach/094e46b/logs/screen-c-api.txt.gz#_Jan_23_11_19_20_554540

And the attachment delete in the cinder-volume logs:

http://logs.openstack.org/89/532689/5/check/nova-multiattach/094e46b/logs/screen-c-vol.txt.gz#_Jan_23_11_19_21_472221

So the volume should go back into in-use status.

Note that this was the old volume before we swapped to the new volume:

http://paste.openstack.org/show/650989/

Looking at the state of the volume when the test times out, the attachment that was deleted is not gone from the volume:

http://paste.openstack.org/show/650990/

But there are no errors in Cinder.

The attachment_delete code in the cinder volume API has some logic to check what remaining attachments exist and what their status values are to determine the overall status for the volume, so we'd have to know what the attachments are and their status once the attachment is deleted (the volume should still be attached to the other server).