lvremove failure on volume delete when creating and deleting volumes with the same name

[Matthew Treinish]

The issue appears when quickly sending api commands to nova-volume and cinder: create volume, get volume, and delete volume are all run in quick succession at least twice with the same volume displayName for the volumes.

In other words creating then deleting a volume and immediatly after the delete is confirmed creating a volume with the same name and then trying to delete the second volume results in lvremove failing to remove the logical volume.

lvremove fails because the lv open count is 1. The open count in this case is caused by the iscsi export still running on the volume. The tgtadm check return no exports during the check because the wrong tid was used to check if there was an iscsi export. In my testing with tempest --tid=2 was used by nova-volume and cinder when the check erroneously returned 22, but manually running the tgtadm show operation with --tid=1 showed the iscsi export.

I believe this may be caused by a race condition with the previous volume's delete when creating the second volume that had the same displayName. When I changed the displayName of the second volume the failure no longer occurs.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/11947

[John Griffith (john-griffith)]

Turns out this can be reproduced fairly easily even using unique or no display_names.

The issue here is that in certain cases the tid selected/returned by tgt-admin is NOT the next sequential target id. As a result the tgt id info stored in the db's iscsi_targets is incorrect and when we attempt to delete it will fail.

I can hack around this and check when the delete fails for this particular situation and FIND the correct id, BUT... the bigger problem here is that the iscsi_targets table data is wrong. Trying to spend a little more time on figuring out a fix for the root cause here, but may have to revert the persistent targets change.

Also, I believe this bug has the same root cause as: https://bugs.launchpad.net/cinder/+bug/1038062