cinder already deleted volume not handled

[Jay Rubenstein]

The cinder-solidfire storage plugin does not handle an already deleted volume properly. In this case with two management applications (cinder and SolidFire-API/GUI), if the SolidFire API or GUI is used to delete a cinder, created volume, and then cinder issues a delete for the volume a second volume in the access group gets deleted.

reproduction:

 created several volumes with cinder using SolidFire as the storage-array
 use the SolidFire API or GUI to delete one of those cinder-volumes (cinder-volume_x)
 use the SolidFire API or GUI to list the cinder-volumes
 use cinder to delete the same volume (cinder-volume_x) deleted in the previous line
 use the SolidFire API or GUI to list the remaining cinder-volumes you will find that the last
 volume in the previous list is missing a volume that was not intended to be deleted has now been
 deleted

 When using only cinder as the management tool this does not happen it is only when both cinder and the native SolidFire management tools are used that one can create this problem

FYI:
I work for Netapp/SolidFire and we are in the process of testing a patch

[Sean (seanwilcox)]

I have confirmed this issue affects the latest devstack release with the following steps :

1. Configure devstack trunk connected to SolidFire cluster for Cinder.
2. Create two volumes via Cinder.
3. Remove one of the Volumes using the Solidfire interface.
4. Got back to Cinder and attempt to remove the same volume removed in step 3.
5. The second volume that was created will be removed from the Solidfire cluster leaving the volume in the Cinder. Cinder will remove the expected volume from its own list of volumes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/584149

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/588342

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/601289

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/603113

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/603113
Reason: Looks like this is just an update of https://review.openstack.org/584149 gone horribly wrong.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/601289
Reason: Looks like this is just an update of https://review.openstack.org/584149 gone horribly wrong.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/588342
Reason: Looks like this is just an update of https://review.openstack.org/584149 gone horribly wrong.

[John Griffith (john-griffith)]

I think your problem actually is here: https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/solidfire.py#L586

The get by id will fail (it's only looking for active volumes IIRC) so it's going to fall through to there and just grab the first item in the list for account response and boom, you done deleted the wrong volume.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/615595

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/615595
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ed2c7b90376fa6f5cd6ef8f46fe9b2d408b0b756
Submitter: Zuul
Branch: stable/queens

commit ed2c7b90376fa6f5cd6ef8f46fe9b2d408b0b756
Author: Jay Rubenstein <email address hidden>
Date: Wed Jul 18 11:39:28 2018 -0600

    SF ensure the correct volume is deleted

    When two management systems are available openstack/cinder and
    SolidFire-API/GUI a volume can be deleted though either interface. Thus
    if a volume is deleted through the SolidFire-interface and then again
    through the openstack/cinder interface it was possible to delete another
    volume.

    The original code made an incorrect assumption on all the possible values returned
    from a query to find the target volume. This could lead to deleting a
    volume that was not the target of the original delete command from openstack/cinder.

    The correction in the module _get_sfvol_by_cinder_vref validates that the
    returned object/volumes-id matchs the requested volume-id with a
    comparison of svol['volumeID'] to int(sf_vid)

    Closes-Bug: #1782373
    Change-Id: I92a8e76c64c4e23dd185875e009132938ace5091
    (cherry picked from commit d1b3adfbe471ad99ddf0dd2b483b77435c157aea)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/623036

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/623036
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=6ceed81d4b25978ac64bf36afa45c021ef95d02f
Submitter: Zuul
Branch: stable/rocky

commit 6ceed81d4b25978ac64bf36afa45c021ef95d02f
Author: Jay Rubenstein <email address hidden>
Date: Wed Jul 18 11:39:28 2018 -0600

    SF ensure the correct volume is deleted

    When two management systems are available openstack/cinder and
    SolidFire-API/GUI a volume can be deleted though either interface. Thus
    if a volume is deleted through the SolidFire-interface and then again
    through the openstack/cinder interface it was possible to delete another
    volume.

    The original code made an incorrect assumption on all the possible values returned
    from a query to find the target volume. This could lead to deleting a
    volume that was not the target of the original delete command from openstack/cinder.

    The correction in the module _get_sfvol_by_cinder_vref validates that the
    returned object/volumes-id matchs the requested volume-id with a
    comparison of svol['volumeID'] to int(sf_vid)

    Closes-Bug: #1782373
    Change-Id: I92a8e76c64c4e23dd185875e009132938ace5091
    (cherry picked from commit d1b3adfbe471ad99ddf0dd2b483b77435c157aea)
    (cherry picked from commit ed2c7b90376fa6f5cd6ef8f46fe9b2d408b0b756)

[Jay Rubenstein (jarbassaidai)]

Bug #https://bugs.launchpad.net/cinder/+bug/1813210 is neutralized by this patch "1782383".
If you see DuplicateSFVolume and this patch "1782383" is on your system then you will not delete random volumes!

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 13.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 14.0.0.0rc1 release candidate.