unable to delete migrated volume with SolidFire as the destination

Bug #1733967 reported by John Griffith
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Undecided
John Griffith

Bug Description

The volume migration code does some things under the covers with UUID and Name swapping that is a bit tricky. The biggest challenge is that it creates a new volume (the target) with its' own UUID and db entry, then attaches the volume, migrates the data, then does a completion routine.

The completion routine is now moved into the object code, and that's good for the most part, but the object code also does a number of internal object things with names and ID's to try and keep track of what's what. In the case of a volume migrated from LVM (or some other backend) to SolidFire things look something like this:

cinder create --volume-type lvm 1
    volume.id = 1ae84477-c32a-4c90-9dfb-5ff804640631
    volume.name = volume-1ae84477-c32a-4c90-9dfb-5ff804640631
    volume.name_id = 1ae84477-c32a-4c90-9dfb-5ff804640631

cinder retype 1ae84477-c32a-4c90-9dfb-5ff804640631 solidfire --migration-policy on-demand
    We get a new volume:
        volume.id = 2730131b-1758-421b-86a8-f486b67ae2be
        volume.name = volume-2730131b-1758-421b-86a8-f486b67ae2be
        volume.name_id = 2730131b-1758-421b-86a8-f486b67ae2be
    Data is copied and we start the ID swapping magic and delete the original volume after
    swapping the ID. The end result is something like this:

        volume.id = 1ae84477-c32a-4c90-9dfb-5ff804640631
        volume.name_id = 2730131b-1758-421b-86a8-f486b67ae2be
The problem is depending on which side of a migration you're on (destination or source) the valid UUID that was used and recorded for creation is different. That means that in the one case the volume.id is valid, but in the case of a volume that was migrated off of the SF backend, it won't be able to be deleted because the id swap so it' needs the old name_id.

Basically, the whole migration thing is a bit flawed, we should probably revisit how it works at some point. For now, just add logic to SolidFire to try both values in delete to cover all of our bases since UUID is no longer really a UUID.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/522410

Changed in cinder:
assignee: nobody → John Griffith (john-griffith)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/522410
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=0e6ee1b1df38059f338ebd93c605dfb51a93a8a5
Submitter: Zuul
Branch: master

commit 0e6ee1b1df38059f338ebd93c605dfb51a93a8a5
Author: John Griffith <email address hidden>
Date: Thu Nov 23 00:30:57 2017 +0000

    Check for migrated UUID in SolidFire delete

    Migration of volumes kinda screws up the use of ID as the
    reference to use for deleting volumes.

    If a volume is migrated, the UUID (ID) gets swapped and no
    longer matches the UUID the volume had when it was initially
    created on the back end device. The result is that deletion
    of that volume fails on the back end because the handle to the
    new UUID doesn't exist.

    On the other hand, you also have to be careful in dealing with
    and cleaning up volumes migrated off of your back end device for
    the same types of reasons.

    This change just adds an additional check in the SF driver so that
    we are querying the back end device for the ID and the name_id.

    This way we make sure if the volume is on our device we clean it up.

    Change-Id: I60f8e8864f236fb92dd03560576377f1a0bdd82b
    Closes-Bug: #1733967

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/528828

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/pike)

Reviewed: https://review.openstack.org/528828
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=24825e932831286481f2eb610ac0ba3682c3f3ad
Submitter: Zuul
Branch: stable/pike

commit 24825e932831286481f2eb610ac0ba3682c3f3ad
Author: John Griffith <email address hidden>
Date: Thu Nov 23 00:30:57 2017 +0000

    Check for migrated UUID in SolidFire delete

    Migration of volumes kinda screws up the use of ID as the
    reference to use for deleting volumes.

    If a volume is migrated, the UUID (ID) gets swapped and no
    longer matches the UUID the volume had when it was initially
    created on the back end device. The result is that deletion
    of that volume fails on the back end because the handle to the
    new UUID doesn't exist.

    On the other hand, you also have to be careful in dealing with
    and cleaning up volumes migrated off of your back end device for
    the same types of reasons.

    This change just adds an additional check in the SF driver so that
    we are querying the back end device for the ID and the name_id.

    This way we make sure if the volume is on our device we clean it up.

    Change-Id: I60f8e8864f236fb92dd03560576377f1a0bdd82b
    Closes-Bug: #1733967
    (cherry picked from commit 0e6ee1b1df38059f338ebd93c605dfb51a93a8a5)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 12.0.0.0b3

This issue was fixed in the openstack/cinder 12.0.0.0b3 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 11.1.0

This issue was fixed in the openstack/cinder 11.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/571299

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/ocata)

Reviewed: https://review.openstack.org/571299
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=43f04f6989671a084ecd391f56af46df7ad5e0ab
Submitter: Zuul
Branch: stable/ocata

commit 43f04f6989671a084ecd391f56af46df7ad5e0ab
Author: John Griffith <email address hidden>
Date: Thu Nov 23 00:30:57 2017 +0000

    Check for migrated UUID in SolidFire delete

    Migration of volumes kinda screws up the use of ID as the
    reference to use for deleting volumes.

    If a volume is migrated, the UUID (ID) gets swapped and no
    longer matches the UUID the volume had when it was initially
    created on the back end device. The result is that deletion
    of that volume fails on the back end because the handle to the
    new UUID doesn't exist.

    On the other hand, you also have to be careful in dealing with
    and cleaning up volumes migrated off of your back end device for
    the same types of reasons.

    This change just adds an additional check in the SF driver so that
    we are querying the back end device for the ID and the name_id.

    This way we make sure if the volume is on our device we clean it up.

    Change-Id: I60f8e8864f236fb92dd03560576377f1a0bdd82b
    Closes-Bug: #1733967
    (cherry picked from commit 0e6ee1b1df38059f338ebd93c605dfb51a93a8a5)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 10.0.8

This issue was fixed in the openstack/cinder 10.0.8 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.