Attempting to Manage an Already Managed Cinder Volume Will Succeed

[Dustin Schoenbrun]

When issuing a manage command to a volume that is already managed by Cinder, the operation does not fail and you will wind up in an unexpected state. The operation will succeed, the backend storage will be renamed, and there will be two Cinder volumes in the available state with only one of them being valid. This appears to affect all Cinder drivers.

Steps to Reproduce on LVM Driver:

1) Create a Cinder volume using "cinder create" command.
2) Look at the name of the backing logical volume using "lvdisplay" command
3) Using the logical volume name issue the manage command "cinder manage <host> --source-name <name_of_lv>"

This will produce two Cinder volumes in the available state (with only one being truly backed by a logical volume) and the logical volume being renamed to match the new cinder volume.

[Rushi Agrawal (rushiagr)]

I was trying to reproduce that, but seems I'm in a soup doing that! As an admin user, I'm successfully able to unmanage a volume, but can't "manage" it back. It says 404 service not found with devstack. :(

[Dustin Schoenbrun (dustin-schoenbrun-v)]

I just attempted the reproduction steps I listed above on the latest devstack available from github and got the following results. I've attached my commands and the output and the relevant log statement from Cinder Scheduler.

[Rushi Agrawal (rushiagr)]

Thanks Dustin. I'm able to reproduce the bug, so marked confirmed..

[Glenn M. Gobeli (ggobeli)]

Hey folks, I've been investigating this issue and have concluded the following:

First of all, the primary issues is that a Cloud Admin can provided a source-name from the backend storage (including LVM) for a volume that is already owned by Cinder if the source-name is the full volume name and not just the ID (e.g. volume-99327fhe2-4857-fh3244); the driver will locate the given source-name on its backend and proceed to place it under Cinder management, thus destroying the original Cinder volume linkage.

I am given to understand that the backend driver can use any name it wants for the volume it creates. I think most use the volume['name'] field which is always volume-<volume ID>. If we assume that the vendor can use anything, then a possible solution is to make changes to the Cinder API level of the manage existing command to do a database lookup to see if the provided source-name exists in the Cinder DB. This would work IF we had a new Cinder DB field, say "backend-name", and IF every Cinder driver was required to populate this field.

We could assume that everyone uses the volume['name'] field in naming the backend volume. Then at the API level we could strip off volume- and do a Cinder DB check for an existing volume ID match.

Need to know how to proceed...

[John Griffith (john-griffith)]

I just cleaned up the manage command a bit here:
https://review.openstack.org/#/c/125491/

Sadly I didn't see this bug or test this case, but it's certainly an issue. Unmanage does seem to work fine for me, but we should be checking on manage that we aren't already managing the volume. That being said we do put onus on the admin here so I'm not inclined to mark it as a critical bug, but def something that needs fixed in Kilo.

[Glenn M. Gobeli (ggobeli)]

So if I am hearing you correctly John, a solution for this bug needs to be on hold until the changes you are making get in. Then we can see how this issue appears in the aftermath; still there or not and how to address a possible resolution.

Correct?

[John Griffith (john-griffith)]

@Glenn,

No sorry for the confusion, I was just pointing that patch out here as there was some input that some folks trying to test this had difficulty getting the command to run. The issue will still be the exact same, we need to add a check for this, but that's not as straight forward as I'd necessarily like. It's going to require diving down into the Drivers I think for them to verfiy/check if the volume is managed.

[Anton Arefiev (aarefiev)]

We could be checking on manage that we are already managing the volume by ref['source-name']( by volume ID), but there is one case: what about situation if user create new volume via lvcreate command. He would want any name, how resolve this situation? Need more info...

[Anton Arefiev (aarefiev)]

Solution: at the volume/api check ref['source-name'] , if it exists in db - return, else create new one.
Question: what if volume was created via lvcreate command with custom name. And we want to manage it in cinder, it still will produce 2 volumes. Is it possible?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/139071

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/156939

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/139071
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=a1e4ad9ff2e9366a5f41b176b5a88d451079557c
Submitter: Jenkins
Branch: master

commit a1e4ad9ff2e9366a5f41b176b5a88d451079557c
Author: Anton Arefiev <email address hidden>
Date: Mon Feb 9 15:30:45 2015 +0200

    Add entry create and cast tasks to manage workflow

    This change adds manage existing api task in flow. The task
    is used in the volume api to provide full value task flow for
    manage existing process. All errors occurred during manage
    flow set volume to 'error' state.

    Entry creating moved from volume api to EntryCreateTask. Also
    added ManageCastTask to provide manage process to scheduler.

    Related-Bug: #1364550
    Change-Id: I12a4311953c1c86d584b5bf2fe2888e5b5127d43

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156939
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=eee9edd479e0e1d7b6f497e40d4eba59f31449b7
Submitter: Jenkins
Branch: master

commit eee9edd479e0e1d7b6f497e40d4eba59f31449b7
Author: Anton Arefiev <email address hidden>
Date: Wed Feb 18 12:44:09 2015 +0200

    Fix lvm manage existing volume

    When trying to manage command to a volume that is already managed by
    Cinder, the operation does not fail and you will end up in an unexpected
    state. The operation will succeed, the backend storage will be renamed,
    and there will be two Cinder volumes in the available state with only
    one of them being valid.

    This change adds volume check on driver level (since we don't know
    source-name, source-id or anything else used as ref for volume, each
    driver should be responsible for figuring out what to do with specified
    volume ref). Check method looking for possible existing volume in db
    to avoid managing already managed volume.

    Change-Id: I0d8d7a3cb2afce90529cd1142b413aca9ac69d4c
    Partial-Bug: #1364550