VMDK Volume attach fails while attaching to an instance that is booted from VMDK volume

[Ajay Bajaj]

I have booted an instance from a volume, successfully booted,
now another volume, i try to attach to same instance, it is failing.
see the stack trace..

2014-07-04 08:56:11.391 TRACE oslo.messaging.rpc.dispatcher raise exception.InvalidDevicePath(path=root_device_name)
2014-07-04 08:56:11.391 TRACE oslo.messaging.rpc.dispatcher InvalidDevicePath: The supplied device path (vda) is invalid.
2014-07-04 08:56:11.391 TRACE oslo.messaging.rpc.dispatcher
2014-07-04 08:56:11.396 ERROR oslo.messaging._drivers.common [req-648122d5-fd39-495b-a3a7-a96bd32091d6 admin admin] Returning exception The supplied device path (vda) is invalid. to caller
2014-07-04 08:56:11.396 ERROR oslo.messaging._drivers.common [req-648122d5-fd39-495b-a3a7-a96bd32091d6 admin admin] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 401, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/exception.py", line 88, in wrapped\n payload)\n', ' File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 286, in decorated_function\n pass\n', ' File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 272, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 314, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 302, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 4201, in reserve_block_device_name\n return do_reserve()\n', ' File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 249, in inner\n return f(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 4188, in do_reserve\n context, instance, bdms, device)\n', ' File "/opt/stack/nova/nova/compute/utils.py", line 106, in get_device_name_for_instance\n mappings[\'root\'], device)\n', ' File "/opt/stack/nova/nova/compute/utils.py", line 155, in get_next_device_name\n raise exception.InvalidDevicePath(path=root_device_name)\n', 'InvalidDevicePath: The supplied device path (vda) is invalid.\n']

The reason behind this issue is: because of the root device_name being set 'vda' in the case of boot from volume, The future volume attaches to the VM fail saying "The supplied device path (vda) is invalid"

[Duncan Thomas (duncan-thomas)]

Try using vdb, or auto - vda is taken by the boot device. No a bug.

[Ajay Bajaj (akbajaj)]

It should belong to nova, so moved it to nova and moved to new.

[Chinmaya Bharadwaj (acbharadwaj)]

i'm able to reproduce the issue when tried from horizon. Since the boot volume gets mounted on 'vda' by default. Future attaches to the VM fail. If the same tried from CLI with the passed (something like : nova boot --flavor 1 --block-device source=volume,id=1730ba2e-5b16-40ae-89d8-745ce6647520,device=/dev/vda,dest=volume,size=2,shutdown=REMOVE,bootindex=0 myvm), volume attaches seem to pass.

[Nikola Đipanov (ndipanov)]

As per IRC conversation - this affects the vmware driver with Cinder vmdk driver.

[Nikola Đipanov (ndipanov)]

FWIW - I was unable to reproduce this with latest devstack running libvirt.

$ cinder create --image-id $CIRROS_ID --display-name boot-cirros 1
$ cinder create --display-name testvol 1
$ nova boot --boot-volume $BOOT-CIRROS-ID --flavor 1 testvm
$ # ssh into the instance all is fine
$ nova volume-attach testvm $TESTVOL-ID
$ # All good - the vm sees it as vdb

[Chinmaya Bharadwaj (acbharadwaj)]

yes, though the clients, there seems to be no problem.
Here, in Horizon device_name is being initialized to "vda".
https://github.com/openstack/horizon/blob/master/openstack_dashboard/dashboards/project/instances/workflows/create_insta
nce.py#L117

[Nikola Đipanov (ndipanov)]

Removing the VMWare tag as looking at the code it seems to affect all drivers.

Also marking as invalid for Nova - Horizon should not be making assumptions about the device name for attach.

[Julie Pichon (jpichon)]

Isn't the mount point set by the user when attaching a volume or launching with a volume? If not I think this may be related to/a duplicate of bug 1280297.

[Chinmaya Bharadwaj (acbharadwaj)]

@jpichon: its not mandatory. and we don't have to force the user to add the device.

[Julie Pichon (jpichon)]

Thanks for the comment Chinmaya, I filed bug 1339760 to track this in Horizon.

[Tracy Jones (tjones-i)]

not in progress if there is no one working on it

[Nikola Đipanov (ndipanov)]

patch posted https://review.openstack.org/#/c/105687/

[Nikola Đipanov (ndipanov)]

We should in addition to the fix above, probably make the compute_utils methods more resilient so that they don't fail for the instances/devices created before the already posted fix.

[Andrew Laski (alaski)]

I have just run into this while testing the xenapi driver, glad to see there's a fix up :)

I have a small patch to make the offending compute_util method more resilient which I can add a test to and push up.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/112975

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/105687
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=28b37c1a707f5e958221b4ee28c4832d081eb706
Submitter: Jenkins
Branch: master

commit 28b37c1a707f5e958221b4ee28c4832d081eb706
Author: Nikola Dipanov <email address hidden>
Date: Wed Jul 9 11:59:15 2014 +0200

    Prepend '/dev/' to supplied dev names in the API

    Make sure that all user input device names end up in the database with
    the '/dev/' prefix. This will make things more consistent and avoid
    issues when attempting to parse device names.

    This includes two main sources - block device mapping that can be passed
    as part of a request or as image metadata, and root device name, passed
    as image metadata.

    Closes-bug: #1337821
    Change-Id: Ibda82f511be99f1a68f2f77c72601a1b006be7a0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/112975
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=426d394b77584bb0f9be89e4a11df937448a5087
Submitter: Jenkins
Branch: master

commit 426d394b77584bb0f9be89e4a11df937448a5087
Author: Andrew Laski <email address hidden>
Date: Fri Aug 8 12:59:24 2014 -0400

    Prepend /dev/ to root_device_name in get_next_device_name

    Because the API docs for booting an instance from a volume have shown to
    use 'vda' for a long time, and it has worked, there may be instances
    with root_device_name = 'vda'. These instances will fail a check in
    get_next_device_name when attempting to attach another volume, making
    that impossible. Another patch fixes new block_device_mappings by
    prepending /dev/ when the root_device_name is populated, but this will
    allow incorrectly populated instances to attach volumes.

    Change-Id: I709a195c7e3f559315a2c307de62cbae8b72ac46
    Partial-Bug: #1337821

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/130528

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/130528
Reason: There has been no response to Dave's -1 since last year so dropping this.