Attach volume with libvirt disregards target device but still reserves it

[Nikola Đipanov]

Running devstack with libvirt/qemu - the problem is that attaching a volume (either by passing it with --block_device_mapping to boot or by using nova volume-attach) completely disregards the device name passed as can be seen fromt the folloowng shell session. However the device remains reserved so subsequent attach attempts will fail on the specified device, and succeed with some other given (which will not be honored again).

The following session is how to reproduce it:

[ndipanov@devstack devstack]$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| 5792f1ed-c5f7-40c6-913f-43aa66c717c7 | available | bootable | 3 | None | |
| abc77933-119b-4105-b085-092c93be36f5 | available | blank_2 | 1 | None | |
| b4de941a-627c-447a-9226-456159d95173 | available | blank | 1 | None | |
+--------------------------------------+-----------+--------------+------+-------------+-------------+
[ndipanov@devstack devstack]$ nova list

[ndipanov@devstack devstack]$ nova boot --image c346fdd1-d438-472b-98f5-b4c5f2b716f8 --flavor 1 --block_device_mapping vdr=b4de941a-627c-447a-9226-456159d95173:::0 --key_name nova_key w_vol
+------------------------+--------------------------------------+
| Property | Value |
+------------------------+--------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | CqgT4dXkq64t |
| config_drive | |
| created | 2012-11-07T14:02:00Z |
| flavor | m1.tiny |
| hostId | |
| id | caa459d5-27ae-4c5b-b190-fd740054a2ec |
| image | cirros-0.3.0-x86_64-uec |
| key_name | nova_key |
| metadata | {} |
| name | w_vol |
| progress | 0 |
| security_groups | [{u'name': u'default'}] |
| status | BUILD |
| tenant_id | 5f68e605463940dda20e876604385c43 |
| updated | 2012-11-07T14:02:01Z |
| user_id | 104895e85fe54ae5a2cc5c5a650f50b0 |
+------------------------+--------------------------------------+
[ndipanov@devstack devstack]$ nova list
        +--------------------------------------+-------+--------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+-------+--------+------------------+
| caa459d5-27ae-4c5b-b190-fd740054a2ec | w_vol | ACTIVE | private=10.0.0.2 |
+--------------------------------------+-------+--------+------------------+
[ndipanov@devstack devstack]$ ssh -o StrictHostKeyChecking=no -i nova_key.priv cirros@10.0.0.2
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
57:ca:fe:e8:4f:b6:30:5a:06:20:d4:37:ed:06:aa:c8.
Please contact your system administrator.
Add correct host key in /home/ndipanov/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/ndipanov/.ssh/known_hosts:2
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
$ cat /proc/partitions
major minor #blocks name

 253 0 24576 vda
 253 16 1048576 vdb
$ ls -la /dev | grep vd.*
lrwxrwxrwx 1 root root 3 Nov 7 07:03 root -> vda
brw------- 1 root root 253, 0 Nov 7 07:02 vda
brw------- 1 root root 253, 16 Nov 7 07:02 vdb
$ Connection to 10.0.0.2 closed.
[ndipanov@devstack devstack]$ cinder list
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| 5792f1ed-c5f7-40c6-913f-43aa66c717c7 | available | bootable | 3 | None | |
| abc77933-119b-4105-b085-092c93be36f5 | available | blank_2 | 1 | None | |
| b4de941a-627c-447a-9226-456159d95173 | in-use | blank | 1 | None | caa459d5-27ae-4c5b-b190-fd740054a2ec |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
(reverse-i-search)`vol': nova boot --image c346fdd1-d438-472b-98f5-b4c5f2b716f8 --flavor 1 --block_device_mapping vdr=b4de941a-627c-447a-9226-456159d95173:::0 --key_name nova_key w_^Cl
[ndipanov@devstack devstack]$ nova volume-attach w_vol abc77933-119b-4105-b085-092c93be36f5 /dev/vdr
ERROR: The supplied device path (/dev/vda) is in use.
Traceback (most recent call last):

  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)

  File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)

  File "/opt/stack/nova/nova/exception.py", line 117, in wrapped
    temp_level, payload)

  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()

  File "/opt/stack/nova/nova/exception.py", line 92, in wrapped
    return f(*args, **kw)

  File "/opt/stack/nova/nova/compute/manager.py", line 175, in decorated_function
    pass

  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()

  File "/opt/stack/nova/nova/compute/manager.py", line 161, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/opt/stack/nova/nova/compute/manager.py", line 202, in decorated_function
    kwargs['instance']['uuid'], e, sys.exc_info())

  File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()

  File "/opt/stack/nova/nova/compute/manager.py", line 190, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/opt/stack/nova/nova/compute/manager.py", line 2167, in reserve_block_device_name
    return do_reserve()

  File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 229, in inner
    retval = f(*args, **kwargs)

  File "/opt/stack/nova/nova/compute/manager.py", line 2160, in do_reserve
    device)

  File "/opt/stack/nova/nova/compute/utils.py", line 110, in get_device_name_for_instance
    raise exception.DevicePathInUse(path=device)

DevicePathInUse: The supplied device path (/dev/vda) is in use.
 (HTTP 400) (Request-ID: req-76852a0d-1c70-4e11-879c-0089956ba02e)
[ndipanov@devstack ~]$ nova volume-attach w_vol abc77933-119b-4105-b085-092c93be36f5 /dev/vdc
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdc |
| id | abc77933-119b-4105-b085-092c93be36f5 |
| serverId | caa459d5-27ae-4c5b-b190-fd740054a2ec |
| volumeId | abc77933-119b-4105-b085-092c93be36f5 |
+----------+--------------------------------------+

This seems to be somehow related to the changes introduced by https://bugs.launchpad.net/nova/+bug/1004328

[Nikola Đipanov (ndipanov)]

Since it is now possible to both boot instances and attach volumes without specifying device names after https://blueprints.launchpad.net/nova/+spec/improve-block-device-handling BP has been implemented. in which case the device names will be handled properly by Nova.

It is still possible to supply device names (for backwards compatibility's sake), which would cause the same behavior as described above. This is really an issue due to the fact that there is no way to make sure libvirt uses the same device name as supplied to it since libvirt only takes this as ordering hints. the best solution really _is_ to rely on Nova to actually choose the device name as per implemented BP.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/189632

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/189632
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0283234e837d9faf807e6e8da6ec6321ee56b31a
Submitter: Jenkins
Branch: master

commit 0283234e837d9faf807e6e8da6ec6321ee56b31a
Author: Nikola Dipanov <email address hidden>
Date: Tue Jun 9 10:05:14 2015 +0100

    libvirt: Always default device names at boot

    Up until now, libvirt would default it's own device names unless they
    were specified by the user at boot. This could lead to inconsistencies,
    so the soft approach was to discourage users from passing in explicit
    device names, by documenting the fact that they will not be honoured,
     but not override them.

    It turns out that this approach causes more trouble, especially when we
    add attaching volumes into the mix. So the approach now is to make
    libvirt _always_ override what is passed in. That way even if the user
    specifies a device name, the attach will not fail due to a device name
    collision (a common problem - see related bug), and Nova book keeping
    will match what we can see in the instance on most OSes.

    In order to get there we now make sure boot starts acting like that
    before we switch attach_volume over too, so that there is consistent
    behavior.

    Also - make tests for defaulting device names more realistic as they now
    actually run the code instead of just making sure the right methods get
    called.

    UpgradeImpact: After this commit, supplying a device name for any of
    the block devices specified as part of the `nova boot` call will not be
    honoured by any libvirt compute nodes.

    Change-Id: I76a7cfd995db6c04f7af48ff8c9acdd55750ed76
    Related-bug: #1075971

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/229612

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/kilo
Review: https://review.openstack.org/229612
Reason: This didn't make test_stamp_pattern magically start working in the kilo compat job.