Volume attached at different path than reported by nova/cinder when using config drive

[Clark Boylan]

When booting a node with config drive then attaching a cinder volume we see conflicting information about where the cinder volume is attached. In this case the config drive is at /dev/vdb, the cinder volume is at /dev/vdc but volume show reports the cinder volume to be at /dev/vdb.

This shows how one can get nova/cinder in this state using openstackclient (assuming that config drives attach to /dev/vdb):

$ OS_CLIENT_CONFIG_FILE=../vexxhost/vexx.yaml venv/bin/openstack --os-cloud openstackci-vexxhost server create --config-drive True --key-name clarkb-work --image ubuntu-trusty --flavor v1-standard-2 clarkb-test
+--------------------------------------+------------------------------------------------------+
| Field | Value |
+--------------------------------------+------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | redacted |
| config_drive | True |
| created | 2016-03-17T21:12:38Z |
| flavor | v1-standard-2 (ca2a6e9c-2236-4107-8905-7ae9427132ff) |
| hostId | |
| id | cdaf7671-5a5c-4118-a422-d8aae096bd1e |
| image | ubuntu-trusty (d42c4ce4-3fa0-4dcb-877e-1d66a05a4f8d) |
| key_name | clarkb-work |
| name | clarkb-test |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| project_id | projectid |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | BUILD |
| updated | 2016-03-17T21:12:38Z |
| user_id | userid |
+--------------------------------------+------------------------------------------------------+
$ OS_CLIENT_CONFIG_FILE=../vexxhost/vexx.yaml venv/bin/openstack --os-cloud openstackci-vexxhost volume create --size 100 clarkb-test
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| attachments | [] |
| availability_zone | ca-ymq-2 |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-03-17T21:13:13.683882 |
| description | None |
| encrypted | False |
| id | a2bba82c-ef08-48fe-9c3c-eafc8644207c |
| multiattach | False |
| name | clarkb-test |
| properties | |
| replication_status | disabled |
| size | 100 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| type | None |
| user_id | userid |
+---------------------+--------------------------------------+
$ OS_CLIENT_CONFIG_FILE=../vexxhost/vexx.yaml venv/bin/openstack --os-cloud openstackci-vexxhost server add volume clarkb-test clarkb-test
$ OS_CLIENT_CONFIG_FILE=../vexxhost/vexx.yaml venv/bin/openstack --os-cloud openstackci-vexxhost volume show clarkb-test
+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'cdaf7671-5a5c-4118-a422-d8aae096bd1e', u'attachment_id': u'18f184b3-123e-4b5c-af66-d5c5b93af5ed', u'host_name': None, |
| | u'volume_id': u'a2bba82c-ef08-48fe-9c3c-eafc8644207c', u'device': u'/dev/vdb', u'id': u'a2bba82c-ef08-48fe-9c3c-eafc8644207c'}] |
| availability_zone | ca-ymq-2 |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-03-17T21:13:13.000000 |
| description | None |
| encrypted | False |
| id | a2bba82c-ef08-48fe-9c3c-eafc8644207c |
| multiattach | False |
| name | clarkb-test |
| os-vol-tenant-attr:tenant_id | tenantid |
| os-volume-replication:driver_data | None |
| os-volume-replication:extended_status | None |
| properties | attached_mode='rw', readonly='False' |
| replication_status | disabled |
| size | 100 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | None |
| user_id | userid |
+---------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+
$ ssh ubuntu@199.19.213.70 ls -l /dev/disk/by-id
total 0
lrwxrwxrwx 1 root root 9 Mar 17 21:13 virtio-a2bba82c-ef08-48fe-9 -> ../../vdc
$ ssh ubuntu@199.19.213.70 ls -l /dev/disk/by-label
total 0
lrwxrwxrwx 1 root root 10 Mar 17 21:12 cloudimg-rootfs -> ../../vda1
lrwxrwxrwx 1 root root 9 Mar 17 21:12 config-2 -> ../../vdb

As a workaround we have specified the device path when attaching the volume but this is apparently deprecated and not expected to work properly. Therefore it is important that volume show report the correct device path.

One possible option here if libvirt cannot provide this info reliably is that cinder/nova seem to reliably use /dev/by-id/virtio-someportionofvolumeuuidhere. The api could possibly return this path instead since it is deterministic.

[Sylvain Bauza (sylvain-bauza)]

Possibly related to https://bugs.launchpad.net/nova/+bug/1558343/comments/2

[Sylvain Bauza (sylvain-bauza)]

Nevermind my previous comment #1

[Matt Riedemann (mriedem)]

Hmm, might be that when nova attaches the volume it's assuming /dev/vdb is available and that's what it tells cinder it's going to attach to:

https://github.com/openstack/nova/blob/a023c32c70b5ddbae122636c26ed32e5dcba66b2/nova/virt/block_device.py#L304

But the block_device_info code isn't taking into account the config drive? Probably need ndipanov to look at this.

[Sean McGinnis (sean-mcginnis)]

Appears to be an issue on the Nova side, but marking Cinder as Invalid for now instead of removing it in case there are any changes required on the Cinder side to address this.

[Matt Riedemann (mriedem)]

Note that in liberty, the libvirt driver should be ignoring any user-requested device name:

https://review.openstack.org/#/c/189632/

So I'm not sure how you can workaround this by specifying a device name.

[Nikola Đipanov (ndipanov)]

Nova's libvirt driver has code that is trying to account for the presence of a config drive, but looking at it - it will always give it the highest letter on it's bus, so volumes will come before it.

http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/blockinfo.py?id=f03bbbab68408e0b3a311256962211b4f5d2cd1e#n599

If this does not match with the behavior libvirt exhibits - it could be a bug in Nova (we try to mimic libvirt), or it may be down to libvirt having changed something.

It would be really good to know the version of libvirt this was tested against.

[Augustina Ragwitz (auggy)]

Clark, could you provide the version of libvirt? Thanks!

[Clark Boylan (cboylan)]

No I cannot because I do not run the cloud I am merely a cloud user. But I can ask the folks at vexxhost to provide this info if they are able.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

[Jim Thomas (jimt)]

I am currently experiencing this problem and would like to add more information.

I have a Mitaka installation that exhibits this behavior when attaching a volume to a running instance with a config drive. The config drive is attached as /dev/vdb, and the requested volume is correctly attached at /dev/vdc. However, the attach point for the volume reported in horizon and in the openstack client is /dev/vdb, the same device as the config drive.

libvirt version:
 libvirtd (libvirt) 1.2.9.3

[Augustina Ragwitz (auggy)]

This was moved to "New" status because the missing libvirt version info has now been provided. Clark confirmed this was a sufficient update to his original bug report.

[Dan Smith (danms)]

I propose we close this as invalid because:

1. We're not going to fix it (and can't for things like KVM)
2. We've discussed removing it from the API in a microversion because it's usually a lie
3. We have other (imperfect, but better) ways of exposing this as of newton (i.e. device metadata)

The use case here is, of course, completely valid and the concerns over not being able to identify devices that are attached (volumes, nics, etc) are very important. The device metadata work set out to fix that, and largely has, IMHO, modulo the unchangeably static nature of configdrive.