Volumes attached to shelved instance may contain incorrect device_name

[Zhenyu Zheng]

Nova supported to attach and detach volumes to/from shelved instances
since microversion 2.20.

When we attach a volume to an instance and
then unshelve the instance the cinder side doesn't include the
device_name information.

How to re-produce:

#1 shelve an instance

stack@SZX1000280461:/opt/devstack$ nova list
---+-------+--------+------------+-------------+------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------------------+
| bd09421c-90b2-411c-99d0-fcf07338c542 | test2 | ACTIVE | - | Running | Kevin=10.0.0.76, 172.24.4.10 |
+--------------------------------------+-------+--------+------------+-------------+------------------------------+

stack@SZX1000280461:/opt/devstack$ nova shelve bd09421c-90b2-411c-99d0-fcf07338c542

stack@SZX1000280461:/opt/devstack$ nova list
+--------------------------------------+-------+-------------------+------------+-------------+------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+-------------------+------------+-------------+------------------------------+
| bd09421c-90b2-411c-99d0-fcf07338c542 | test2 | SHELVED_OFFLOADED | - | Shutdown | Kevin=10.0.0.76, 172.24.4.10 |
+--------------------------------------+-------+-------------------+------------+-------------+------------------------------+

# 2 attach a cinder volume to the shelved insntace:

stack@SZX1000280461:/opt/devstack$ cinder list
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+
| a72f9642-ca8f-4c2e-bfe0-362c6220d498 | available | test1 | 1 | lvmdriver-1 | false | |
+--------------------------------------+-----------+-------+------+-------------+----------+-------------+

stack@SZX1000280461:/opt/devstack$ nova volume-attach bd09421c-90b2-411c-99d0-fcf07338c542 a72f9642-ca8f-4c2e-bfe0-362c6220d498
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | - |
| id | a72f9642-ca8f-4c2e-bfe0-362c6220d498 |
| serverId | bd09421c-90b2-411c-99d0-fcf07338c542 |
| volumeId | a72f9642-ca8f-4c2e-bfe0-362c6220d498 |
+----------+--------------------------------------+

stack@SZX1000280461:/opt/devstack$ cinder list
+--------------------------------------+--------+-------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+-------+------+-------------+----------+--------------------------------------+
| a72f9642-ca8f-4c2e-bfe0-362c6220d498 | in-use | test1 | 1 | lvmdriver-1 | false | bd09421c-90b2-411c-99d0-fcf07338c542 |
+--------------------------------------+--------+-------+------+-------------+----------+--------------------------------------+

stack@SZX1000280461:/opt/devstack$ nova show test2
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| Kevin network | 10.0.0.76, 172.24.4.10 |
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hostname | test2 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000004 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-z869pf5v |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | shelved_offloaded |
| OS-SRV-USG:launched_at | 2017-03-23T02:34:04.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2017-03-23T02:33:19Z |
| description | - |
| flavor | ds1G (d2) |
| hostId | |
| host_status | |
| id | bd09421c-90b2-411c-99d0-fcf07338c542 |
| image | CentOS (d826a98b-4ab5-4ce1-93f4-394193abaa51) |
| key_name | kevin_322 |
| locked | False |
| metadata | {} |
| name | test2 |
| os-extended-volumes:volumes_attached | [{"id": "a72f9642-ca8f-4c2e-bfe0-362c6220d498", "delete_on_termination": false}] |
| security_groups | default |
| status | SHELVED_OFFLOADED |
| tags | [] |
| tenant_id | 4f705543436648abbacf5719d3397c0f |
| updated | 2017-03-23T02:57:54Z |
| user_id | 46a372c04e38463b898dc71626d8846a |
+--------------------------------------+----------------------------------------------------------------------------------+

#3 Unshelve the instance:

stack@SZX1000280461:/opt/devstack$ nova unshelve bd09421c-90b2-411c-99d0-fcf07338c542

stack@SZX1000280461:/opt/devstack$ nova list
+--------------------------------------+-------+--------+------------+-------------+------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+------------------------------+
| bd09421c-90b2-411c-99d0-fcf07338c542 | test2 | ACTIVE | - | Running | Kevin=10.0.0.76, 172.24.4.10 |
+--------------------------------------+-------+--------+------------+-------------+------------------------------+

#4 Check details in cinder:
root@SZX1000280461:~# cinder show a72f9642-ca8f-4c2e-bfe0-362c6220d498
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{'server_id': 'bd09421c-90b2-411c-99d0-fcf07338c542', 'attachment_id': '16e65459-23e9-4af9-8481-c75366d4f51b', 'attached_at': '2017-03-23T06:11:26.000000', 'host_name': None, 'volume_id': 'a72f9642-ca8f-4c2e-bfe0-362c6220d498', 'device': None, 'id': 'a72f9642-ca8f-4c2e-bfe0-362c6220d498'}] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-03-23T02:49:07.000000 |
| description | |
| encrypted | False |
| id | a72f9642-ca8f-4c2e-bfe0-362c6220d498 |
| metadata | {'attached_mode': 'rw'} |
| migration_status | None |
| multiattach | False |
| name | test1 |
| os-vol-host-attr:host | SZX1000280461@lvmdriver-1#lvmdriver-1 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 4f705543436648abbacf5719d3397c0f |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| updated_at | 2017-03-23T06:11:58.000000 |
| user_id | 46a372c04e38463b898dc71626d8846a |
| volume_type | lvmdriver-1 |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

After studied more, I discovered that this is because _attach_volume_shelved_offloaded uses _create_volume_bdm with param: _is_local_createion=True and it just create an bdm in DB and then call cinder to modify its DB. When the instance is unshelved, the actual attach operation will take place and there will be no updates in cinder DB.

This causes TWO problems,
1. nova volume-attach can be used with user provided device_name, in general case this name will be checked or ignored. In the shelved case, it will pass directly to Cinder thus Cinder will have wrong device_name [1], for example, I provide /dev/vda, when I do cinder show, it will show /dev/vda:
[1] http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n3719

root@SZX1000280461:/var/log/nova# nova volume-attach bd09421c-90b2-411c-99d0-fcf07338c542 a72f9642-ca8f-4c2e-bfe0-362c6220d498 /dev/vda
+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | - |
| id | a72f9642-ca8f-4c2e-bfe0-362c6220d498 |
| serverId | bd09421c-90b2-411c-99d0-fcf07338c542 |
| volumeId | a72f9642-ca8f-4c2e-bfe0-362c6220d498 |
+----------+--------------------------------------+

root@SZX1000280461:/var/log/nova# cinder show a72f9642-ca8f-4c2e-bfe0-362c6220d498
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{'server_id': 'bd09421c-90b2-411c-99d0-fcf07338c542', 'attachment_id': '46dc8d6a-9148-4e91-bc28-c1947c618ebb', 'attached_at': '2017-03-23T08:00:37.000000', 'host_name': None, 'volume_id': 'a72f9642-ca8f-4c2e-bfe0-362c6220d498', 'device': '/dev/vda', 'id': 'a72f9642-ca8f-4c2e-bfe0-362c6220d498'}] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2017-03-23T02:49:07.000000 |
| description | |
| encrypted | False |
| id | a72f9642-ca8f-4c2e-bfe0-362c6220d498 |
| metadata | {'attached_mode': 'rw'} |
| migration_status | None |
| multiattach | False |
| name | test1 |
| os-vol-host-attr:host | SZX1000280461@lvmdriver-1#lvmdriver-1 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 4f705543436648abbacf5719d3397c0f |
| replication_status | None |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| updated_at | 2017-03-23T08:00:37.000000 |
| user_id | 46a372c04e38463b898dc71626d8846a |
| volume_type | lvmdriver-1 |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

root@SZX1000280461:/var/log/nova# nova volume-attachments bd09421c-90b2-411c-99d0-fcf07338c542
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| ID | DEVICE | SERVER ID | VOLUME ID |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| a72f9642-ca8f-4c2e-bfe0-362c6220d498 | /dev/vdc | bd09421c-90b2-411c-99d0-fcf07338c542 | a72f9642-ca8f-4c2e-bfe0-362c6220d498 |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+

2. if user don't provide device_name, the Cinder side device_name will always be None.

[Matt Riedemann (mriedem)]

This is mostly working as designed.

When attaching a volume to a shelved offloaded instance, there is no host for the instance so we can't call to the compute manager to reserve the device like we do for a normal attach here:

http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n3687

So we create the BDM in the API and don't set a device name, because we can't say it's going to be the device name when we unshelve the instance:

http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/api.py#n3678

We've long talked about just removing the device_name option from the volume attach API because we can't guarantee that what the user requests is what's going to be in the guest, the hypervisor controls that, not nova. The libvirt driver completely ignores the user-requested device name.

So I think we want a spec to microversion the device_name field out of the volume attach request since it's just confusing and we can't honor it.

The thing we need to check is it the os-attach API in Cinder requires the mountpoint (which is the device name in the nova BDM).

[Matt Riedemann (mriedem)]

It looks like cinder's os-attach API does require the mountpoint parameter:

https://github.com/openstack/cinder/blob/504f129714c60eb2d24192462113db971a798b1a/cinder/api/contrib/volume_actions.py#L62

In fact, it looks like a request would fail with a 500 error if the mountpoint is not provided since it doesn't check and would raise a KeyError if not provided.

[Matt Riedemann (mriedem)]

Cinder bug 1575675 is for the 500 error in the os-attach API when mountpoint is not provided.

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

[Sean Dague (sdague)]

Working as design, device_name is since removed from the API.