Check destination_type when booting with bdm provided

Bug #1644725 reported by Zhenyu Zheng
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Unassigned
OpenStack Compute (nova)
Opinion
Medium
Unassigned
python-novaclient
Fix Released
Medium
Zhenyu Zheng

Bug Description

When booting instance with block_device_mapping provided, in the current implementation, the "destination_type" is allowed to be None, and this lead to un-sync between Nova and Cinder:

Step 1: Booting with block_device_mapping, leave destination_type to be None:

root@SZX1000191849:/var/log/nova# nova --debug boot --flavor 1 --image 2ba75018-403f-407b-864a-08564022e1f8 --nic net-id=cce1d2f1-acf4-4646-abdc-069f8d0dbb71 --block-device 'source=volume,id=9f49d5b0-3625-46a2-9ed4-d82f19949148' test_bdm

the corresponding REST call is:
DEBUG (session:342) REQ: curl -g -i -X POST http://10.229.45.17:8774/v2.1/os-volumes_boot -H "Accept: application/json" -H "User-Agent: python-novaclient" -H "OpenStack-API-Version: compute 2.37" -H "X-OpenStack-Nova-API-Version: 2.37" -H "X-Auth-Token: {SHA1}4d8c2c43338e1c4d96e08bcd1c2f3ff36de14154" -H "Content-Type: application/json" -d '{"server": {"name": "test_bdm", "imageRef": "2ba75018-403f-407b-864a-08564022e1f8", "block_device_mapping_v2": [{"source_type": "image", "delete_on_termination": true, "boot_index": 0, "uuid": "2ba75018-403f-407b-864a-08564022e1f8", "destination_type": "local"}, {"source_type": "volume", "uuid": "9f49d5b0-3625-46a2-9ed4-d82f19949148"}], "flavorRef": "1", "max_count": 1, "min_count": 1, "networks": [{"uuid": "cce1d2f1-acf4-4646-abdc-069f8d0dbb71"}]}}'

Step 2: After the instance is successfully launched, the detailed info is like this:

root@SZX1000191849:/var/log/nova# nova show 83d9ec32-93e0-441a-ae10-00e08b65de0b
+--------------------------------------+----------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | SZX1000191849 |
| OS-EXT-SRV-ATTR:hostname | test-bdm |
| OS-EXT-SRV-ATTR:hypervisor_hostname | SZX1000191849 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000016 |
| OS-EXT-SRV-ATTR:kernel_id | 87c9afd6-3a47-4a4c-a804-6b456d68136d |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | acd02b28-6484-4f90-a5e7-bba7159343e1 |
| OS-EXT-SRV-ATTR:reservation_id | r-fiqwkq02 |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-11-25T06:50:36.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-11-25T06:50:30Z |
| description | - |
| flavor | m1.tiny (1) |
| hostId | 410c49382deba8e205044b231169b2a656b27b6c994815366712b62e |
| host_status | UP |
| id | 83d9ec32-93e0-441a-ae10-00e08b65de0b |
| image | cirros-0.3.4-x86_64-uec (2ba75018-403f-407b-864a-08564022e1f8) |
| key_name | - |
| locked | False |
| metadata | {} |
| name | test_bdm |
| os-extended-volumes:volumes_attached | [{"id": "9f49d5b0-3625-46a2-9ed4-d82f19949148", "delete_on_termination": false}] |
| private network | 2001:db8:8000::5, 10.0.0.13 |
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tags | [] |
| tenant_id | 405a8b8100ae47ffaa89730681ee400f |
| updated | 2016-11-25T06:50:36Z |
| user_id | 43db6c7e51754c0d8c8b87278144f789 |
+--------------------------------------+----------------------------------------------------------------------------------+

As been shown, a volume attach is recored in Nova

Step 3: Check the volume status in Cinder:

root@SZX1000191849:/var/log/nova# cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 9f49d5b0-3625-46a2-9ed4-d82f19949148 | available | - | 1 | lvmdriver-1 | false | |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+

The volume is still available.

The volume is actually not attached by checking the instance's XML.
We should forrbid the boot call with None destination_type both in Nova and NovaClient.

Changed in nova:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
description: updated
Changed in python-novaclient:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-novaclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/402364

Changed in python-novaclient:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/402372

Changed in nova:
status: New → In Progress
Changed in nova:
importance: Undecided → Medium
Changed in python-novaclient:
importance: Undecided → Medium
Revision history for this message
Ghanshyam Mann (ghanshyammann) wrote :

But Destination_type in bdm is optional, checks in nova will make it backward incompatible. I am not sure why cinder does not mark volume in-use.

Revision history for this message
Ken'ichi Ohmichi (oomichi) wrote :

Humm, interesting problem. I will try reproducing this problem on the gate with a new Tempest test.

Revision history for this message
Ken'ichi Ohmichi (oomichi) wrote :
Revision history for this message
Zhenyu Zheng (zhengzhenyu) wrote :

@Ken'ichi Ohmichi, maybe you will also be interested in this one:
https://bugs.launchpad.net/nova/+bug/1645175

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to python-novaclient (master)

Reviewed: https://review.openstack.org/402364
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=a58959dd9ec3962cbb6d273f136286809678b0c4
Submitter: Jenkins
Branch: master

commit a58959dd9ec3962cbb6d273f136286809678b0c4
Author: Kevin_Zheng <email address hidden>
Date: Fri Nov 25 16:55:18 2016 +0800

    Check source_type and destination_type when booting with bdm provided

    When booting instance with block_device_mapping provided,
    in the current implementation, the "destination_type" is
    allowed to be None, and this lead to un-sync between Nova
    and Cinder: Nova will have a bdm in its record while the
    volume is still in "available" status in cinder.

    Also we should check source_type here

    This patch adds a check for source_type and
    destination_type, source_type shoud be within
    'volume', 'image', 'snapshot' or 'blank';
    When empty destination_type is provided, it
    will be assigned to an default value according
    to source_type, if it is provided with a wrong
    value, command error will raise and information
    will shown.

    Change-Id: If6c7d8ed58a94d262ac6c300e75aca4cdf99ff1d
    partial-bug: #1644725

Changed in python-openstackclient:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
no longer affects: python-openstackclient
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Zhenyu Zheng (<email address hidden>) on branch: master
Review: https://review.openstack.org/402372

Revision history for this message
Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

Changed in nova:
status: In Progress → New
assignee: Zhenyu Zheng (zhengzhenyu) → nobody
Revision history for this message
Sean Dague (sdague) wrote :

It does feel like it might be better to fix this on the client side. Marking as opinion as the patch author abandoned the nova patch. It is welcome to come back later.

Changed in nova:
status: New → Opinion
Revision history for this message
Takashi Natsume (natsume-takashi) wrote :

For nova, the issue was fixed in the following bug report and patch.

https://bugs.launchpad.net/nova/+bug/1525806
https://review.openstack.org/#/c/258788/

Revision history for this message
Takashi Natsume (natsume-takashi) wrote :

The issue was fixed in python-novaclient 7.0.0.

Changed in python-novaclient:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.