OpenStack Compute (nova)

Launch instance failed using instances' snapshot created volume

Bug #1476114 reported by Zhenyu Zheng
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Won't Fix
Undecided
Kimiya Yokoo
OpenStack Compute (nova)
Confirmed
Medium
omkar_telee

Bug Description

Launching instance fails when using a volume that is created using a snapshot of a volume-backended instance.

How to reproduce:

Step 1:
Create an volume backended instance.

root@zheng-dev1:/var/log/nova# nova boot --flavor 1 --boot-volume daaddb77-4257-4ccd-86f2-220b31a0ce9b --nic net-id=8744ee96-7690-43bb-89b4-fcac805557bc test1

root@zheng-dev1:/var/log/nova# nova list
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+
| ef3c6074-4d38-4d7b-8d93-d0ace58d3a6a | test1 | ACTIVE | - | Running | public=2001:db8::6, 172.24.4.5 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+

Step 2:
Create a snapshot of this instance using nova image-create, this will create an image in glance.

root@zheng-dev1:/var/log/nova# nova image-create ef3c6074-4d38-4d7b-8d93-d0ace58d3a6a test-image
root@zheng-dev1:/var/log/nova# glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| 7bdff9a3-d051-4e75-bcd3-de69dbffe063 | cirros-0.3.4-x86_64-uec | ami | ami | 25165824 | active |
| 2af2dce2-f778-4d73-b827-5281741fc1cf | cirros-0.3.4-x86_64-uec-kernel | aki | aki | 4979632 | active |
| 60ea7020-fcc1-4535-af5e-0e894a01a44a | cirros-0.3.4-x86_64-uec-ramdisk | ari | ari | 3740163 | active |
| ce7b2d17-196a-4871-bc1b-9dcb184863be | test-image | | | | active |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+

Step 3:
Create a new volume using the previously created image.

root@zheng-dev1:/var/log/nova# cinder create --image-id ce7b2d17-196a-4871-bc1b-9dcb184863be --name test-volume 1
+---------------------------------------+--------------------------------------+
| Property | Value |
+---------------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2015-07-20T06:44:41.000000 |
| description | None |
| encrypted | False |
| id | cc21dc7d-aa4b-4e24-8f11-8b916c5d6347 |
| metadata | {} |
| multiattach | False |
| name | test-volume |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | b8112a8d8227490eba99419b8a8c2555 |
| os-volume-replication:driver_data | None |
| os-volume-replication:extended_status | None |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| user_id | ed64bccd0227444fa02dbd7695769a7d |
| volume_type | lvmdriver-1 |
+---------------------------------------+--------------------------------------+
root@zheng-dev1:/var/log/nova# cinder list
+--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+
| cc21dc7d-aa4b-4e24-8f11-8b916c5d6347 | available | test-volume | 1 | lvmdriver-1 | true | |
| daaddb77-4257-4ccd-86f2-220b31a0ce9b | in-use | bootable-volume | 1 | lvmdriver-1 | true | ef3c6074-4d38-4d7b-8d93-d0ace58d3a6a |
+--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+

Step 4:
Using this volume to boot an new instance.

root@zheng-dev1:/var/log/nova# nova boot --flavor 1 --boot-volume cc21dc7d-aa4b-4e24-8f11-8b916c5d6347 --nic net-id=8744ee96-7690-43bb-89b4-fcac805557bc test2

ERROR LOG:

2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack [req-079d1169-0bbb-4b0a-bb1b-a72467585385 ed64bccd0227444fa02dbd7695769a7d b8112a8d8227490eba99419b8a8c2555 - - -] Caught error: 'unicode' object has no attribute 'update'
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack Traceback (most recent call last):
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 126, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return req.get_response(self.application)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack application, catch_exc_info=False)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 536, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack response = request.get_response(self._app)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack application, catch_exc_info=False)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack response = self.app(environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 756, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack content_type, body, accept)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 821, in _process_stack
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 911, in dispatch
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return method(req=request, **action_args)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 643, in create
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack self._handle_create_exception(*sys.exc_info())
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 472, in _handle_create_exception
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack six.reraise(*exc_info)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 628, in create
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack check_server_group_quota=check_server_group_quota)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/hooks.py", line 149, in inner
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack rv = f(*args, **kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 1479, in create
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack check_server_group_quota=check_server_group_quota)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 1114, in _create_instance
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack block_device_mapping, legacy_bdm)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 738, in _check_and_transform_bdm
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack base_options, instance_type, image_meta, root_device_name)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 684, in _get_image_defined_bdms
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack image_defined_bdms = image_properties.get('block_device_mapping', [])
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/block_device.py", line 83, in __init__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack bdm_dict.update(kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack AttributeError: 'unicode' object has no attribute 'update'
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack
2015-07-20 14:45:27.455 12962 INFO nova.api.openstack [req-079d1169-0bbb-4b0a-bb1b-a72467585385 ed64bccd0227444fa02dbd7695769a7d b8112a8d8227490eba99419b8a8c2555 - - -] http://10.250.10.17:8774/v2/b8112a8d8227490eba99419b8a8c2555/os-volumes_boot returned with HTTP 500

Zhenyu Zheng (zhengzhenyu)
Changed in nova:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/203500

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Zhenyu Zheng (<email address hidden>) on branch: master
Review: https://review.openstack.org/203500

Davanum Srinivas (DIMS) (dims-v)
Changed in nova:
assignee: Zhenyu Zheng (zhengzhenyu) → nobody
status: In Progress → Confirmed
Revision history for this message
John Garbutt (johngarbutt) wrote :

So looks like the above fix was OK ish. We need to double check this still happens, then fix up this case in the API

As this is a 500 error the fix does not require a microversion, it's a bug that could potentially be backported.

tags: added: api low-hanging-fruit
tags: added: volumes
Changed in nova:
importance: Undecided → Medium
Zhenyu Zheng (zhengzhenyu)
Changed in nova:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
Revision history for this message
Zhenyu Zheng (zhengzhenyu) wrote :

@John Garbutt, I have checked and it seems correct now in master branch.

Revision history for this message
Zhenyu Zheng (zhengzhenyu) wrote :

reproduced in master:
root@zhenyu-dev:/opt/stack/nova# git log
commit 1a643bf934e5b113a61b0dc9f812e66f55f19769
Merge: d966d33 fb9a0b8
Author: Jenkins <email address hidden>
Date: Wed May 17 00:26:21 2017 +0000

    Merge "Make NovaException format errors fatal for tests"

Nova API log
http://paste.openstack.org/show/609733/

Kimiya Yokoo (y-kimi)
Changed in cinder:
assignee: nobody → Kimiya Yokoo (y-kimi)
Kimiya Yokoo (y-kimi)
Changed in cinder:
status: New → Confirmed
Kimiya Yokoo (y-kimi)
Changed in cinder:
status: Confirmed → In Progress
Sean Dague (sdague)
Changed in nova:
assignee: Zhenyu Zheng (zhengzhenyu) → nobody
omkar_telee (omkar-telee)
Changed in nova:
assignee: nobody → omkar_telee (omkar-telee)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/473061
Reason: No updates in a long time. Feel free to restore and update if you wish to continue working on this.

Sean McGinnis (sean-mcginnis)
Changed in cinder:
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.