Launch instance failed using instances' snapshot created volume

Bug #1476114 reported by Zhenyu Zheng on 2015-07-20
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Undecided
Kimiya Yokoo
OpenStack Compute (nova)
Medium
omkar_telee

Bug Description

Launching instance fails when using a volume that is created using a snapshot of a volume-backended instance.

How to reproduce:

Step 1:
Create an volume backended instance.

root@zheng-dev1:/var/log/nova# nova boot --flavor 1 --boot-volume daaddb77-4257-4ccd-86f2-220b31a0ce9b --nic net-id=8744ee96-7690-43bb-89b4-fcac805557bc test1

root@zheng-dev1:/var/log/nova# nova list
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+
| ef3c6074-4d38-4d7b-8d93-d0ace58d3a6a | test1 | ACTIVE | - | Running | public=2001:db8::6, 172.24.4.5 |
+--------------------------------------+-------+--------+------------+-------------+--------------------------------+

Step 2:
Create a snapshot of this instance using nova image-create, this will create an image in glance.

root@zheng-dev1:/var/log/nova# nova image-create ef3c6074-4d38-4d7b-8d93-d0ace58d3a6a test-image
root@zheng-dev1:/var/log/nova# glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| 7bdff9a3-d051-4e75-bcd3-de69dbffe063 | cirros-0.3.4-x86_64-uec | ami | ami | 25165824 | active |
| 2af2dce2-f778-4d73-b827-5281741fc1cf | cirros-0.3.4-x86_64-uec-kernel | aki | aki | 4979632 | active |
| 60ea7020-fcc1-4535-af5e-0e894a01a44a | cirros-0.3.4-x86_64-uec-ramdisk | ari | ari | 3740163 | active |
| ce7b2d17-196a-4871-bc1b-9dcb184863be | test-image | | | | active |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+

Step 3:
Create a new volume using the previously created image.

root@zheng-dev1:/var/log/nova# cinder create --image-id ce7b2d17-196a-4871-bc1b-9dcb184863be --name test-volume 1
+---------------------------------------+--------------------------------------+
| Property | Value |
+---------------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2015-07-20T06:44:41.000000 |
| description | None |
| encrypted | False |
| id | cc21dc7d-aa4b-4e24-8f11-8b916c5d6347 |
| metadata | {} |
| multiattach | False |
| name | test-volume |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | b8112a8d8227490eba99419b8a8c2555 |
| os-volume-replication:driver_data | None |
| os-volume-replication:extended_status | None |
| replication_status | disabled |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| user_id | ed64bccd0227444fa02dbd7695769a7d |
| volume_type | lvmdriver-1 |
+---------------------------------------+--------------------------------------+
root@zheng-dev1:/var/log/nova# cinder list
+--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+
| cc21dc7d-aa4b-4e24-8f11-8b916c5d6347 | available | test-volume | 1 | lvmdriver-1 | true | |
| daaddb77-4257-4ccd-86f2-220b31a0ce9b | in-use | bootable-volume | 1 | lvmdriver-1 | true | ef3c6074-4d38-4d7b-8d93-d0ace58d3a6a |
+--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+

Step 4:
Using this volume to boot an new instance.

root@zheng-dev1:/var/log/nova# nova boot --flavor 1 --boot-volume cc21dc7d-aa4b-4e24-8f11-8b916c5d6347 --nic net-id=8744ee96-7690-43bb-89b4-fcac805557bc test2

ERROR LOG:

2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack [req-079d1169-0bbb-4b0a-bb1b-a72467585385 ed64bccd0227444fa02dbd7695769a7d b8112a8d8227490eba99419b8a8c2555 - - -] Caught error: 'unicode' object has no attribute 'update'
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack Traceback (most recent call last):
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 126, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return req.get_response(self.application)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack application, catch_exc_info=False)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 536, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack response = request.get_response(self._app)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack application, catch_exc_info=False)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1281, in call_application
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 136, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack response = self.app(environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return resp(environ, start_response)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 756, in __call__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack content_type, body, accept)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 821, in _process_stack
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 911, in dispatch
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack return method(req=request, **action_args)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 643, in create
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack self._handle_create_exception(*sys.exc_info())
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 472, in _handle_create_exception
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack six.reraise(*exc_info)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 628, in create
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack check_server_group_quota=check_server_group_quota)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/hooks.py", line 149, in inner
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack rv = f(*args, **kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 1479, in create
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack check_server_group_quota=check_server_group_quota)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 1114, in _create_instance
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack block_device_mapping, legacy_bdm)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 738, in _check_and_transform_bdm
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack base_options, instance_type, image_meta, root_device_name)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/compute/api.py", line 684, in _get_image_defined_bdms
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack image_defined_bdms = image_properties.get('block_device_mapping', [])
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack File "/opt/stack/nova/nova/block_device.py", line 83, in __init__
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack bdm_dict.update(kwargs)
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack AttributeError: 'unicode' object has no attribute 'update'
2015-07-20 14:45:27.450 12962 ERROR nova.api.openstack
2015-07-20 14:45:27.455 12962 INFO nova.api.openstack [req-079d1169-0bbb-4b0a-bb1b-a72467585385 ed64bccd0227444fa02dbd7695769a7d b8112a8d8227490eba99419b8a8c2555 - - -] http://10.250.10.17:8774/v2/b8112a8d8227490eba99419b8a8c2555/os-volumes_boot returned with HTTP 500

Changed in nova:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)

Fix proposed to branch: master
Review: https://review.openstack.org/203500

Changed in nova:
status: New → In Progress

Change abandoned by Zhenyu Zheng (<email address hidden>) on branch: master
Review: https://review.openstack.org/203500

Changed in nova:
assignee: Zhenyu Zheng (zhengzhenyu) → nobody
status: In Progress → Confirmed
John Garbutt (johngarbutt) wrote :

So looks like the above fix was OK ish. We need to double check this still happens, then fix up this case in the API

As this is a 500 error the fix does not require a microversion, it's a bug that could potentially be backported.

tags: added: api low-hanging-fruit
tags: added: volumes
Changed in nova:
importance: Undecided → Medium
Changed in nova:
assignee: nobody → Zhenyu Zheng (zhengzhenyu)
Zhenyu Zheng (zhengzhenyu) wrote :

@John Garbutt, I have checked and it seems correct now in master branch.

Zhenyu Zheng (zhengzhenyu) wrote :

reproduced in master:
root@zhenyu-dev:/opt/stack/nova# git log
commit 1a643bf934e5b113a61b0dc9f812e66f55f19769
Merge: d966d33 fb9a0b8
Author: Jenkins <email address hidden>
Date: Wed May 17 00:26:21 2017 +0000

    Merge "Make NovaException format errors fatal for tests"

Nova API log
http://paste.openstack.org/show/609733/

Kimiya Yokoo (y-kimi) on 2017-06-05
Changed in cinder:
assignee: nobody → Kimiya Yokoo (y-kimi)
Kimiya Yokoo (y-kimi) on 2017-06-05
Changed in cinder:
status: New → Confirmed
Kimiya Yokoo (y-kimi) on 2017-06-05
Changed in cinder:
status: Confirmed → In Progress
Sean Dague (sdague) on 2017-06-23
Changed in nova:
assignee: Zhenyu Zheng (zhengzhenyu) → nobody
Changed in nova:
assignee: nobody → omkar_telee (omkar-telee)

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/473061
Reason: No updates in a long time. Feel free to restore and update if you wish to continue working on this.

Changed in cinder:
status: In Progress → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers