Ubuntu
nova package

Nova unable to download images from glance with rbd backend (juno/rc1)

Bug #1379434 reported by James Page
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Invalid
Undecided
Unassigned
nova (Ubuntu)
Invalid
Undecided
Unassigned
python-glance-store (Ubuntu)
Fix Released
High
Unassigned

Bug Description

2014-10-09 17:00:41.356 9242 AUDIT nova.compute.manager [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Starting instance...
2014-10-09 17:00:41.435 9242 WARNING nova.compute.resource_tracker [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Host field should not be set on the instance until resources have been claimed.
2014-10-09 17:00:41.435 9242 WARNING nova.compute.resource_tracker [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Node field should not be set on the instance until resources have been claimed.
2014-10-09 17:00:41.437 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Attempting claim: memory 512 MB, disk 1 GB
2014-10-09 17:00:41.437 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Total memory: 3952 MB, used: 512.00 MB
2014-10-09 17:00:41.438 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] memory limit: 5928.00 MB, free: 5416.00 MB
2014-10-09 17:00:41.438 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Total disk: 9 GB, used: 0.00 GB
2014-10-09 17:00:41.439 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] disk limit not specified, defaulting to unlimited
2014-10-09 17:00:41.449 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Claim successful
2014-10-09 17:00:41.550 9242 INFO nova.scheduler.client.report [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] Compute_service record updated for ('juju-t-machine-15', 'juju-t-machine-15.openstacklocal')
2014-10-09 17:00:41.871 9242 INFO nova.virt.libvirt.driver [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Creating image
2014-10-09 17:00:41.929 9242 ERROR glanceclient.common.http [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 ] Request returned failure status 500.
2014-10-09 17:00:41.930 9242 ERROR nova.compute.manager [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Instance failed to spawn
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Traceback (most recent call last):
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2231, in _build_resources
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] yield resources
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2101, in _build_and_run_instance
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] block_device_info=block_device_info)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2613, in spawn
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] admin_pass=admin_password)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3003, in _create_image
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] project_id=instance['project_id'])
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 208, in cache
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] *args, **kwargs)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 451, in create_image
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] prepare_template(target=base, max_size=size, *args, **kwargs)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 272, in inner
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] return f(*args, **kwargs)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 198, in fetch_func_sync
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] fetch_func(target=target, *args, **kwargs)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 452, in fetch_image
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] max_size=max_size)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 79, in fetch_to_raw
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] max_size=max_size)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 73, in fetch
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] IMAGE_API.download(context, image_href, dest_path=path)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 178, in download
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] dst_path=dest_path)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 359, in download
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] _reraise_translated_image_exception(image_id)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 357, in download
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] image_chunks = self._client.call(context, 1, 'data', image_id)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 232, in call
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] return getattr(client.images, method)(*args, **kwargs)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 142, in data
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] % urlparse.quote(str(image_id)))
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 253, in get
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] return self._request('GET', url, **kwargs)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 221, in _request
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] raise exc.from_response(resp, resp.content)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] HTTPInternalServerError: HTTPInternalServerError (HTTP 500)
2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]
2014-10-09 17:00:41.932 9242 AUDIT nova.compute.manager [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Terminating instance
2014-10-09 17:00:41.941 9242 WARNING nova.virt.libvirt.driver [-] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] During wait destroy, instance disappeared.
2014-10-09 17:00:42.137 9242 INFO nova.virt.libvirt.driver [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Deleting instance files /var/lib/nova/instances/826a1d2d-a341-45c2-b070-c20f30b4d5aa_del
2014-10-09 17:00:42.138 9242 INFO nova.virt.libvirt.driver [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Deletion of /var/lib/nova/instances/826a1d2d-a341-45c2-b070-c20f30b4d5aa_del complete

Revision history for this message
James Page (james-page) wrote : Re: Nova unable to download glance images juno/rc1
Download full text (3.3 KiB)

Error from glance:

2014-10-09 17:07:03.271 30820 INFO glance.wsgi.server [9a8e44ae-6626-40a3-897f-37113fe0b379 18ce05ec99fd459da3de1d53fc1dfcfe ed33913720264fff8b60db68ef191e50 - - -] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 384, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 663, in __call__
    return self.app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 394, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 206, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 683, in __call__
    request, **action_args)
  File "/usr/lib/python2.7/dist-packages/glance/common/wsgi.py", line 707, in dispatch
    return method(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 494, in show
    image_meta['location'])
  File "/usr/lib/python2.7/dist-packages/glance/api/v1/images.py", line 456, in _get_from_store
    image_data, image_size = src_store.get(loc, context=context)
ValueError: too many values to unpack
2014-10-09 17:07:03.274 30820 INFO glance.wsgi.server [9a8e44ae-6626-40a3...

Read more...

summary: - Nova unable to download glance images
+ Nova unable to download glance images juno/rc1
Revision history for this message
James Page (james-page) wrote :

This actually appears to be a bug in the rbd driver in glance-store

summary: - Nova unable to download glance images juno/rc1
+ Nova unable to download images from glance with rbd backend (juno/rc1)
Changed in python-glance-store (Ubuntu):
importance: Undecided → High
Revision history for this message
James Page (james-page) wrote :

Adding juno-rc-potential tag, as although this is not technically in glance, glance_store is a fairly important dependency; Ubuntu has cherry picked several fixes for ceph rbd support.

tags: added: juno-rc-potential
Changed in nova (Ubuntu):
status: New → Invalid
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package python-glance-store - 0.1.8-1ubuntu2

---------------
python-glance-store (0.1.8-1ubuntu2) utopic; urgency=medium

  * debian/patches/fix-rbd-chunk-read.patch: Cherry pick from upstream
    VCS to correctly use READ_CHUNKSIZE and correct the return of get().
    (LP: #1379434)
 -- Chuck Short <email address hidden> Thu, 09 Oct 2014 17:26:10 -0400

Changed in python-glance-store (Ubuntu):
status: New → Fix Released
Revision history for this message
Zhi Yan Liu (lzy-dev) wrote :

The store.get() api in glance_store-0.1.8 provided correct signature and logic [0], and glance has used that version as a requirement [1].

[0] https://github.com/openstack/glance_store/blob/0.1.8/glance_store/_drivers/rbd.py#L207

[1] https://github.com/openstack/glance/blob/2014.2.rc1/requirements.txt#L60

Changed in glance:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.