Comment 0 for bug 1198566

We have a multi-node Openstack cluster running Folsom 2012.2.3 on Ubuntu 12.04. A few days ago we added a new compute node, and found that we were unable to launch new instances from a pre-existing Ubuntu Server 12.04 LTS image stored in glance. Each spawning attempt would deposit a glance client exception (shown below) in the compute node's nova-compute.log.

After quite a lot of investigation, I found that the original --location URL (used during "glance image-create") of the Ubuntu Server image had gone out of date. This was evidently causing a BadStoreUri exception on the glance server during instance spawning, resulting in a 500 error being returned to our new compute node's glance client. I was able to resolve the problem by re-importing an Ubuntu Server 12.04 LTS image from a working mirror.

Improved error logged would have saved us hours of troubleshooting.

2013-06-27 21:19:24 ERROR nova.compute.manager [req-f8d7c23a-e8ad-4059-bea4-4fc588a6afe0 9d8968d3f17f4697aaf923c14651ce7b e5fb3c6db0db4e9c86d0301005e2e5bb] [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] Instance failed to spawn
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] Traceback (most recent call last):
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 756, in _spawn
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] block_device_info)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] temp_level, payload)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] self.gen.next()
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] return f(*args, **kw)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1099, in spawn
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] admin_pass=admin_password)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1365, in _create_image
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] project_id=instance['project_id'])
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 131, in cache
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] *args, **kwargs)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 178, in create_image
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] prepare_template(target=base, *args, **kwargs)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 795, in inner
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] retval = f(*args, **kwargs)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 122, in call_if_not_exists
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] fetch_func(target=target, *args, **kwargs)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 440, in fetch_image
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] images.fetch_to_raw(context, image_id, target, user_id, project_id)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in fetch_to_raw
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] fetch(context, image_href, path_tmp, user_id, project_id)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 82, in fetch
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] image_service.download(context, image_id, image_file)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 222, in download
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] _reraise_translated_image_exception(image_id)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 220, in download
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] image_chunks = self._client.call(context, 1, 'data', image_id)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 138, in call
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] return getattr(client.images, method)(*args, **kwargs)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 108, in data
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] resp, body = self.api.raw_request('GET', '/v1/images/%s' % image_id)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 191, in raw_request
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] return self._http_request(url, method, **kwargs)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 158, in _http_request
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] raise exc.from_response(resp)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] HTTPInternalServerError: HTTPInternalServerError (HTTP 500)
2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]