OpenStack Compute (nova)

direct downloads plugin schema errors on python 2.6

Bug #1208656 reported by Dan Prince on 2013-08-06

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	Undecided	Dan Prince	OpenStack Compute (nova) 2013.2 "havana"

Bug Description

An hour ago (commit 6f9ed562d5f6df1dc16029dcaffd4d2e6601099d) I started seeing the following stack traces in Nova's compute.log file on RHEL/Centos builds when trying to boot an instance:

https://github.com/openstack/nova/commit/6f9ed562d5f6df1dc16029dcaffd4d2e6601099d

From a high level the issue seems to be related to the fact that the new 'Add plug-in modules for direct downloads of glance locations' makes use of the newer glance v2 API under the covers which seems to be causing some Schema errors on distro's that use python 2.6.

-----

2013-08-06 00:15:32.013 28131 ERROR nova.virt.libvirt.driver [-] [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] During wait destroy, instance disappeared.
2013-08-06 00:15:33.944 28131 ERROR nova.compute.manager [req-b369d3fe-d4e0-49c3-8d09-908ecb6ab32d c311bc3b6c054b1bae5809faf0940fdc 587235ae1f784c73a74843b4d3c68219] [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1090, in _build_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1345, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1341, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1668, in spawn\n admin_pass=admin_password)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1931, in _create_image\n project_id=instance[\'project_id\'])\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 158, in cache\n *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 218, in create_image\n prepare_template(target=base, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 246, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 146, in call_if_not_exists\n fetch_func(target=target, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 613, in fetch_image\n images.fetch_to_raw(context, image_id, target, user_id, project_id)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/images.py", line 196, in fetch_to_raw\n fetch(context, image_href, path_tmp, user_id, project_id)\n', ' File "/usr/lib/python2.6/site-packages/nova/virt/images.py", line 191, in fetch\n image_service.download(context, image_id, image_file)\n', ' File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 316, in download\n locations = self._get_locations(context, image_id)\n', ' File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 293, in _get_locations\n _reraise_translated_image_exception(image_id)\n', ' File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 291, in _get_locations\n image_meta = client.call(context, 2, \'get\', image_id)\n', ' File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 201, in call\n return getattr(client.images, method)(*args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/glanceclient/v2/images.py", line 74, in get\n return self.model(**body)\n', ' File "/usr/lib/python2.6/site-packages/warlock/core.py", line 32, in __init__\n base_class.__init__(self, *args, **kwargs)\n', ' File "/usr/lib/python2.6/site-packages/warlock/model.py", line 33, in __init__\n self.validate(d)\n', ' File "/usr/lib/python2.6/site-packages/warlock/model.py", line 131, in validate\n jsonschema.validate(obj, self.schema)\n', ' File "/usr/lib/python2.6/site-packages/jsonschema.py", line 791, in validate\n cls.check_schema(schema)\n', ' File "/usr/lib/python2.6/site-packages/jsonschema.py", line 163, in check_schema\n error.message, validator=error.validator, path=error.path,\n', "SchemaError: {u'required': [u'url', u'metadata'], u'type': u'object', u'properties': {u'url': {u'type': u'string', u'maxLength': 255}, u'metadata': {u'type': u'object'}}} is not of type {u'$ref': u'#'}, u'array'\n"]
2013-08-06 00:15:38.837 28131 ERROR nova.compute.manager [req-b369d3fe-d4e0-49c3-8d09-908ecb6ab32d c311bc3b6c054b1bae5809faf0940fdc 587235ae1f784c73a74843b4d3c68219] [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] Instance failed to spawn
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] Traceback (most recent call last):
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1341, in _spawn
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] block_device_info)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1668, in spawn
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] admin_pass=admin_password)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1931, in _create_image
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] project_id=instance['project_id'])
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 158, in cache
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] *args, **kwargs)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 218, in create_image
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] prepare_template(target=base, *args, **kwargs)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 246, in inner
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] return f(*args, **kwargs)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 146, in call_if_not_exists
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] fetch_func(target=target, *args, **kwargs)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 613, in fetch_image
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] images.fetch_to_raw(context, image_id, target, user_id, project_id)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/virt/images.py", line 196, in fetch_to_raw
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] fetch(context, image_href, path_tmp, user_id, project_id)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/virt/images.py", line 191, in fetch
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] image_service.download(context, image_id, image_file)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 316, in download
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] locations = self._get_locations(context, image_id)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 293, in _get_locations
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] _reraise_translated_image_exception(image_id)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 291, in _get_locations
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] image_meta = client.call(context, 2, 'get', image_id)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 201, in call
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] return getattr(client.images, method)(*args, **kwargs)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/glanceclient/v2/images.py", line 74, in get
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] return self.model(**body)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/warlock/core.py", line 32, in __init__
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] base_class.__init__(self, *args, **kwargs)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/warlock/model.py", line 33, in __init__
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] self.validate(d)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/warlock/model.py", line 131, in validate
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] jsonschema.validate(obj, self.schema)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/jsonschema.py", line 791, in validate
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] cls.check_schema(schema)
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] File "/usr/lib/python2.6/site-packages/jsonschema.py", line 163, in check_schema
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] error.message, validator=error.validator, path=error.path,
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44] SchemaError: {u'required': [u'url', u'metadata'], u'type': u'object', u'properties': {u'url': {u'type': u'string', u'maxLength': 255}, u'metadata': {u'type': u'object'}}} is not of type {u'$ref': u'#'}, u'array'
2013-08-06 00:15:38.837 28131 TRACE nova.compute.manager [instance: 8a4ed51c-3387-459e-b25f-4ad8dfe99c44]

Dan Prince (dan-prince) on 2013-08-06

affects:

glance → nova

Dan Prince (dan-prince) on 2013-08-06

Changed in nova:
assignee:	nobody → Dan Prince (dan-prince)
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-08-06: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/40426

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2013-08-06: Fix merged to nova (master)

Reviewed: https://review.openstack.org/40426
Committed: http://github.com/openstack/nova/commit/ae1040232dc681d718d940af11108322044a23be
Submitter: Jenkins
Branch: master

commit ae1040232dc681d718d940af11108322044a23be
Author: Dan Prince <email address hidden>
Date: Tue Aug 6 10:15:27 2013 -0400

Avoid extra glance v2 locations call!

    In 7df1c911 we added code to attempt to direct download
    images from glance locations. This feature is meant to be off
    by default however when it is disabled it adds an unnecessary
    glance V2 API call.

    This patch removes the extra get_locations call when
    CONF.allowed_direct_url_schemes is empty (disabled). This effectively
    fixes Nova on Python some 2.6 distros which fail to work with the
    recent Glance V2 schema code.

Fixes LP Bug #1208656

Change-Id: I0989ed4f248cd1126d4703ad6498cfd66685ae31

Changed in nova:
status:	In Progress → Fix Committed

Revision history for this message

Matt Riedemann (mriedem) wrote on 2013-08-10:

Might want to take a look at this bug: https://bugs.launchpad.net/glance/+bug/1202391

Thierry Carrez (ttx) on 2013-09-05

Changed in nova:
milestone:	none → havana-3
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2013-10-17

Changed in nova:
milestone:	havana-3 → 2013.2

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.