flavor extra spec api fails with XML content type if key contains a colon

[Andrea Frittoli]

The flavor extra spec API extension (os-extra_specs) fails with "HTTP 500" when content-type application/xml is requested if the extra spec key contains a colon.

For example:

curl [endpoint]/flavors/[ID]/os-extra_specs -H "Accept: application/json" -H "X-Auth-Token: $TOKEN"
{"extra_specs": {"foo:bar": "999"}}

curl -i [endpoint]/flavors/[ID]/os-extra_specs -H "Accept: application/xml" -H "X-Auth-Token: $TOKEN"
{"extra_specs": {"foo:bar": "999"}}
HTTP/1.1 500 Internal Server Error

The stack trace shows that the XML parser tries to interpret the ":" in key as if it would be a XML namespace, which fails, as the namespace is not valid:

2013-08-19 13:08:14.374 27521 DEBUG nova.api.openstack.wsgi [req-afe0c3c8-e7d6-48c5-84f1-782260850e6b redacted redacted] Calling method <bound method FlavorExtraSpecsController.index of <nova.api.openstack.compute.contrib.flavorextraspecs.FlavorExtraSpecsController object at 0x2c01b90>> _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:927
2013-08-19 13:08:14.377 27521 ERROR nova.api.openstack [req-afe0c3c8-e7d6-48c5-84f1-782260850e6b redacted redacted] Caught error: Invalid tag name u'foo:bar'
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack Traceback (most recent call last):
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 110, in __call__
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack return req.get_response(self.application)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack application, catch_exc_info=False)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack return resp(environ, start_response)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/hp/middleware/cs_auth_token.py", line 160, in __call__
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack return super(CsAuthProtocol, self).__call__(env, start_response)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 461, in __call__
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack return self.app(env, start_response)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack return resp(environ, start_response)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack return resp(environ, start_response)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack response = self.app(environ, start_response)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack return resp(environ, start_response)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 903, in __call__
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack content_type, body, accept)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 992, in _process_stack
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack self.default_serializers)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 603, in serialize
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack response.body = serializer.serialize(self.obj)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/xmlutil.py", line 590, in serialize
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack elem = self.make_tree(obj)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/xmlutil.py", line 618, in make_tree
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack return self._serialize(None, obj, siblings, nsmap)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/xmlutil.py", line 573, in _serialize
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack self._serialize(elem, datum, nieces)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/xmlutil.py", line 554, in _serialize
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack elems = siblings[0].render(parent, obj, siblings[1:], nsmap)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/xmlutil.py", line 426, in render
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack elems.append((self._render(parent, datum, patches, nsmap), datum))
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/xmlutil.py", line 369, in _render
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack elem = etree.Element(tagname, nsmap=nsmap)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "lxml.etree.pyx", line 2568, in lxml.etree.Element (src/lxml/lxml.etree.c:52878)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "apihelpers.pxi", line 126, in lxml.etree._makeElement (src/lxml/lxml.etree.c:11497)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack File "apihelpers.pxi", line 1542, in lxml.etree._tagValidOrRaise (src/lxml/lxml.etree.c:23956)
2013-08-19 13:08:14.377 27521 TRACE nova.api.openstack ValueError: Invalid tag name u'foo:bar'

The issue affects tempest as well, as the test for this does not setup the extra spec using a colon in it, which is what the ComputeCapabilitiesFilter in the scheduler expects.

[Jake Liu (jake-liu)]

 Hi Andrea,

I cannot reproduce this issue with your steps.

REQ: curl -i http://192.168.0.106:8774/v2/33d982fa68a44289af94978544eb790d/flavors/1/os-extra_specs -X POST -H "X-Auth-Project-Id: admin" -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: 14126dcd29474f69b6283a563f5d0696" -d '{"extra_specs": {"foo:bar": "100"}}'

RESP: [200] CaseInsensitiveDict({'date': 'Mon, 19 Aug 2013 14:26:35 GMT', 'content-length': '35', 'content-type': 'application/json', 'x-compute-request-id': 'req-e4837622-40a9-4330-9720-f05618448ded'})
RESP BODY: {"extra_specs": {"foo:bar": "100"}}

root@jake1:~# nova flavor-show 1
+----------------------------+----------------------+
| Property | Value |
+----------------------------+----------------------+
| name | m1.tiny |
| ram | 512 |
| OS-FLV-DISABLED:disabled | False |
| vcpus | 1 |
| extra_specs | {u'foo:bar': u'100'} |
| swap | |
| os-flavor-access:is_public | True |
| rxtx_factor | 1.0 |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 1 |
| id | 1 |
+----------------------------+----------------------+

Which version are you using? I was using devstack from latest code. Thanks.

[Jake Liu (jake-liu)]

Oops, my fault, you are using XML while I was using Json.

[Andrea Frittoli (andrea-frittoli)]

Exactly, the issue is specific to XML only.

[Christopher Yeoh (cyeoh-0)]

capabilities has a similar problem. Should try to make the fix consistent (which probably means breaking backward compatibility, perhaps for the V3 API only) https://bugs.launchpad.net/nova/+bug/1205983

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/46718

[Matt Riedemann (mriedem)]

I don't think there is a fix to make in tempest for this so once the problem is resolved on the nova side it should fix any test issues in tempest, right?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/46718
Committed: http://github.com/openstack/nova/commit/80b11279ba9853a9ab743ec6987c3229fff0b5e4
Submitter: Jenkins
Branch: master

commit 80b11279ba9853a9ab743ec6987c3229fff0b5e4
Author: guohliu <email address hidden>
Date: Mon Sep 16 19:21:26 2013 +0800

    Fixes Invalid tag name error when using k:v tagname

    This patch fixes the invalid tag name error when k:v type
    tagname appears in response of api during xml serialization,
    the tagname which contains colon couldn't be supported by
    default since it is reserved by xml namespace, this patch
    solve it by adding the k of k:v tagname into namespace.

    Fixes bug: #1213927

    Change-Id: Ib20efb378425cbe5deef45fbac561955cbe0f7bc

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/53528

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/53528
Committed: http://github.com/openstack/nova/commit/51cfd40f97e509d78c3981689ac2d81ed3ba6457
Submitter: Jenkins
Branch: stable/havana

commit 51cfd40f97e509d78c3981689ac2d81ed3ba6457
Author: guohliu <email address hidden>
Date: Mon Sep 16 19:21:26 2013 +0800

    Fixes Invalid tag name error when using k:v tagname

    This patch fixes the invalid tag name error when k:v type
    tagname appears in response of api during xml serialization,
    the tagname which contains colon couldn't be supported by
    default since it is reserved by xml namespace, this patch
    solve it by adding the k of k:v tagname into namespace.

    Fixes bug: #1213927

    Change-Id: Ib20efb378425cbe5deef45fbac561955cbe0f7bc
    (cherry picked from commit 80b11279ba9853a9ab743ec6987c3229fff0b5e4)