Unicode encoding error exists in extended Nova API, when the data contain unicode

[David Jia]

We have developed an extended Nova API, the API query disks at first, then add a disk to an instance.
After querying, if disk has non-english disk name, unicode will be converted to str in nova/api/openstack/wsgi.py line 451
"node = doc.createTextNode(str(data))", then unicode encoding error exists.

[David Jia (jiataotj)]

Here is my fixing: https://review.openstack.org/#/c/73953/

[Mark McLoughlin (markmc)]

I find it hard to believe there isn't an example of this problem with an existing part of Nova's API rather than just an out-of-tree extension

Could you try to reproduce this issue with an in-tree API?

[David Jia (jiataotj)]

Mark,

Thanks for your comments, I can understand your concern. Honestly, I didn't encounter the same issue against nova in-tree APIs, I found the issue in one IBM extended API. However, I'll try to reproduce the issue agianst in-tree APIs to prove it, hope I could find it soon

[Mark McLoughlin (markmc)]

Some attempts to reproduce the issue with instance passwords: https://gist.github.com/markmc/9484531

ServerXMLSerializationTest.test_create() shows the serialization bug, however ServersSampleXmlTest.test_servers_post() shows there are bigger problems than just the XML serialization issue

[David Jia (jiataotj)]

Mark,

I tried test cases https://gist.github.com/markmc/9484531 in my dev env, yes, there are some failure exist.

I'd like to clarify that we need classify those failure to catch the real bug, there are two type failure:

The first type exists against nova.tests.api.openstack.compute.test_servers.ServerXMLSerializationTest.test_create, it's not an openstack source code issue, according to trace log, it's cause by using str in Tempest test cases codes.

Traceback (most recent call last):
  File "nova/tests/api/openstack/compute/test_servers.py", line 4322, in test_create
    self.ADMINS_ATTRIBUTES)
  File "nova/tests/api/openstack/compute/test_servers.py", line 4288, in _validate_required_attributes
    self.assertEqual(root.get(key), str(server_dict[key]))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

The second type exists against sever get/list/details/post APIs due to apply unicode instance password in api sample server post req json and xml. I agree that it's a bigger problem, according to trace log, the exception are thrown from other python packages.

It worth to do more investigation for the second type issues, but I must say that it's out the bug's, which I'm working on, scope, we'd better open another bug for it. Please let me know your opinion.

Furthermore, is unicode instance adminPass really supported in real user scenarios? I haven't received this kind issue feedback from users

[Sean Dague (sdague)]

This is only an issue if you hit the XML API, right?

[Christopher Yeoh (cyeoh-0)]

Mark - the failures with those modifications are purely on the test side - the XML serialization of adminPass avoids the usual XMLDictSerializer and does not attempt to do a str() on the data as it knows it doesn't need to do any conversion.

It does look like there is a general problem with doing a str on something which is potentially unicode in the wsgi code though

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/124678

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/124684

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/124678
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=53fe8696314fb73ca9943fce998d96fa6d0414b4
Submitter: Jenkins
Branch: master

commit 53fe8696314fb73ca9943fce998d96fa6d0414b4
Author: Chris Yeoh <email address hidden>
Date: Mon Sep 29 16:01:04 2014 +0930

    Fix XML UnicodeEncode serialization error

    The generic Nova XMLSerializer code will currently attempt
    to cast to str the value for all leaf nodes. This patch
    ensures that no attempt is made to convert unicode which
    can cause a UnicodeEncode error. We don't need to convert
    unicode for XML text and regardless we encode to UTF-8 at
    a later point.

    Change-Id: I8135d2b9a67db62b0eafdd301b7fdb67a5dd72cc
    Closes-Bug: #1279172

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/125374

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/125374
Reason: oops, this needs to wait for stable/juno.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/124684
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6c5968a2218024e7995d4cf3135ddc8255b035ba
Submitter: Jenkins
Branch: master

commit 6c5968a2218024e7995d4cf3135ddc8255b035ba
Author: Chris Yeoh <email address hidden>
Date: Mon Sep 29 18:21:59 2014 +0930

    Extend XML unicode test coverage

    Extends the coverage of the servers XML serialization test to
    cover unicode values. Fixes common function to be handle fields
    being unicode. No actual bug fix, but this is test coverage which
    we always should have had, and an investigation into another unicode
    bug revealed the missing coverage and problem with the tests
    not handling unicode values.

    Change-Id: I6255ad1cf9a3429c0d87cb6709c11c1b65df9ba5
    Related-Bug: 1279172

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/129307

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/125374
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7cdb643cb3100dd100de471b5026b13ab94c21f7
Submitter: Jenkins
Branch: stable/icehouse

commit 7cdb643cb3100dd100de471b5026b13ab94c21f7
Author: Chris Yeoh <email address hidden>
Date: Mon Sep 29 16:01:04 2014 +0930

    Fix XML UnicodeEncode serialization error

    The generic Nova XMLSerializer code will currently attempt
    to cast to str the value for all leaf nodes. This patch
    ensures that no attempt is made to convert unicode which
    can cause a UnicodeEncode error. We don't need to convert
    unicode for XML text and regardless we encode to UTF-8 at
    a later point.

    Change-Id: I8135d2b9a67db62b0eafdd301b7fdb67a5dd72cc
    Closes-Bug: #1279172
    (cherry picked from commit 53fe8696314fb73ca9943fce998d96fa6d0414b4)
    (cherry picked from commit 3136cfc11f034cf0b17888f7348e80d38d563c2e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/129307
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3136cfc11f034cf0b17888f7348e80d38d563c2e
Submitter: Jenkins
Branch: stable/juno

commit 3136cfc11f034cf0b17888f7348e80d38d563c2e
Author: Chris Yeoh <email address hidden>
Date: Mon Sep 29 16:01:04 2014 +0930

    Fix XML UnicodeEncode serialization error

    The generic Nova XMLSerializer code will currently attempt
    to cast to str the value for all leaf nodes. This patch
    ensures that no attempt is made to convert unicode which
    can cause a UnicodeEncode error. We don't need to convert
    unicode for XML text and regardless we encode to UTF-8 at
    a later point.

    Change-Id: I8135d2b9a67db62b0eafdd301b7fdb67a5dd72cc
    Closes-Bug: #1279172
    (cherry picked from commit 53fe8696314fb73ca9943fce998d96fa6d0414b4)