UnicodeEncodeError when Running CLI keystone tenant-delete action with non-English characters

[Chen Xiao]

keystone tenant-delete action with non-English characters occurs UnicodeEncodeError in log file, but the tenant has been delete in fact.
input below command:
[[root@osee22-compute02 ˜]# keystone tenant-delete testソ十豹

[root@osee22-compute02 ˜]# keystone tenant-list
+----------------------------------+---------+---------+
| id | name | enabled |
+----------------------------------+---------+---------+
| e7308b169a784693b39c60e152c7d01a | Public | True |
| 9b78773f34264a428073412ef7d5485e | service | True |
+----------------------------------+---------+---------+

log:
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi Traceback (most recent call last):
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 238, in __call__
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi result = method(context, **params)
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/identity/controllers.py", line 124, in delete_project
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi self.assignment_api.delete_project(tenant_id)
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/notifications.py", line 44, in wrapper
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi result = f(*args, **kwargs)
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/assignment/core.py", line 93, in delete_project
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi project['domain_id'])
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/dogpile/cache/region.py", line 939, in invalidate
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi key = key_generator(*arg, **kw)
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi File "/usr/lib/python2.6/site-packages/dogpile/cache/util.py", line 86, in generate_key
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi return namespace + "|" + " ".join(map(to_str, args))
2013-09-25 21:06:50.280 3622 TRACE keystone.common.wsgi UnicodeEncodeError: 'ascii' codec can't encode characters in position 4-6: ordinal not in range(128)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/50938

[Morgan Fainberg (mdrnstm)]

I've got a lead on a bit more of the core of the issue on this bug (vs. just a simple encode at the method call). This appears to be an issue in the cache key-generator when handling unicode strings, specifically in the stringificaiton (default) mechanism. I am working on a better stringification method for cache-key generation that will be unicode friendly. Once this new mechanism is implemented, use of the to_str argument on the @cache.on_arguments decorator will resolve the issue.

[Chen Xiao (chenxiao)]

Hi, Morgan Fainberg,
     Your method to solve this bug is better, you are right, wish your solution, and I want to know whether you change the dogpile.cache code for solve this problem

[Dolph Mathews (dolph)]

The cache key generator is configurable in dogpile.cache, so we can provide our own implementation from keystone's side (even if it's just a wrapper for dogpile's implementation).

[Morgan Fainberg (mdrnstm)]

This is actually even easier than providing our own cache-key generator, I am working on a replacement for the stringification method that covers this error. The stringify method is passed in to the decorator itself. I have isolated the actual issue and I am working on implementing a clean mechanism to handle this without causing other encode/decode headaches. Even if we provided our own complete cache-key generator, this issue would likely still need to be addressed in a similar manner.

[Dolph Mathews (dolph)]

Raising the priority on this issue, as merely disabling caching is not a sufficient workaround (cache keys are still generated, regardless). Additionally, this will affect any cached driver call that expects unicode arguments.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/51504

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/51511

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/51504
Committed: http://github.com/openstack/keystone/commit/a4ee3ae9673d49de35409f8f5512ea4b18e4fcaa
Submitter: Jenkins
Branch: master

commit a4ee3ae9673d49de35409f8f5512ea4b18e4fcaa
Author: Morgan Fainberg <email address hidden>
Date: Sun Oct 13 18:34:24 2013 -0700

    Handle unicode at the caching layer more elegantly

    This patchset resolves an issue where in some cases unicode would
    cause the cache key generator to raise a UnicodeEncodeError due to
    the name/value being outside of the standard ascii character set.
    Included is a fix to the cache backend debug code to utilize repr
    for passing the keys/values to the logger.

    Tests in test_backend provided by chenxiao <email address hidden>

    Closes-bug: 1237892
    Change-Id: I4df1c9eb6b4a1367defdbb6bcbcd1f1f992348e5

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/51511
Committed: http://github.com/openstack/keystone/commit/a0ae37eb3ea96a394f10d3f38e8b454d2d7a98e0
Submitter: Jenkins
Branch: milestone-proposed

commit a0ae37eb3ea96a394f10d3f38e8b454d2d7a98e0
Author: Morgan Fainberg <email address hidden>
Date: Sun Oct 13 18:34:24 2013 -0700

    Handle unicode at the caching layer more elegantly

    This patchset resolves an issue where in some cases unicode would
    cause the cache key generator to raise a UnicodeEncodeError due to
    the name/value being outside of the standard ascii character set.
    Included is a fix to the cache backend debug code to utilize repr
    for passing the keys/values to the logger.

    Tests in test_backend provided by chenxiao <email address hidden>

    Closes-bug: 1237892
    Change-Id: Ic99503987851128cd41d83ad1ea50dc4a132fbd3