Mixing str and unicode leads to a UnicodeDecodeError

[Cyril Roelandt]

In keystoneclient/middleware/s3token.py, the following code:

class S3Token(object):
    ...
    def __call__(self, environ, start_response):
        ...
        environ['PATH_INFO'] = environ['PATH_INFO'].replace(account,
                                                            new_tenant_name)

sometimes leads to the following error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 49: ordinal not in range(128).

This happens because "environ['PATH_INFO']" and "account" have type "str", while "new_tenant_name" is unicode. When dealing with unicode characters, we end up doing something like:

>>> x = "fooébar"
>>> x.replace('o', u'u')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 3: ordinal not in range(128)

The "new_tenant_name" is Unicode because keystone returns the tenant id as a Unicode string. We should encode it before calling str.replace().

You may find a reproducible test case involving Swift and Swift3 at https://bugs.launchpad.net/swift3/+bug/1333600 .

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/161762

[OpenStack Infra (hudson-openstack)]

Change abandoned by Cyril Roelandt (<email address hidden>) on branch: master
Review: https://review.openstack.org/161762
Reason: The relevant code is now part of keystonemiddleware.

[Charles Hsu (charles0126)]

Hi Cyril,

I opened a bug in keystonemiddleware.
https://bugs.launchpad.net/keystonemiddleware/+bug/1465525

And do you want to summit a patch to keystonemiddleware? Or I can help this?

[Cyril Roelandt (cyril-roelandt)]

There is already a patch https://review.openstack.org/#/c/179777/ :)

[Charles Hsu (charles0126)]

@Cyril, That's awesome! Thanks!

[Steve Martinelli (stevemar)]

looks like this was fixed in keystonemiddleware, marking as fix-released