Returned strings should not be forced into unicode

[Sean McGinnis]

[Impact]

In Kilo, with ldap integrated, user_id can have non uuid format ( e.g. Gāo )
Then after upgrading kilo to mitaka ( in trusty env )

Gāo user can't create cinder volume with unicode error.

Affected Xenial, UCA Mitaka ( this is modified after i re-testing all )

Please refer to error code like below [1]

[Test Case]

1. deploy openstack env
2. source novarc ( as admin )
3. keystone user-create --name test
4. openstack user set --password-prompt test
4. log into keystone node and install mysql-client
5. connect to mysql with keystone user
6. do below query
- set foreign_key_checks = 0;
- update local_user set user_id='Gāo' where user_id='UUID';
- update user set id='Gāo' where id='UUID';
- set foreign_key_checks = 1;
7. keystone user-role-add --user test --tenant admin --role Admin
8. source novarc ( as test )
9. cinder create --name test 1

got unicode error on cinder-api node logs

[Regression Potential]
This patch make code handles unicode non uuid form user id. Basically this has little risk. but cinder-api service need to be restarted.

[Others]

[1] ###########

2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/taskflow/types/failure.py", line 343, in reraise
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault six.reraise(*self._exc_info)
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault result = task.execute(**arguments)
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/api/create_volume.py", line 527, in execute
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault volume = objects.Volume(context=context, **volume_properties)
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/cinder/objects/volume.py", line 180, in __init__
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault super(Volume, self).__init__(*args, **kwargs)
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 304, in __init__
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault setattr(self, key, kwargs[key])
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 72, in setter
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault field_value = field.coerce(self, name, value)
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 192, in coerce
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault return self._type.coerce(obj, attr, value)
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 325, in coerce
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault "for further details" % value, FutureWarning)
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/logging/__init__.py", line 1709, in _showwarning
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault s = warnings.formatwarning(message, category, filename, lineno, line)
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault File "/usr/lib/python2.7/warnings.py", line 38, in formatwarning
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault s = "%s:%s: %s: %s\n" % (filename, lineno, category.__name__, message)
2018-03-14 10:58:56.932 2198 ERROR cinder.api.middleware.fault UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 9: ordinal not in range(128)

Original Description
--------------------
Change https://review.openstack.org/#/c/557778/ changed the return value for coerce() to explicitly force the value to be unicode. This causes problems with Python 2 runtimes, where the expected value of a string is no longer a string but now unicode (see: https://github.com/openstack/cinder/blob/b89e73a2ac315927d2eee42129597b14e87ba685/cinder/tests/unit/volume/drivers/dell_emc/vmax/test_vmax.py#L3412)

The correct fix should be to use either six.text_type to convert it, or format it into a string as is being done, but without forcing it to be a unicode string. "%s" will end up being the correct type depending on whether it is running under py2 or py3.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/560639

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/560639
Committed: https://git.openstack.org/cgit/openstack/oslo.versionedobjects/commit/?id=b719764ba85d5e689d8276ad40c82985c84f03f9
Submitter: Zuul
Branch: master

commit b719764ba85d5e689d8276ad40c82985c84f03f9
Author: Sean McGinnis <email address hidden>
Date: Wed Apr 11 21:02:53 2018 +0000

    Don't force unicode strings for UUID coercion

    Change Ic6b6308fb1960ec40407e6efde30137b64543e72 attempts to
    fix difference between Python 2 and Python 3 values by switching
    from using str() to formatting into a unicode string (u"%s").

    This is equivalent to changing str() to unicode(), but that is
    not correct for expected default string types for Python 2.
    This requires either using six.text_type(), or just formatting
    into a string, without forcing unicode ("%s"), to be correct on
    either runtime.

    Change-Id: I178f14cdc670d7a696778891e587ef75de208fc2
    Closes-bug: #1763179

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/561674

[Corey Bryant (corey.bryant)]

I've uploaded this for bionic (queens), artful (pike), and ocata. Bionic and Artful are awaiting SRU team review.

[Corey Bryant (corey.bryant)]

Hello Sean, or anyone else affected,

Accepted python-oslo.versionedobjects into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Sean McGinnis (sean-mcginnis)]

Great - thanks Corey! I will try to check those out.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.versionedobjects 1.33.1 release.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-oslo.versionedobjects - 1.31.2-0ubuntu3

---------------
python-oslo.versionedobjects (1.31.2-0ubuntu3) bionic; urgency=medium

  * d/p/dont-force-unicode-strings-for-UUID-coercion.patch:
    This reverts and changes the fix provided by
    d/p/Fixing-uuid-coerce-function-for-unicode-non-uuid-form.patch
    (LP: #1763179).

 -- Corey Bryant <email address hidden> Tue, 17 Apr 2018 08:33:51 -0400

[Corey Bryant (corey.bryant)]

Hello Sean, or anyone else affected,

Accepted python-oslo.versionedobjects into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-oslo.versionedobjects has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-oslo.versionedobjects - 1.31.2-0ubuntu3~cloud0
---------------

 python-oslo.versionedobjects (1.31.2-0ubuntu3~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-oslo.versionedobjects (1.31.2-0ubuntu3) bionic; urgency=medium
 .
   * d/p/dont-force-unicode-strings-for-UUID-coercion.patch:
     This reverts and changes the fix provided by
     d/p/Fixing-uuid-coerce-function-for-unicode-non-uuid-form.patch
     (LP: #1763179).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561674
Committed: https://git.openstack.org/cgit/openstack/oslo.versionedobjects/commit/?id=b607dad267aff8ce87f956e9f2586b9e44a58f44
Submitter: Zuul
Branch: stable/queens

commit b607dad267aff8ce87f956e9f2586b9e44a58f44
Author: Sean McGinnis <email address hidden>
Date: Wed Apr 11 21:02:53 2018 +0000

    Don't force unicode strings for UUID coercion

    Change Ic6b6308fb1960ec40407e6efde30137b64543e72 attempts to
    fix difference between Python 2 and Python 3 values by switching
    from using str() to formatting into a unicode string (u"%s").

    This is equivalent to changing str() to unicode(), but that is
    not correct for expected default string types for Python 2.
    This requires either using six.text_type(), or just formatting
    into a string, without forcing unicode ("%s"), to be correct on
    either runtime.

    Change-Id: I178f14cdc670d7a696778891e587ef75de208fc2
    Closes-bug: #1763179
    (cherry picked from commit b719764ba85d5e689d8276ad40c82985c84f03f9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/561928
Committed: https://git.openstack.org/cgit/openstack/oslo.versionedobjects/commit/?id=c4e7defc88a79952c93b151310688f6cf39d3e6d
Submitter: Zuul
Branch: stable/pike

commit c4e7defc88a79952c93b151310688f6cf39d3e6d
Author: Sean McGinnis <email address hidden>
Date: Wed Apr 11 21:02:53 2018 +0000

    Don't force unicode strings for UUID coercion

    Change Ic6b6308fb1960ec40407e6efde30137b64543e72 attempts to
    fix difference between Python 2 and Python 3 values by switching
    from using str() to formatting into a unicode string (u"%s").

    This is equivalent to changing str() to unicode(), but that is
    not correct for expected default string types for Python 2.
    This requires either using six.text_type(), or just formatting
    into a string, without forcing unicode ("%s"), to be correct on
    either runtime.

    Change-Id: I178f14cdc670d7a696778891e587ef75de208fc2
    Closes-bug: #1763179
    (cherry picked from commit b719764ba85d5e689d8276ad40c82985c84f03f9)

[Łukasz Zemczak (sil2100)]

Hello Sean, or anyone else affected,

Accepted python-oslo.versionedobjects into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-oslo.versionedobjects/1.26.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Sean, or anyone else affected,

Accepted python-oslo.versionedobjects into pike-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:pike-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-pike-needed to verification-pike-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-pike-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.versionedobjects 1.31.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.versionedobjects 1.26.2 release.

[Seyeong Kim (seyeongkim)]

for artful

I updated it on #1761629

verified.

[Seyeong Kim (seyeongkim)]

for pike

tested the same on #1761629

Thanks

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-oslo.versionedobjects - 1.26.0-0ubuntu2

---------------
python-oslo.versionedobjects (1.26.0-0ubuntu2) artful; urgency=medium

  [ Corey Bryant ]
  * d/p/dont-force-unicode-strings-for-UUID-coercion.patch:
    This reverts and changes the fix provided by
    d/p/Fixing-uuid-coerce-function-for-unicode-non-uuid-form.patch
    (LP: #1763179).

  [ Seyeong Kim ]
  * Fixing UUID coerce function for unicode non uuid form id (LP: #1760918)
    - d/p/Fixing-uuid-coerce-function-for-unicode-non-uuid-form.patch

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/pike branch.

 -- Corey Bryant <email address hidden> Tue, 17 Apr 2018 08:47:39 -0400

[Corey Bryant (corey.bryant)]

Regression testing has passed successfully for ocata-proposed:

======
Totals
======
Ran: 102 tests in 1806.1829 sec.
 - Passed: 94
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 1104.3589 sec.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-oslo.versionedobjects has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-oslo.versionedobjects - 1.21.0-0ubuntu1~cloud2
---------------

 python-oslo.versionedobjects (1.21.0-0ubuntu1~cloud2) xenial-ocata; urgency=medium
 .
   * d/p/dont-force-unicode-strings-for-UUID-coercion.patch:
     This reverts and changes the fix provided by
     d/p/Fixing-uuid-coerce-function-for-unicode-non-uuid-form.patch
     (LP: #1763179).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-oslo.versionedobjects has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-oslo.versionedobjects - 1.26.0-0ubuntu2~cloud0
---------------

 python-oslo.versionedobjects (1.26.0-0ubuntu2~cloud0) xenial-pike; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-oslo.versionedobjects (1.26.0-0ubuntu2) artful; urgency=medium
 .
   [ Corey Bryant ]
   * d/p/dont-force-unicode-strings-for-UUID-coercion.patch:
     This reverts and changes the fix provided by
     d/p/Fixing-uuid-coerce-function-for-unicode-non-uuid-form.patch
     (LP: #1763179).
 .
   [ Seyeong Kim ]
   * Fixing UUID coerce function for unicode non uuid form id (LP: #1760918)
     - d/p/Fixing-uuid-coerce-function-for-unicode-non-uuid-form.patch
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/pike branch.

[Corey Bryant (corey.bryant)]

I've uploaded a new version of the xenial package to the xenial unapproved queue [1] where it is awaiting review by the Ubuntu SRU team.

[1] https://launchpad.net/ubuntu/xenial/+queue?queue_state=1&queue_text=

[Robie Basak (racb)]

Hello Sean, or anyone else affected,

Accepted python-oslo.versionedobjects into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-oslo.versionedobjects/1.8.0-1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (seyeongkim)]

hello

I verified this is working for xenial.

before patch, return value of command wa 500,
after patch, it successfully created.

thanks

dpkg -l | grep cinder
ii cinder-api 2:8.1.1-0ubuntu3 all Cinder storage service - API server
ii cinder-common 2:8.1.1-0ubuntu3 all Cinder storage service - common files
ii cinder-scheduler 2:8.1.1-0ubuntu3 all Cinder storage service - Scheduler server
ii cinder-volume 2:8.1.1-0ubuntu3 all Cinder storage service - Volume server
ii python-cinder 2:8.1.1-0ubuntu3 all Cinder Python libraries

[Łukasz Zemczak (sil2100)]

Could you please mention which version of python-oslo.versionedobjects was used for verification?

[Seyeong Kim (seyeongkim)]

Sorry about that sil2100,

I re-verified this. and attaching oslo.versionedobjects version

dpkg -l | grep oslo.versionedobjects
ii python-oslo.versionedobjects 1.8.0-1ubuntu1 all deals with DB schema versions and code expectations - Python 2.x

[Corey Bryant (corey.bryant)]

Hello Sean, or anyone else affected,

Accepted python-oslo.versionedobjects into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-oslo.versionedobjects - 1.8.0-1ubuntu1

---------------
python-oslo.versionedobjects (1.8.0-1ubuntu1) xenial; urgency=medium

  * d/p/fixing-UUID-coerce-function-for-unicode-non-uuid-for.patch:
    Cherry-pick upstream patches to enable use of unicode user ID in Python2.7
    (LP: #1763179).

 -- Corey Bryant <email address hidden> Mon, 04 Jun 2018 08:29:48 -0400

[Seyeong Kim (seyeongkim)]

Hello,

I verified this with test case's description.

and pkg version is below

dpkg -l | grep python-oslo.versionedobjects
ii python-oslo.versionedobjects 1.8.0-1ubuntu1~cloud0 all deals with DB schema versions and code expectations - Python 2.x

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-oslo.versionedobjects has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-oslo.versionedobjects - 1.8.0-1ubuntu1~cloud0
---------------

 python-oslo.versionedobjects (1.8.0-1ubuntu1~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-oslo.versionedobjects (1.8.0-1ubuntu1) xenial; urgency=medium
 .
   * d/p/fixing-UUID-coerce-function-for-unicode-non-uuid-for.patch:
     Cherry-pick upstream patches to enable use of unicode user ID in Python2.7
     (LP: #1763179).