py3: inconsistent encoding of token fields

[Drew Freiberger]

When using an LDAP domain user on a bionic-rocky cloud within horizon, we are unable to see the projects listed in the project selection drop-down, and are unable to query resources from any projects to which we are assigned the role Member.

It appears that the following log entries in keystone may be helpful to troubleshooting this issue:

(keystone.middleware.auth): 2019-06-10 19:47:02,700 DEBUG RBAC: auth_context: {'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_id': None, 'domain_name': None, 'group_ids': [], 'token': <TokenModel (audit_id=8_4AHHWtSQ2JjTiwU7Kh0g, audit_chain_id=['8_4AHHWtSQ2JjTiwU7Kh0g']) at 0x7fed2c7909b0>, 'user_id': b'd4fb94cfa3ce0f7829d76fe44697488e7765d88e29f5a896f57d43caadb0fad4', 'user_domain_id': '997b3e91271140feb1635eefba7c65a1', 'system_scope': None, 'project_id': None, 'project_domain_id': None, 'roles': [], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []}
(keystone.server.flask.application): 2019-06-10 19:47:02,700 DEBUG Dispatching request to legacy mapper: /v3/users
(keystone.server.flask.application): 2019-06-10 19:47:02,700 DEBUG SCRIPT_NAME: `/v3`, PATH_INFO: `/users/d4fb94cfa3ce0f7829d76fe44697488e7765d88e29f5a896f57d43caadb0fad4/projects`
(routes.middleware): 2019-06-10 19:47:02,700 DEBUG Matched GET /users/d4fb94cfa3ce0f7829d76fe44697488e7765d88e29f5a896f57d43caadb0fad4/projects
(routes.middleware): 2019-06-10 19:47:02,700 DEBUG Route path: '/users/{user_id}/projects', defaults: {'action': 'list_user_projects', 'controller': <keystone.assignment.controllers.ProjectAssignmentV3 object at 0x7fed2ec52ef0>}
(routes.middleware): 2019-06-10 19:47:02,700 DEBUG Match dict: {'user_id': 'd4fb94cfa3ce0f7829d76fe44697488e7765d88e29f5a896f57d43caadb0fad4', 'action': 'list_user_projects', 'controller': <keystone.assignment.controllers.ProjectAssignmentV3 object at 0x7fed2ec52ef0>}
(keystone.common.wsgi): 2019-06-10 19:47:02,700 INFO GET https://keystone.mysite:5000/v3/users/d4fb94cfa3ce0f7829d76fe44697488e7765d88e29f5a896f57d43caadb0fad4/projects
(keystone.common.controller): 2019-06-10 19:47:02,700 DEBUG RBAC: Adding query filter params ()
(keystone.common.authorization): 2019-06-10 19:47:02,700 DEBUG RBAC: Authorizing identity:list_user_projects(user_id=d4fb94cfa3ce0f7829d76fe44697488e7765d88e29f5a896f57d43caadb0fad4)
(keystone.policy.backends.rules): 2019-06-10 19:47:02,701 DEBUG enforce identity:list_user_projects: {'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_id': None, 'domain_name': None, 'group_ids': [], 'token': <TokenModel (audit_id=8_4AHHWtSQ2JjTiwU7Kh0g, audit_chain_id=['8_4AHHWtSQ2JjTiwU7Kh0g']) at 0x7fed2c7909b0>, 'user_id': b'd4fb94cfa3ce0f7829d76fe44697488e7765d88e29f5a896f57d43caadb0fad4', 'user_domain_id': '997b3e91271140feb1635eefba7c65a1', 'system_scope': None, 'project_id': None, 'project_domain_id': None, 'roles': [], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []}
(keystone.common.wsgi): 2019-06-10 19:47:02,702 WARNING You are not authorized to perform the requested action: identity:list_user_projects.

It actually appears elsewhere in the keystone.log that there is a string which has encapsulated bytecode data in it (or vice versa).

(keystone.common.wsgi): 2019-06-10 19:46:59,019 INFO POST https://keystone.mysite:5000/v3/auth/tokens
(sqlalchemy.orm.path_registry): 2019-06-10 19:46:59,021 DEBUG set 'memoized_setups' on path 'EntityRegistry((<Mapper at 0x7fed2eccfc50; RevocationEvent>,))' to '{}'
(sqlalchemy.pool.QueuePool): 2019-06-10 19:46:59,021 DEBUG Connection <pymysql.connections.Connection object at 0x7fed2c7d8320> checked out from pool
(sqlalchemy.pool.QueuePool): 2019-06-10 19:46:59,024 DEBUG Connection <pymysql.connections.Connection object at 0x7fed2c7d8320> being returned to pool
(sqlalchemy.pool.QueuePool): 2019-06-10 19:46:59,024 DEBUG Connection <pymysql.connections.Connection object at 0x7fed2c7d8320> rollback-on-return, via agent
(keystone.auth.core): 2019-06-10 19:46:59,025 DEBUG MFA Rules not processed for user `b'd4fb94cfa3ce0f7829d76fe44697488e7765d88e29f5a896f57d43caadb0fad4'`. Rule list: `[]` (Enabled: `True`).
(keystone.common.wsgi): 2019-06-10 19:46:59,025 ERROR a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/keystone/common/wsgi.py", line 148, in __call__
    result = method(req, **params)
  File "/usr/lib/python3/dist-packages/keystone/auth/controllers.py", line 102, in authenticate_for_token
    app_cred_id=app_cred_id, parent_audit_id=token_audit_id)
  File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 116, in wrapped
    __ret_val = __f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/keystone/token/provider.py", line 251, in issue_token
    token_id, issued_at = self.driver.generate_id_and_issued_at(token)
  File "/usr/lib/python3/dist-packages/keystone/token/providers/fernet/core.py", line 61, in generate_id_and_issued_at
    app_cred_id=token.application_credential_id
  File "/usr/lib/python3/dist-packages/keystone/token/token_formatters.py", line 159, in create_token
    protocol_id, access_token_id, app_cred_id
  File "/usr/lib/python3/dist-packages/keystone/token/token_formatters.py", line 483, in assemble
    b_user_id = cls.attempt_convert_uuid_hex_to_bytes(user_id)
  File "/usr/lib/python3/dist-packages/keystone/token/token_formatters.py", line 337, in attempt_convert_uuid_hex_to_bytes
    return (True, cls.convert_uuid_hex_to_bytes(value))
  File "/usr/lib/python3/dist-packages/keystone/token/token_formatters.py", line 290, in convert_uuid_hex_to_bytes
    uuid_obj = uuid.UUID(uuid_string)
  File "/usr/lib/python3.6/uuid.py", line 137, in __init__
    hex = hex.replace('urn:', '').replace('uuid:', '')
TypeError: a bytes-like object is required, not 'str'
(dogpile.lock): 2019-06-10 19:46:59,087 DEBUG value creation lock <dogpile.cache.region.CacheRegion._LockWrapper object at 0x7fed27e994e0> acquired
(dogpile.lock): 2019-06-10 19:46:59,087 DEBUG Calling creation function

[Drew Freiberger (afreiberger)]

Subscribed field-high as this is affecting usability of a production cloud.

[Drew Freiberger (afreiberger)]

Also of note, we have not seen success in listing LDAP group members, either. We believe all the ldap flags are properly configured for that, but it's also an anomaly. Not sure if it's same bug or different bug.

[James Page (james-page)]

The stack trace would indicate that the user_id parameter is binary encoded under py3.

[James Page (james-page)]

WARNING You are not authorized to perform the requested action: identity:list_user_projects is probably also pertinent here.

[James Page (james-page)]

>>> uuid.UUID(b'bf989240-79be-4155-8473-f12f1d13ed3d')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.7/uuid.py", line 157, in __init__
    hex = hex.replace('urn:', '').replace('uuid:', '')
TypeError: a bytes-like object is required, not 'str'

[James Page (james-page)]

user_id is passed right from the top of the stack as part of the authentication context

[James Page (james-page)]

Value: b'd4fb94cfa3ce0f7829d76fe44697488e7765d88e29f5a896f57d43caadb0fad4'

[James Page (james-page)]

I think this is a bit of a behaviour difference between py2 and py3 - Python 2 will throw a ValueError when passed a binary encoded string, Python 3 throws a TypeError.

[James Page (james-page)]

That said I don't think the user_id string should be byte encoded.

[Drew Freiberger (afreiberger)]

I performed a quick update to add in an exception for TypeError along with the ValueError exception handler, and was able to clean up the traceback, but it still doesn't function as expected.

/usr/lib/python3/dist-packages/keystone/token/token_formatters.py:

    def attempt_convert_uuid_hex_to_bytes(cls, value):
        """Attempt to convert value to bytes or return value.

        :param value: value to attempt to convert to bytes
        :returns: tuple containing boolean indicating whether user_id was
                  stored as bytes and uuid value as bytes or the original value

        """
        try:
            return (True, cls.convert_uuid_hex_to_bytes(value))
        except ValueError:
            # this might not be a UUID, depending on the situation (i.e.
            # federation)
            return (False, value)
        except TypeError:
            # lp#1832265 - py3 uuid may raise TypeError instead of ValueError
            return (False, value)

As I did, I saw the log lines flow through showing user_id as a string, and then immediately after, as a byte-encoded value in the log.

(keystone.common.controller): 2019-06-11 18:12:07,270 DEBUG RBAC: Adding query filter params ()
(keystone.common.authorization): 2019-06-11 18:12:07,270 DEBUG RBAC: Authorizing identity:list_user_projects(user_id=d4fb94cfa3ce0f7829d76fe44697488e7765d88e29f5a896f57d43caadb0fad4)
**** somewhere in here is where user_id ends up getting bytestring encoded ****
(keystone.policy.backends.rules): 2019-06-11 18:12:07,270 DEBUG enforce identity:list_user_projects: {'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_id': None, 'domain_name': None, 'group_ids': [], 'token': <TokenModel (audit_id=azlKaSkfR3arBR5tGIzRlw, audit_chain_id=['azlKaSkfR3arBR5tGIzRlw']) at 0x7fd2e1d53e80>, 'user_id': b'd4fb94cfa3ce0f7829d76fe44697488e7765d88e29f5a896f57d43caadb0fad4', 'user_domain_id': '997b3e91271140feb1635eefba7c65a1', 'system_scope': None, 'project_id': None, 'project_domain_id': None, 'roles': [], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []}

[Drew Freiberger (afreiberger)]

Taking a cue from def disassemble, I added in an exception handler for this binary_type user_id validation in (keystone.identity.backends.ldap.core). This appears to resolve the LDAP access issues within horizon as far as I'm able to tell. Basically, this is handling any call to the ldap UserApi subclass call for self.user.get(*attribute), to check for byte-encoded values coming from the ldap backend and strip them.

/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py diff:

*** a/core.py 2019-06-11 20:29:23.148303839 +0000
--- b/core.py 2019-06-11 20:27:23.854658160 +0000
*************** class UserApi(common_ldap.EnabledEmuMixI
*** 308,313 ****
--- 308,318 ----
      def get(self, user_id, ldap_filter=None):
          obj = super(UserApi, self).get(user_id, ldap_filter=ldap_filter)
          obj['options'] = {} # options always empty
+
+ # lp#1832265
+ if six.PY3 and isinstance(obj, six.binary_type):
+ obj = obj.decode('utf-8')
+
          return obj

      def get_filtered(self, user_id):

[Drew Freiberger (afreiberger)]

ok, that patch actually didn't work. what I'm finding is that I'm getting about 1 in 50 page refreshes in Horizon passing a token without byte-encoding which is giving me false-positives.

The obj in that function above is:
{'id': '30339', 'name': 'testuser', 'email': '<email address hidden>', 'dn': 'uid=testuser,cn=users,dc=mysite,dc=com', 'enabled': True, 'options': {}}

It's actually the user_id being passed to this function that is the byte-encoded object.

[Seyeong Kim (seyeongkim)]

FYI.

I was able to reproduce this issue in my local
and below code fixes this

inside token/token_formatters.py

https://review.opendev.org/#/c/663607/3/nova/virt/libvirt/storage/rbd_utils.py

but not sure it is proper for keystone as well?

[Drew Freiberger (afreiberger)]

  - Attempted to hack in the fix suggested by seyeong to cleanup byte-encoded strings, but
    getting only half fixed responses. half the actions still fail, I think this is because I placed
    this into attempt_convert_uuid_hex_to_bytes, rather than convert_uuid_hex_to_bytes.
    https://github.com/openstack/keystone/blob/master/keystone/token/token_formatters.py#L314,
  - It looks like there are a couple of contexts for assembling tokens
    that have domain_id being encapsulated through a different function,
    which may be part of the issue.
  - Dug through the ldap backend and found that there is a function that
    is supposed to take all the byte-coded data from ldap and turn it
    into strings and such for python usage:
    https://github.com/openstack/keystone/blob/master/keystone/identity/backends/ldap/common.py#L146
  - Tossing in debugging in this section of the code has resulted in
    showing everything coming out of the ldap result is pulled into
    python with proper formatting as all the variables get strained
    through a utf8_decode process when returned from ldap, so the issue
    is not in the ldap query response mangling.
    https://github.com/openstack/keystone/blob/master/keystone/identity/backends/ldap/common.py#L86
  - This leads me to believe the issue actually lies either in the
    database or in the code generating the tokens, which is why I'm most
    interested in the assemble routines in token/token_formatters.py.

[Drew Freiberger (afreiberger)]

Here's the patch I've been testing. It results in still having some failures to get project list.

https://pastebin.ubuntu.com/p/YC3KqH8WQJ/

[Drew Freiberger (afreiberger)]

Also tried this. seeing some amount of stripping happening in logs, for the user_id, but still unable to see projects drop-down in horizon when selected on secondary project membership.

Seyeong, if you're looking into reproducing, try adding a second project and make the user a Member of both projects, and then see after switching projects if you still see the projects drop-down.

*** token/token_formatters.py 2019-06-13 14:18:48.628706082 +0000
--- /home/ubuntu/token_formatters.py 2019-06-13 14:12:28.855950201 +0000
*************** class BasePayload(object):
*** 275,308 ****

          :param payload: this variant of payload
          :returns: a tuple of the payloads component data

          """
          raise NotImplementedError()

      @classmethod
      def convert_uuid_hex_to_bytes(cls, uuid_string):
          """Compress UUID formatted strings to bytes.

          :param uuid_string: uuid string to compress to bytes
          :returns: a byte representation of the uuid

          """
+ if isinstance(uuid_string, six.binary_type):
+ LOG.warning("DREW attempt_convert uuid_string: {}".format(uuid_string))
+ uuid_string = uuid_string.decode('utf-8')
+ LOG.warning("DREW attempt_convert uuid_string scrubbed: {}".format(uuid_string))
          uuid_obj = uuid.UUID(uuid_string)
          return uuid_obj.bytes

      @classmethod
      def convert_uuid_bytes_to_hex(cls, uuid_byte_string):
          """Generate uuid.hex format based on byte string.

          :param uuid_byte_string: uuid string to generate from
          :returns: uuid hex formatted string

          """
          uuid_obj = uuid.UUID(bytes=uuid_byte_string)
          return uuid_obj.hex

[Drew Freiberger (afreiberger)]

upgrading to field-critical.

The patch in place allows for some functionality but not enough for cloud to be usable by LDAP domain users.

[Ryan Beisner (1chb1n)]

This has an upstream bug element to it -- is that raised yet?

[Ryan Beisner (1chb1n)]

Please attach a sanitized bundle.yaml as an example reproducer to this bug.

[James Page (james-page)]

I feel this might be two bugs; one related to encoding of LDAP responses and the other due to behavioural changes - if that is the case we need to split them apart.

With regards to python3 encoding - the last three commits in master branch on token_formatters.py are related to py3 encoding - I think we need to pick those:

https://opendev.org/openstack/keystone/commit/79f468bad64c04a61e829029657fa6e905c610dd
https://opendev.org/openstack/keystone/commit/44c1b3d28407b66d2a854105f210d53f7cdd6b6f
https://opendev.org/openstack/keystone/commit/af3aef940c0162f12752a65282368d16c2d17c4f

One of the first two should cover the user_id encoding issue.

[James Page (james-page)]

Uploading test package to https://launchpad.net/~james-page/+archive/ubuntu/rocky

[James Page (james-page)]

The commits picked may not go far enough as they just seem to touch the Federated token formatter classes - may need to expand that to cover other Payload types.

[James Page (james-page)]

Tl;DR - I think the disassemble functions need to deal with the encoding better under py3; if the user_id gets into keystone decoded, then it should be dealt with correctly throughout.

[James Page (james-page)]

Raising a bug task against keystone as I think that we may need to expanded the decode coverage in token_formatters.

[James Page (james-page)]

Marking charm task as invalid - the py3 encoding issues are in keystone itself, not the deployment tooling.

[James Page (james-page)]

Behavioural bug reference bug 1832766

[Drew Freiberger (afreiberger)]

Positive outcome for initial testing of the ppa:james-page/rocky patch. contacting other users of this cloud for confirmation.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/665617

[Launchpad Janitor (janitor)]

This bug was fixed in the package keystone - 2:16.0.0~b1~git2019061326.4f1667679-0ubuntu2

---------------
keystone (2:16.0.0~b1~git2019061326.4f1667679-0ubuntu2) eoan; urgency=medium

  * d/p/token-consistently-decode-binary-types.patch: Ensure binary
    types are consistently decoded under Python 3 (LP: #1832265).

 -- James Page <email address hidden> Fri, 21 Jun 2019 05:32:39 +0100

[Dmitrii Shcherbakov (dmitriis)]

Ran into a related problem during debugging of dashboard errors ("Unable to retrieve key pairs") with a Rocky cloud & identity federation.

There was no clear indication as to why failures occurred.

https://paste.ubuntu.com/p/v5HXyyWXC2/ (full pdb trace)

At a high level I was getting validation failures for the identity provider (which was enabled in Keystone and was otherwise correct in terms of config) in the /v3/auth/token code path.

I narrowed it down to a validation error due to a type mismatch (bytes vs str):

1) the error occurs in send_notification:

> /usr/lib/python3/dist-packages/keystone/auth/plugins/mapped.py(101)handle_scoped_token()->None
-> send_notification(taxonomy.OUTCOME_SUCCESS)
(Pdb) l
 96 # send off failed authentication notification, raise the exception
 97 # after sending the notification
 98 send_notification(taxonomy.OUTCOME_FAILURE)
 99 raise
100 else:
101 -> send_notification(taxonomy.OUTCOME_SUCCESS)

# ...

2) this is how the validation error looks like:

(Pdb) setattr(self, FED_CRED_KEYNAME_IDENTITY_PROVIDER, identity_provider)
*** ValueError: identity_provider failed validation: <function FederatedCredential.<lambda> at 0x7fa0016ef9d8>

3) the lambda function where the error occurs

 67 class FederatedCredential(Credential):
 68 identity_provider = cadftype.ValidatorDescriptor(
 69 FED_CRED_KEYNAME_IDENTITY_PROVIDER,
 70 -> lambda x: isinstance(x, six.string_types))
 71 user = cadftype.ValidatorDescriptor(
 72 FED_CRED_KEYNAME_USER,
 73 lambda x: isinstance(x, six.string_types))
 74 groups = cadftype.ValidatorDescriptor(
 75 FED_CRED_KEYNAME_GROUPS,

4) type comparison (b'adfs' is the identity provider name):

((Pdb)) x
b'adfs'
((Pdb)) six.string_types
(<class 'str'>,)
((Pdb)) type(x)
<class 'bytes'>

Using a package from James' PPA helped as I am not getting errors in the same code-path anymore.

apt policy keystone
keystone:
  Installed: 2:14.1.0-0ubuntu2~ubuntu18.04.1~ppa201906140719
  Candidate: 2:14.1.0-0ubuntu2~ubuntu18.04.1~ppa201906140719
  Version table:
 *** 2:14.1.0-0ubuntu2~ubuntu18.04.1~ppa201906140719 500

When clicking through tabs very fast I encountered a glitch which results in the following error messages being displayed (see the screencast in the attachment):

Error: "Unable to retrieve key pairs"/"Unable to retrieve images"/""Unable to retrieve server groups"
Warning: "Policy check failed"

I tried to set breakpoints in the same place - the same validation error does NOT occur with the patch so this is something else unrelated to py2 vs py3 string handling.

[James Page (james-page)]

Ubuntu SRU information:

[Impact]
Due to inconsistent decode/encode of bytestrings under py3, keystone ldap integration is broken when keystone is run under Python 3.

[Test Case]
Deploy keystone
Configure to use LDAP

[Regression Potential]
The proposed patch has been +2'ed by upstream and validated as resolving this issue by the original bug reporter; the change simply ensures that any encoded values are decoded before use.

[Brian Murray (brian-murray)]

Hello Drew, or anyone else affected,

Accepted keystone into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/keystone/2:15.0.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Drew, or anyone else affected,

Accepted keystone into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

Hello Drew, or anyone else affected,

Accepted keystone into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[tom king (kingttx)]

James, I have the same issue. Which package do you want the version on? We have a PoC on Rocky using this charm and have this exact issue.

Do we put this repository on the controllers or everywhere (compute nodes, ceph, etc.)?

Thank you.

[Drew Freiberger (afreiberger)]

@kingttx You will only need to add this to all the keystone charm's units. Then run apt-get upgrade to install the updated keystone* packages, and lastly run 'service apache2 restart' in order to reload the wsgi scripts.

We have tested both bionic-rocky-proposed and bionic-stein-proposed with success.

[tom king (kingttx)]

@Drew Freiberger - that fixed my issue as well. I did have some problem with 'Empty Response' for a few minutes after updating the packages and restarting apache on the three Keystone nodes, but it finally started working and the Horizon issues seemed to be fixed.

Now on to the next error.

Thank you, folks!
Tom

[Corey Bryant (corey.bryant)]

@Drew/@Tom, If you've successfully tested disco-proposed, rocky-proposed or stein-proposed on bionic, can you tag the bug accordingly with verification-* tags? Thanks, Corey

[Drew Freiberger (afreiberger)]

I've updated the rocky/stein tags to verificiation-rocky/stein-done as requested per my comment #36.

[Edward Hope-Morley (hopem)]

Disco verified:

Test output:

ubuntu@hopem-bastion:~/stsstack-bundles/openstack$ openstack user list --domain userdomain
+------------------------------------------------------------------+----------+
| ID | Name |
+------------------------------------------------------------------+----------+
| 556fcbee7ba012ac2d452f4d7c63b38f2e943b5920ee55104224cabbd73384e3 | Jane Doe |
| 05a92b81187802919b4f0c5bbcb8d42106924aa1cc5ac1670cff66339a82351a | John Doe |
+------------------------------------------------------------------+----------+
ubuntu@hopem-bastion:~/stsstack-bundles/openstack$ juju ssh keystone/0 -- 'dpkg -l | grep keystone'
ii keystone 2:15.0.0-0ubuntu1.1 all OpenStack identity service - Daemons
ii keystone-common 2:15.0.0-0ubuntu1.1 all OpenStack identity service - Common files
ii python3-keystone 2:15.0.0-0ubuntu1.1 all OpenStack identity service - Python 3 library
ii python3-keystoneauth1 3.13.1-0ubuntu1 all authentication library for OpenStack Identity - Python 3.x
ii python3-keystoneclient 1:3.19.0-0ubuntu1 all client library for the OpenStack Keystone API - Python 3.x
ii python3-keystonemiddleware 6.0.0-0ubuntu1 all Middleware for OpenStack Identity (Keystone) - Python 3.x
Connection to 10.5.0.38 closed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/665617
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=ffa0918f5a92fd18c86703916d768012b0bea61b
Submitter: Zuul
Branch: master

commit ffa0918f5a92fd18c86703916d768012b0bea61b
Author: James Page <email address hidden>
Date: Mon Jun 17 09:56:11 2019 +0100

    token: consistently decode binary types

    Ensure that any binary types unpacked from message payloads
    are correctly converted from binary to text type.

    Under Python 3 msgpack returns the serialized input as a
    byte string. Similar to other msgpack'd values in the payload,
    we need to explicitly decode it to a string value.

    This is specifically more of an issue under Python 3; however
    the decode operation is safe back to Python 2 so there is no
    need to limit the decode codepath to just Python 3.

    Change-Id: Ib1073acf5677a60714d0a386de3bcd14ce6cd134
    Closes-Bug: 1832265

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for keystone has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package keystone - 2:15.0.0-0ubuntu1.1

---------------
keystone (2:15.0.0-0ubuntu1.1) disco; urgency=medium

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/stein branch.

  [ James Page ]
  * d/p/token-consistently-decode-binary-types.patch: Ensure binary
    types are consistently decoded under Python 3 (LP: #1832265).

 -- Corey Bryant <email address hidden> Fri, 12 Jul 2019 14:54:08 +0100

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for keystone has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package keystone - 2:15.0.0-0ubuntu1.1~cloud0
---------------

 keystone (2:15.0.0-0ubuntu1.1~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 keystone (2:15.0.0-0ubuntu1.1) disco; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/stein branch.
 .
   [ James Page ]
   * d/p/token-consistently-decode-binary-types.patch: Ensure binary
     types are consistently decoded under Python 3 (LP: #1832265).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for keystone has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package keystone - 2:14.1.0-0ubuntu1.1~cloud0
---------------

 keystone (2:14.1.0-0ubuntu1.1~cloud0) bionic; urgency=medium
 .
   * d/p/token-consistently-decode-binary-types.patch: Ensure binary
     types are consistently decoded under Python 3 (LP: #1832265).

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 16.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/690070

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690070
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=ac3d3125aeebbc7eed4b2586f782dc2b7fc3685b
Submitter: Zuul
Branch: stable/stein

commit ac3d3125aeebbc7eed4b2586f782dc2b7fc3685b
Author: James Page <email address hidden>
Date: Mon Jun 17 09:56:11 2019 +0100

    token: consistently decode binary types

    Ensure that any binary types unpacked from message payloads
    are correctly converted from binary to text type.

    Under Python 3 msgpack returns the serialized input as a
    byte string. Similar to other msgpack'd values in the payload,
    we need to explicitly decode it to a string value.

    This is specifically more of an issue under Python 3; however
    the decode operation is safe back to Python 2 so there is no
    need to limit the decode codepath to just Python 3.

    Conflicts:
        keystone/token/token_formatters.py

    Note: the file conflict is caused by patch
    I9529d6bee3e5bb1f618f40f225f69e2ad7e3f64a which is only present in
    stable/train.

    Change-Id: Ib1073acf5677a60714d0a386de3bcd14ce6cd134
    Closes-Bug: 1832265
    (cherry picked from commit ffa0918f5a92fd18c86703916d768012b0bea61b)