Fernet tokens fail for some users with LDAP identity backend

[Eric Brown]

The following bug fixed most situations where when using Fernet + LDAP identify backend.
        https://bugs.launchpad.net/keystone/+bug/1459382

However, some users have trouble, resulting in a UserNotFound exception in the logs with a UUID. Here's the error:
2015-09-18 20:04:47.313 12979 WARNING keystone.common.wsgi [-] Could not find user: 457269632042726f776e203732363230

So the issue is this. The user DN query + filter will return my user as:
   CN=Eric Brown 72620,OU=PAO_Users,OU=PaloAlto_California_USA,OU=NALA,OU=SITES,OU=Engineering,DC=vmware,DC=com

Therefore, I have to use CN as the user id attribute. My user id would therefore be "Eric Brown 72620". The fernet token_formatters.py attempts to convert this user id into a UUID. And in my case that is successful. It results in UUID of 457269632042726f776e203732363230. Of course, a user id of 457269632042726f776e203732363230 doesn't exist in LDAP, so as a result I get a UserNotFound. So I don't understand why the convert_uuid_bytes_to_hex is ever used in the case of LDAP backend.

For other users, the token_formatters.convert_uuid_bytes_to_hex() raises a ValueError and everything works. Here's an example that illustrates the behavior

>>> import uuid
>>> uuid_obj = uuid.UUID(bytes='Eric Brown 72620')
>>> uuid_obj.hex
'457269632042726f776e203732363230'

>>> import uuid
>>> uuid_obj = uuid.UUID(bytes='Your Mama')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/uuid.py", line 144, in __init__
    raise ValueError('bytes is not a 16-char string')
ValueError: bytes is not a 16-char string

Here's the complete traceback (after adding some additional debug):

2015-09-18 20:04:47.312 12979 WARNING keystone.common.wsgi [-] EWB Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 449, in __call__
    response = self.process_request(request)
  File "/usr/lib/python2.7/dist-packages/keystone/middleware/core.py", line 238, in process_request
    auth_context = self._build_auth_context(request)
  File "/usr/lib/python2.7/dist-packages/keystone/middleware/core.py", line 218, in _build_auth_context
    token_data=self.token_provider_api.validate_token(token_id))
  File "/usr/lib/python2.7/dist-packages/keystone/token/provider.py", line 198, in validate_token
    token = self._validate_token(unique_id)
  File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1013, in decorate
    should_cache_fn)
  File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 640, in get_or_create
    async_creator) as value:
  File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 158, in __enter__
    return self._enter()
  File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 98, in _enter
    generated = self._enter_create(createdtime)
  File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 149, in _enter_create
    created = self.creator()
  File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 612, in gen_value
    created_value = creator()
  File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1009, in creator
    return fn(*arg, **kw)
  File "/usr/lib/python2.7/dist-packages/keystone/token/provider.py", line 261, in _validate_token
    return self.driver.validate_v3_token(token_id)
  File "/usr/lib/python2.7/dist-packages/keystone/token/providers/fernet/core.py", line 258, in validate_v3_token
    audit_info=audit_ids)
  File "/usr/lib/python2.7/dist-packages/keystone/token/providers/common.py", line 441, in get_token_data
    self._populate_user(token_data, user_id, trust)
  File "/usr/lib/python2.7/dist-packages/keystone/token/providers/common.py", line 275, in _populate_user
    user_ref = self.identity_api.get_user(user_id)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 342, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 353, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1013, in decorate
    should_cache_fn)
  File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 640, in get_or_create
    async_creator) as value:
  File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 158, in __enter__
    return self._enter()
  File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 98, in _enter
    generated = self._enter_create(createdtime)
  File "/usr/lib/python2.7/dist-packages/dogpile/core/dogpile.py", line 149, in _enter_create
    created = self.creator()
  File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 612, in gen_value
    created_value = creator()
  File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1009, in creator
    return fn(*arg, **kw)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 753, in get_user
    ref = driver.get_user(entity_id)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap.py", line 79, in get_user
    return self.user.get_filtered(user_id)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap.py", line 264, in get_filtered
    user = self.get(user_id)
  File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 1859, in get
    ref = super(EnabledEmuMixIn, self).get(object_id, ldap_filter)
  File "/usr/lib/python2.7/dist-packages/keystone/common/ldap/core.py", line 1489, in get
    raise self._not_found(object_id)
UserNotFound: Could not find user: 457269632042726f776e203732363230

[Dolph Mathews (dolph)]

We have two methods designed to safely pack & unpack values in this scenario:

  https://github.com/openstack/keystone/blob/f3e3a653f9c9ce0f9a7ba842eff118e5887eb388/keystone/token/providers/fernet/token_formatters.py#L336-L362

But we must be using the variants of those calls that do not expect to encounter value errors:

  https://github.com/openstack/keystone/blob/f3e3a653f9c9ce0f9a7ba842eff118e5887eb388/keystone/token/providers/fernet/token_formatters.py#L285-L311

[Eric Brown (ericwb)]

Forgot to mention the recreate procedure.
- Log into Horizon with my user name / password
- Then Horizon blows up with "global name '_' is not defined" because I can't seem to handle the 404 UserNotFound.
- The keystone.log shows UserNotFound:
2015-09-18 20:04:47.313 12979 WARNING keystone.common.wsgi [-] Could not find user: 457269632042726f776e203732363230

[Eric Brown (ericwb)]

The root cause is that:
        uuid_obj = uuid.UUID('Eric Brown 72620 ')
will fail on the assembly or creation of the token, thus falling back to pure string user_id value.

But on the disassemble, a ValueError is not raised. This is because this user_id just happens to be 16 bytes, thus when casted into a byte string, it works fine and a UUID in hex is returned.
       uuid_obj = uuid.UUID(bytes='Eric Brown 72620')

So any 16-character user_id would have the same issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/226121

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/226121
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=794e1510cc91fbe0277e291bc2cabdfba478bef3
Submitter: Jenkins
Branch: master

commit 794e1510cc91fbe0277e291bc2cabdfba478bef3
Author: Eric Brown <email address hidden>
Date: Mon Sep 21 18:24:11 2015 -0700

    Handle 16-char non-uuid user IDs in payload

    If a user_id just happens to be of 16 character length, this will
    cause the convert_uuid_bytes_to_hex function to improperly return
    a UUID value instead of the user_id string unconverted.

    This patch modifies the payload to indicate whether the ID was in
    fact a UUID and the attempt to convert to bytes was successful.

    This change has effect on more than just user IDs. It also resovles
    potential issues with project IDs, group IDs, IDP IDs, and scope IDs.

    Change-Id: Ia4a4f760d67d8bbc22759c48fc800aef016b84ed
    Closes-Bug: #1497461

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/235142

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/236092

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/235142
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=9ec4e61940190624dca71f4024b09226d3f4ed08
Submitter: Jenkins
Branch: stable/liberty

commit 9ec4e61940190624dca71f4024b09226d3f4ed08
Author: Eric Brown <email address hidden>
Date: Mon Sep 21 18:24:11 2015 -0700

    Handle 16-char non-uuid user IDs in payload

    If a user_id just happens to be of 16 character length, this will
    cause the convert_uuid_bytes_to_hex function to improperly return
    a UUID value instead of the user_id string unconverted.

    This patch modifies the payload to indicate whether the ID was in
    fact a UUID and the attempt to convert to bytes was successful.

    This change has effect on more than just user IDs. It also resolves
    potential issues with project IDs, group IDs, IDP IDs, and scope IDs.

    Change-Id: Ia4a4f760d67d8bbc22759c48fc800aef016b84ed
    Closes-Bug: #1497461
    (cherry picked from commit 794e1510cc91fbe0277e291bc2cabdfba478bef3)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/keystone 9.0.0.0b1 development milestone.

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/keystone 8.0.1 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/236092
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=bd94a41eefa4a1208f06886c598b75cab8339250
Submitter: Jenkins
Branch: stable/kilo

commit bd94a41eefa4a1208f06886c598b75cab8339250
Author: Eric Brown <email address hidden>
Date: Mon Sep 21 18:24:11 2015 -0700

    Handle 16-char non-uuid user IDs in payload

    If a user_id just happens to be of 16 character length, this will
    cause the convert_uuid_bytes_to_hex function to improperly return
    a UUID value instead of the user_id string unconverted.

    This patch modifies the payload to indicate whether the ID was in
    fact a UUID and the attempt to convert to bytes was successful.

    This change has effect on more than just user IDs. It also resolves
    potential issues with project IDs, group IDs, IDP IDs, and scope IDs.

    Closes-Bug: #1497461

    Conflicts:
        keystone/tests/unit/token/test_fernet_provider.py
        keystone/token/providers/fernet/token_formatters.py

    Change-Id: Ia4a4f760d67d8bbc22759c48fc800aef016b84ed
    (cherry picked from commit 9ec4e61940190624dca71f4024b09226d3f4ed08)

[Andrey Larionov (anlarionov)]

Merging such compatibility breaking commit into stable branch is something i cannot understand. This patch makes unable to co-exists keystone of version 2015.1.2 and 2015.1.3 in one installation. Also this patch makes all previous issued tokens to be invalid, due to format change. Hope in future such things will not happen often