mapping yield no valid identity result in HTTP 500 error

[Guang Yee]

A mapping which yield no valid identity (i.e. no local user or group) will result in HTTP 500 instead of 401. There are two issues.

1. We automatically return a default ephemeral user mapped_properties when mapping yield no valid local identity or groups.
2. In the mapped auth plugin, we assume the mapped_properties contains a valid local identity or group.

To reproduce the problem:

1. Set up WebSSO or K2K.
2. Create a mapping rule for the given IdP and protocol which yield neither local identity or group. For example,

    [
             {
                 "local": [
                     {
                        "user": {
                            "type": "local",
                            "name": "{0}",
                            "domain": {
                                "name": "{1}"
                            },
                            "type": "local"
                        }
                     }
                ],
                "remote": [
                    {
                        "type": "openstack_user"
                    },
                    {
                        "type": "openstack_user_domain"
                    },
                    {
                        "type": "openstack_roles",
                        "any_one_of": [
                            "bogus"
                        ]
                    }
                ]
            }
        ]

3. do the federation dance and you'll get a HTTP 500 and a traceback as pretty as this one.

2016-03-14 17:16:05.536 12497 DEBUG keystone.federation.utils [req-159bde9f-8a2d-4885-af31-304be9af8db7 - - - - -] updating a direct mapping: [u'Unset'] 2016-03-14 17:16:05.536 _verify_all_requirements /opt/stack/keystone/keystone/federation/utils.py:796
2016-03-14 17:16:05.536 12497 DEBUG keystone.federation.utils [req-159bde9f-8a2d-4885-af31-304be9af8db7 - - - - -] identity_values: [] 2016-03-14 17:16:05.536 process /opt/stack/keystone/keystone/federation/utils.py:534
2016-03-14 17:16:05.536 12497 DEBUG keystone.federation.utils [req-159bde9f-8a2d-4885-af31-304be9af8db7 - - - - -] mapped_properties: {'group_ids': [], 'user': {'domain': {'id': 'Federated'}, 'type': 'ephemeral'}, 'group_names': []} 2016-03-14 17:16:05.536 process /opt/stack/keystone/keystone/federation/utils.py:536
2016-03-14 17:16:05.620 12497 ERROR keystone.common.wsgi [req-159bde9f-8a2d-4885-af31-304be9af8db7 - - - - -] 'name'
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi Traceback (most recent call last):
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/common/wsgi.py", line 249, in __call__
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi result = method(context, **params)
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/federation/controllers.py", line 302, in federated_authentication
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi return self.authenticate_for_token(context, auth=auth)
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/auth/controllers.py", line 396, in authenticate_for_token
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi self.authenticate(context, auth_info, auth_context)
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/auth/controllers.py", line 520, in authenticate
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi auth_context)
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/auth/plugins/mapped.py", line 65, in authenticate
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi self.identity_api)
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/auth/plugins/mapped.py", line 144, in handle_unscoped_token
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi get_user_unique_id_and_display_name(context, mapped_properties)
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi File "/opt/stack/keystone/keystone/auth/plugins/mapped.py", line 253, in get_user_unique_id_and_display_name
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi return (user['id'], user['name'])
2016-03-14 17:16:05.620 12497 TRACE keystone.common.wsgi KeyError: 'name'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/293184

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/293184
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=e5dcb3b4b6bdecd0947cba32cb3732ca52ed07c3
Submitter: Jenkins
Branch: master

commit e5dcb3b4b6bdecd0947cba32cb3732ca52ed07c3
Author: guang-yee <email address hidden>
Date: Tue Mar 15 17:29:42 2016 -0700

    Mapping which yield no identities should result in ValidationError

    Currently mapping produce a bogus "blind" default identity when no
    rules match the incoming attributes. This is unnecessary and downright
    dangerous. There's absolutely no use case for the "blind" identity.
    Furthermore, consumers of mapped properties assumed that the "blind"
    identity is legit. This lead to expected failures such as KeyError when they
    try to reference the required identity attributes such as user['name'].

    We should raise ValidationError if the rules yield no valid identity.
    This patch also removed the tests where the bogus "blind" identity is
    expected.

    Change-Id: I117621673ffc0b4f8e2c48721329daa3b6090327
    Closes-Bug: 1557238

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/keystone 9.0.0.0rc1 release candidate.

[Sohan Sangwan (sohaninfo)]

I am using latest version of Keystone. I checked the above fix is available. I am still getting HTTP 500 error as follows:

{"error": {"message": "An unexpected error prevented the server from fulfilling your request.", "code": 500, "title": "Internal Server Error"}}

-------

Here follows the log:

 GET http://keystoneopenam.com:5000/v3/auth/OS-FEDERATION/websso/oidc?origin=http://localhost/horizon/auth/websso/
2016-08-22 17:51:58.704790 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi [req-21ba563c-09f0-48a2-a09c-51c6451a911b - - - - -] 'name'
2016-08-22 17:51:58.704924 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi Traceback (most recent call last):
2016-08-22 17:51:58.704967 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 249, in __call__
2016-08-22 17:51:58.705005 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi result = method(context, **params)
2016-08-22 17:51:58.705042 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/federation/controllers.py", line 320, in federated_sso_auth
2016-08-22 17:51:58.705079 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi protocol_id)
2016-08-22 17:51:58.705115 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/federation/controllers.py", line 302, in federated_authentication
2016-08-22 17:51:58.705151 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi return self.authenticate_for_token(context, auth=auth)
2016-08-22 17:51:58.705187 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 396, in authenticate_for_token
2016-08-22 17:51:58.705224 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi self.authenticate(context, auth_info, auth_context)
2016-08-22 17:51:58.705259 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/auth/controllers.py", line 520, in authenticate
2016-08-22 17:51:58.705295 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi auth_context)
2016-08-22 17:51:58.705331 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/auth/plugins/mapped.py", line 65, in authenticate
2016-08-22 17:51:58.705367 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi self.identity_api)
2016-08-22 17:51:58.705403 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/auth/plugins/mapped.py", line 149, in handle_unscoped_token
2016-08-22 17:51:58.705438 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi get_user_unique_id_and_display_name(context, mapped_properties)
2016-08-22 17:51:58.705474 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi File "/usr/lib/python2.7/dist-packages/keystone/auth/plugins/mapped.py", line 258, in get_user_unique_id_and_display_name
2016-08-22 17:51:58.705519 2016-08-22 17:51:58.703 14509 ERROR keystone.common.wsgi return (user['id'], user['name'])
2016...

[Sohan Sangwan (sohaninfo)]

The mapping I am using is as follows:

[
  {
    "local": [
      {
        "group": {
          "id": "3d609d380c1a488e837e28a4ea43a399"
          }
        }
      ],
    "remote": [
        {
          "type": "HTTP_OIDC_ISS",
          "any_one_of": [
            "https://accounts.google.com"
            ]
          }
        ]
  }
]

[Sylvain Benner (syl20bnr)]

I got a meaningful 401 in M release.

In N release with same config (ADFS federation with saml2) I get this generic error message with code 500, it forced me to dig into Keystone code to trace the origin from the error and found this change [1] which is responsible.

If I revert the change to just write to the log then an unscoped token with no group is returned. From my point of view this is the correct behavior right? I just found out that federation is not limited to group membership anymore which is great, so why raising an exception if no group can be mapped ? Is it because the mappings try to map remote properties to local group ? I find pretty handy that a rule trying to map both user and groups can still yield a token even if there is no group matching in Keystone.

As an aside, I would be immensely grateful for anyone telling me how to get the stack trace directly when such exception appears in the browser agent, tracing it manually was painfull ;-( (OTOH I learnt a bit more about the code).

[Sylvain Benner (syl20bnr)]

missing link above.

[1] https://github.com/openstack/keystone/commit/7ba53701989490667d220a3faecae2b484a007c5#diff-123d55255e45e44f9baa492e882762b5R386

[Sylvain Benner (syl20bnr)]

The exception is thrown even when there are matching groups defined in Keystone.

Here is my mapping rule:

[
  {
    "local": [
      {
        "user": {
          "name": "{0}"
        },
        "domain": {
          "name": "our_default_domain"
        }
      },
      {
        "groups": "{1}",
        "domain": {
          "name": "our_default_domain"
        }
      }
    ],
    "remote": [
      {
        "type": "UPN"
      },
      {
        "type": "GROUPS"
      }
    ]
  }
]

Result if I let the exception to be raised:

{"error": {"message": "An unexpected error prevented the server from fulfilling your request.", "code": 500, "title": "Internal Server Error"}}

Result if I comment it:

{"token": {"issued_at": "2016-11-12T17:33:46.000000Z", "audit_ids": ["uwj6lVPqS2e0vgs4wCtcuQ"], "methods": ["saml2"], "expires_at": "2016-11-12T18:33:46.000000Z", "user": {"OS-FEDERATION": {"identity_provider": {"id": "xxxx-adfs"}, "protocol": {"id": "saml2"}, "groups": [{"id": "fe3f975be4da43c698ea3e7cf9e40b00"}]}, "domain": {"id": "Federated", "name": "Federated"}, "id": "0827722a5ebe4ef1af6c3d66b9ab55ec", "name": "<email address hidden>"}}}

[Lance Bragstad (lbragstad)]

Was this a regression? If so, then we should reopen the bug and close bug 1648798

[Lance Bragstad (lbragstad)]

Adding this to keystone's office hours so that we can figure out if this was actually a regression.