keystonemiddleware

keystonemiddleware translate 401 from keystone to 503

Bug #1747655 reported by lvmxh on 2018-02-06

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	keystonemiddleware	Fix Released	Undecided	Colleen Murphy

Bug Description

in cyborg or mogan project

Access a valid url by rest api with an invalid user name， it will return 503 error as follow:
$ curl -H "X-AUTH-TOKEN:$TOKEN" GET http://10.0.0.72:6667/v1/deployables
curl: (6) Could not resolve host: GET
<html>
<head>
  <title>503 Service Unavailable</title>
</head>
<body>
  <h1>503 Service Unavailable</h1>
  The server is currently unavailable. Please try again at a later time.<br /><br />

</body>

But the log show: it is 401.

2018-02-06 11:45:53.050 14183 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}: Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-96d7b949-a6eb-4172-842c-c893ee069373)
2018-02-06 11:45:53.098 14183 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}: Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9a0584ad-d308-44c2-8365-8499bdef0bfd)
2018-02-06 11:45:53.098 14183 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Identity server rejected authorization necessary to fetch token data: ServiceError: Identity server rejected authorization necessary to fetch token data

That's because auth_token._identity.verify_token() translate it to 503.
see the detail in the code.
https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/_identity.py#L234

        except ksa_exceptions.Unauthorized as e:
            self._LOG.info('Identity server rejected authorization')
            self._LOG.warning('Identity response: %s', e.response.text)
            if retry:
                self._LOG.info('Retrying validation')
                return self.verify_token(user_token, False)
            msg = _('Identity server rejected authorization necessary to '
                    'fetch token data')
            raise ksm_exceptions.ServiceError(msg)

Tags:

Revision history for this message

lvmxh (shaohef) wrote on 2018-02-06:

Other project also have the same issue.

That's is puzzle to end user.

Please fix it.

How to fix it:

define a new exception in _exceptions.py as follow:
class Unauthorized(exceptions.KeystoneMiddlewareException):
pass

https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/_exceptions.py

and keep it as Unauthorized in _identity.py as follow
        except ksa_exceptions.Unauthorized as e:
            self._LOG.info('Identity server rejected authorization')
            self._LOG.warning('Identity response: %s', e.response.text)
            if retry:
                self._LOG.info('Retrying validation')
                return self.verify_token(user_token, False)
            msg = _('Identity server rejected authorization necessary to '
                    'fetch token data')
            raise ksm_exceptions.Unauthorized(msg)

https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/_identity.py#L234

Revision history for this message

lvmxh (shaohef) wrote on 2018-02-06:

or return explicit that "503 Keystone Service Unavailable" as follow:

$ curl -H "X-AUTH-TOKEN:$TOKEN" GET http://10.0.0.72:6667/v1/deployables
curl: (6) Could not resolve host: GET
<html>
<head>
  <title>503 __***Keystone***__ Service Unavailable</title>
</head>
<body>
  <h1>503 Service Unavailable</h1>
  The server is currently unavailable. Please try again at a later time.<br /><br />

</body>

Revision history for this message

lvmxh (shaohef) wrote on 2018-02-06:

any way, have better let end user knew this is Exception from keystone auth.

Revision history for this message

lvmxh (shaohef) wrote on 2018-02-06:

This is the firstly path introduce ServiceError

https://github.com/openstack/keystonemiddleware/commit/7dcbdf774803a92bfc24704dec5f919ac75ad315#diff-91ac2359a4313d4dd281baef56c764e2R226

Revision history for this message

Jamie Lennox (jamielennox) wrote on 2018-02-06:

I'm not at an instance so i'm doing this from memory, however I think this is actually correct.

When you have a service, lets say nova, that configures auth_token middleware you have to give it a user/pass combination that is capable of validating tokens. This is configured in [keystone_authtoken].

These credentials are used to get the service token, the token for the nova user.

When nova calls keystone it sends the service token as X-Auth-Token and the user's incoming token as X-Service-Token (or something like this, can't remember exactly).

If you are getting a 401 then the user/pass you gave to nova is wrong and so keystone is saying you don't have permission to actually validate the token. If the user's token is wrong i think it returns a 403 (maybe 404).

That 401 from keystone means you have the wrong configuration in nova, and so returning 503 to the user is appropriate as it's entirely an admin configuration problem. If keystone returns 403 then it means the user's token is wrong and it will return a 401 to the user.

Revision history for this message

lvmxh (shaohef) wrote on 2018-02-07:

Hi Jamie Lennox:

We can keep 503.
But we can let user to know the error reason ASAP instead of guess, do you agree?

As a end user:
1. How do you know that the user/pass is wrong from the return response message?
"503 Service Unavailable" is meaningless to user.
"The server is currently unavailable. Please try again at a later time" is also misleading.

2. as you mentioned, developer knows ksa_exceptions.Unauthorized can be 401/403 (maybe 404).
IMHO He can do some improvement, for example classifies these exceptions(401/403).
Instead return the uniform msg as the code show.

            msg = _('Identity server rejected authorization necessary to '
                    'fetch token data')
            raise ksm_exceptions.ServiceError(msg)

The restful response also swallowed the above msg.

Revision history for this message

lvmxh (shaohef) wrote on 2018-02-07:

Hi Jamie Lennox:

Sometimes, admin make sure he has config right. But he forgots to create a user in keystone.
So "503 Service Unavailable. The server is currently unavailable. Please try again at a later time." can not hint that he miss something.

Revision history for this message

lvmxh (shaohef) wrote on 2018-02-07:

I also do an example:

make wrong user/pass in config. and restart the API(such as cyborg, mogan, neutron...)

For the previous token is stored in local cache, it is still valid for the coming request.

Not sure this is reasonable.

Revision history for this message

wangxiyuan (wangxiyuan) wrote on 2018-02-08:

++ for what Jamie said. It's correct to return 503 not 401, because this is an error made by service, not the user. For user, 503 means it's not his fault, there is maybe something wrong with the service.And the error reason should not be exposed to the user of cause.

Also agree with lvmxh that the message can be improved but not for the error message. When 503 happens, admin should check the error log, not the API response message. So I think we can improve the LOG here;
https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/_identity.py#L227

for the cache problem you mentioned, the previous token is cached means that that token is valid. Whatever then you change the user/passwd in the config file, that token is still created by Keystone and the related user is still exist in Keystone and is not changed. Then it's surely valid. So I think it's reasonable.

Revision history for this message

Chris Dent (cdent) wrote on 2018-02-20:

#10

See also https://bugs.launchpad.net/keystonemiddleware/+bug/1749797 where there's also a feeling that the error message could be improved. The client of a service that is using keystonemiddleware needs to know that it is keystone that is causing the 503, not the service they are accessing. Without something in the response body, they can't know.

Restricting that info to just the log doesn't help the client at all and since we cannot predict all the contexts within which the client will operate, we need to be flexible.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-20: Fix proposed to keystonemiddleware (master)

#11

Fix proposed to branch: master
Review: https://review.openstack.org/546108

Changed in keystonemiddleware:
assignee:	nobody → Chris Dent (cdent)
status:	New → In Progress

OpenStack Infra (hudson-openstack) on 2018-02-20

Changed in keystonemiddleware:
assignee:	Chris Dent (cdent) → Colleen Murphy (krinkle)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-20: Fix merged to keystonemiddleware (master)

#12

Reviewed: https://review.openstack.org/546108
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=d3352ff422db6ba6a5e7bd4f7220af0d97efd0ac
Submitter: Zuul
Branch: master

commit d3352ff422db6ba6a5e7bd4f7220af0d97efd0ac
Author: Chris Dent <email address hidden>
Date: Tue Feb 20 10:31:49 2018 +0000

Identify the keystone service when raising 503

    When the keystonemiddleware is used directly in the WSGI stack of an
    application, the 503 that is raised when the keystone service errors
    or cannot be reached needs to identify that keystone is the service
    that has failed, otherwise it appears to the client that it is the
    service they are trying to access is down, which is misleading.

    This addresses the problem in the most straightforward way possible:
    the exception that causes the 503 is given a message including the
    word "Keystone".

    The call method in BaseAuthTokenTestCase gains an
    expected_body_string kwarg. If not None, the response body (as
    a six.text_type) is compared with the value.

    Change-Id: Idf211e7bc99139744af232f5ea3ecb4be41551ca
    Closes-Bug: #1747655
    Closes-Bug: #1749797