keystonemiddleware translate 401 from keystone to 503

Bug #1747655 reported by lvmxh on 2018-02-06
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
keystonemiddleware
Undecided
Colleen Murphy

Bug Description

in cyborg or mogan project

Access a valid url by rest api with an invalid user name, it will return 503 error as follow:
$ curl -H "X-AUTH-TOKEN:$TOKEN" GET http://10.0.0.72:6667/v1/deployables
curl: (6) Could not resolve host: GET
<html>
 <head>
  <title>503 Service Unavailable</title>
 </head>
 <body>
  <h1>503 Service Unavailable</h1>
  The server is currently unavailable. Please try again at a later time.<br /><br />

 </body>

But the log show: it is 401.

2018-02-06 11:45:53.050 14183 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}: Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-96d7b949-a6eb-4172-842c-c893ee069373)
2018-02-06 11:45:53.098 14183 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}: Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-9a0584ad-d308-44c2-8365-8499bdef0bfd)
2018-02-06 11:45:53.098 14183 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Identity server rejected authorization necessary to fetch token data: ServiceError: Identity server rejected authorization necessary to fetch token data

That's because auth_token._identity.verify_token() translate it to 503.
see the detail in the code.
https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/_identity.py#L234

        except ksa_exceptions.Unauthorized as e:
            self._LOG.info('Identity server rejected authorization')
            self._LOG.warning('Identity response: %s', e.response.text)
            if retry:
                self._LOG.info('Retrying validation')
                return self.verify_token(user_token, False)
            msg = _('Identity server rejected authorization necessary to '
                    'fetch token data')
            raise ksm_exceptions.ServiceError(msg)

lvmxh (shaohef) wrote :

Other project also have the same issue.

That's is puzzle to end user.

Please fix it.

How to fix it:

define a new exception in _exceptions.py as follow:
class Unauthorized(exceptions.KeystoneMiddlewareException):
    pass

https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/_exceptions.py

and keep it as Unauthorized in _identity.py as follow
        except ksa_exceptions.Unauthorized as e:
            self._LOG.info('Identity server rejected authorization')
            self._LOG.warning('Identity response: %s', e.response.text)
            if retry:
                self._LOG.info('Retrying validation')
                return self.verify_token(user_token, False)
            msg = _('Identity server rejected authorization necessary to '
                    'fetch token data')
            raise ksm_exceptions.Unauthorized(msg)

https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/_identity.py#L234

lvmxh (shaohef) wrote :

or return explicit that "503 Keystone Service Unavailable" as follow:

$ curl -H "X-AUTH-TOKEN:$TOKEN" GET http://10.0.0.72:6667/v1/deployables
curl: (6) Could not resolve host: GET
<html>
 <head>
  <title>503 __***Keystone***__ Service Unavailable</title>
 </head>
 <body>
  <h1>503 Service Unavailable</h1>
  The server is currently unavailable. Please try again at a later time.<br /><br />

 </body>

lvmxh (shaohef) wrote :

any way, have better let end user knew this is Exception from keystone auth.

Jamie Lennox (jamielennox) wrote :

I'm not at an instance so i'm doing this from memory, however I think this is actually correct.

When you have a service, lets say nova, that configures auth_token middleware you have to give it a user/pass combination that is capable of validating tokens. This is configured in [keystone_authtoken].

These credentials are used to get the service token, the token for the nova user.

When nova calls keystone it sends the service token as X-Auth-Token and the user's incoming token as X-Service-Token (or something like this, can't remember exactly).

If you are getting a 401 then the user/pass you gave to nova is wrong and so keystone is saying you don't have permission to actually validate the token. If the user's token is wrong i think it returns a 403 (maybe 404).

That 401 from keystone means you have the wrong configuration in nova, and so returning 503 to the user is appropriate as it's entirely an admin configuration problem. If keystone returns 403 then it means the user's token is wrong and it will return a 401 to the user.

lvmxh (shaohef) wrote :

Hi Jamie Lennox:

We can keep 503.
But we can let user to know the error reason ASAP instead of guess, do you agree?

As a end user:
1. How do you know that the user/pass is wrong from the return response message?
"503 Service Unavailable" is meaningless to user.
"The server is currently unavailable. Please try again at a later time" is also misleading.

2. as you mentioned, developer knows ksa_exceptions.Unauthorized can be 401/403 (maybe 404).
IMHO He can do some improvement, for example classifies these exceptions(401/403).
Instead return the uniform msg as the code show.

            msg = _('Identity server rejected authorization necessary to '
                    'fetch token data')
            raise ksm_exceptions.ServiceError(msg)

The restful response also swallowed the above msg.

lvmxh (shaohef) wrote :

Hi Jamie Lennox:

Sometimes, admin make sure he has config right. But he forgots to create a user in keystone.
So "503 Service Unavailable. The server is currently unavailable. Please try again at a later time." can not hint that he miss something.

lvmxh (shaohef) wrote :

I also do an example:

make wrong user/pass in config. and restart the API(such as cyborg, mogan, neutron...)

For the previous token is stored in local cache, it is still valid for the coming request.

Not sure this is reasonable.

wangxiyuan (wangxiyuan) wrote :

++ for what Jamie said. It's correct to return 503 not 401, because this is an error made by service, not the user. For user, 503 means it's not his fault, there is maybe something wrong with the service.And the error reason should not be exposed to the user of cause.

Also agree with lvmxh that the message can be improved but not for the error message. When 503 happens, admin should check the error log, not the API response message. So I think we can improve the LOG here;
https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/_identity.py#L227

for the cache problem you mentioned, the previous token is cached means that that token is valid. Whatever then you change the user/passwd in the config file, that token is still created by Keystone and the related user is still exist in Keystone and is not changed. Then it's surely valid. So I think it's reasonable.

Chris Dent (cdent) wrote :

See also https://bugs.launchpad.net/keystonemiddleware/+bug/1749797 where there's also a feeling that the error message could be improved. The client of a service that is using keystonemiddleware needs to know that it is keystone that is causing the 503, not the service they are accessing. Without something in the response body, they can't know.

Restricting that info to just the log doesn't help the client at all and since we cannot predict all the contexts within which the client will operate, we need to be flexible.

Fix proposed to branch: master
Review: https://review.openstack.org/546108

Changed in keystonemiddleware:
assignee: nobody → Chris Dent (cdent)
status: New → In Progress
Changed in keystonemiddleware:
assignee: Chris Dent (cdent) → Colleen Murphy (krinkle)

Reviewed: https://review.openstack.org/546108
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=d3352ff422db6ba6a5e7bd4f7220af0d97efd0ac
Submitter: Zuul
Branch: master

commit d3352ff422db6ba6a5e7bd4f7220af0d97efd0ac
Author: Chris Dent <email address hidden>
Date: Tue Feb 20 10:31:49 2018 +0000

    Identify the keystone service when raising 503

    When the keystonemiddleware is used directly in the WSGI stack of an
    application, the 503 that is raised when the keystone service errors
    or cannot be reached needs to identify that keystone is the service
    that has failed, otherwise it appears to the client that it is the
    service they are trying to access is down, which is misleading.

    This addresses the problem in the most straightforward way possible:
    the exception that causes the 503 is given a message including the
    word "Keystone".

    The call method in BaseAuthTokenTestCase gains an
    expected_body_string kwarg. If not None, the response body (as
    a six.text_type) is compared with the value.

    Change-Id: Idf211e7bc99139744af232f5ea3ecb4be41551ca
    Closes-Bug: #1747655
    Closes-Bug: #1749797

Changed in keystonemiddleware:
status: In Progress → Fix Released

This issue was fixed in the openstack/keystonemiddleware 5.0.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers