auth_token does not updated when endpoint is updated

Bug #1813739 reported by Yang Youseok on 2019-01-29
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
keystonemiddleware
Undecided
Yang Youseok

Bug Description

nova-api using auth_token middleware does not find identity server when identity endpoint is not created yet emitting EndpointNotfound Exception. It's working normally since there is no proper endpoint in auth_token.

The problem is even after user create proper identity endpoint used by nova-api, auth_token does not re-authenticate til expiration time is reached. Operator can restart nova-api and re-initialize token auth though, I think it should be working without restart.

It does not related to caching in auth_token, since every different token also failed after identity endpoint created. It results from adapter in _identity_server does not be updated using original auth (original catalog which does not have identity endpoint yet).

So I think it's better to have any ways to be notified when endpoint updated in auth_token.

Thanks.

Changed in keystonemiddleware:
assignee: nobody → Yang Youseok (ileixe)
status: New → In Progress
Colleen Murphy (krinkle) wrote :

Could you include the traceback that you are seeing when this happens?

Yang Youseok (ileixe) wrote :
Download full text (14.1 KiB)

@Collen Sure. This is nova-api.log

2019-02-04 15:57:30.257 3569 DEBUG keystonemiddleware.auth_token [-] Identity endpoint not found. fetch_token /opt/openstack/src/keystonemiddleware/keystonemiddleware/auth_token/__init__.py:780
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token Traceback (most recent call last):
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/src/keystonemiddleware/keystonemiddleware/auth_token/__init__.py", line 762, in fetch_token
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token allow_expired=allow_expired)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/src/keystonemiddleware/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token allow_expired=allow_expired)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/src/keystonemiddleware/keystonemiddleware/auth_token/_identity.py", line 108, in verify_token
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token allow_expired=allow_expired)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token return wrapped(*args, **kwargs)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 110, in validate
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token allow_expired=allow_expired)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token return wrapped(*args, **kwargs)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 89, in get_token_data
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token resp, body = self._client.get(url, headers=headers)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 217, in get
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token return self.request(url, 'GET', **kwargs)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 374, in request
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 142, in request
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token return self.session.request(url, method, **kwargs)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2...

Yang Youseok (ileixe) wrote :

Traceback pastebin above: https://pastebin.com/75a7zjfg

Reviewed: https://review.openstack.org/633695
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=4e51cb8e6b4968fcb68903dce7e773b218f85bb7
Submitter: Zuul
Branch: master

commit 4e51cb8e6b4968fcb68903dce7e773b218f85bb7
Author: Yang Youseok <email address hidden>
Date: Tue Jan 29 18:59:12 2019 +0900

    Add auth invalidation in auth_token for identity endpoint update

    Currently auth_token middleware does not concern identity endpoint
    update since service catalog is not updated after service having
    auth_token middleware started.

    Add invalidation logic when EndpointNotfound exception occurs so
    that auth_token middleware can be notified of sevice catalog update
    without restart.

    Change-Id: I631ee1538883d732fe3987b172d987f703dad5c0
    Closes-Bug: #1813739

Changed in keystonemiddleware:
status: In Progress → Fix Released

This issue was fixed in the openstack/keystonemiddleware 6.0.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers