[SRU] admin token is not properly refreshed if it expires in v1.0.0

[HT_Sergio]

[Impact]

When a service (nova, cinder, etc) checks a user's token, it's possible the service's token has become invalid and needs to be refreshed before checking the user's token. However, there is a bug in keystonemiddleware v1.0.0 which doesn't properly refresh the token, so the invalid token is used twice and keystonemiddleware incorrectly asserts that the user's token is invalid. This causes all API requests to return 401 Unauthorized until the service is restarted:
Nova:
ERROR: Unauthorized (HTTP 401) (Request-ID: ...)

Cinder:
ERROR: Unauthorized (HTTP 401)

Glance:
Request returned failure status.
Invalid OpenStack Identity credentials.

This bug is fixed in v1.1.0

I'm creating this issue because Ubuntu packages v1.0.0 so potentially many people are running into this problem but I didn't see a bug report for it. The solution is to use a newer version of keystonemiddleware.

[Test Case]

1. start the service with a username, password, and tenant
2. perform some API request, so the server (ie. nova) gets a token and caches it internally
3. restart memcache, purging the service's cached token
4. perform the API request again

[Regression Potential]

The fix provided is minimal and has very low regression potential.

[Morgan Fainberg (mdrnstm)]

This has been corrected (but did not have a bug assigned) as of the 1.1.0 release of keystonemiddleware. If there are any deployers with this bug, it is recommended that you move to 1.1.0 or later of keystonemiddleware.

All deployers should be sure to use real user/passwords for the service users (e.g. Nova, etc).

[Corey Bryant (corey.bryant)]

Ubuntu has python-keystonemiddleware 1.0.0 in Juno so we'd have to backport the fix to 1.0.0. From that perspective, unfortunately, the fix in 1.1.0 is a large rework of the code. However, in discussing with HT_Sergio it appears we can fix this with a one-line change by setting the following line to 'self._admin_token', which will reset the correct variable name such that _get_admin_token() will fetch a new token:
 https://github.com/openstack/keystonemiddleware/blob/1.0.0/keystonemiddleware/auth_token.py#L1122

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Chris J Arges (arges)]

Hello HT_Sergio, or anyone else affected,

Accepted python-keystonemiddleware into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-keystonemiddleware/1.0.0-1ubuntu0.14.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[HT_Sergio (sergio-martins)]

Chris,

I don't have a 14.10 environment to test with, but I confirmed that this fixes the issue in 14.04 using the Juno release from http://ubuntu-cloud.archive.canonical.com/ubuntu trusty-updates/juno main. In my environment I reproduced the problem with v1.0.0-1~cloud0 installed. I then upgraded to 1.0.0-1ubuntu0.14.10.1 and was not able to reproduce the problem.

I also looked at a diff of the packages to confirm the right thing was changed. Looks good.

Thank you guys for working on this!!

[HT_Sergio (sergio-martins)]

The packages for python-keystonemiddleware available in 14.04 from the CloudArchive repo and in 14.10 from the official Ubuntu repo are the same (python-keystonemiddleware 1.0.0-1).

[Corey Bryant (corey.bryant)]

This also passes regression tests on juno. Thanks HT_Sergio!

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-keystonemiddleware - 1.0.0-1ubuntu0.14.10.1

---------------
python-keystonemiddleware (1.0.0-1ubuntu0.14.10.1) utopic; urgency=medium

  * d/p/refresh-expired-admin-token.patch: Fix bug to enable refresh of
    expired admin token. Thanks to Sergio Martins for recommending
    this fix (LP: #1460833).

 -- Corey Bryant <email address hidden> Wed, 17 Jun 2015 15:13:55 +0100

[Chris J Arges (arges)]

The verification of the Stable Release Update for python-keystonemiddleware has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.