cached tokens break Liberty to Mitaka upgrade
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Identity (keystone) |
Fix Released
|
High
|
Colleen Murphy | ||
Mitaka |
Fix Released
|
High
|
Colleen Murphy | ||
Newton |
Fix Released
|
High
|
Colleen Murphy |
Bug Description
Sequence of events.
- Fernet tokens (didnt test with UUID)
- Running cluster with Liberty from about 6 weeks ago, so close to stable.
- Upgrade Keystone to Mitaka (automated)
- Tokens fail to issue for about 5 minutes, after this time, all the cached tokens are gone
- Everything works after that. See also Work-around at bottom.
Annotated logs:
Token call works to this point.
db_sync is running here, but code is still Liberty, DB now Mitaka:
An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-04dcb954-
An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-d27eee3a-
An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-265b6261-
Puppet bounces Keystone, the restarted code is Mitaka:
Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
Tokens fail to generate here due to the caching format changing. This will continue for about 5 minutes or so, I suspect it depends on whats in the cache and timeouts.
An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-8b835f67-
An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-b92bcd56-
An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-a787163f-
An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-e2ab7bf1-
Keystone log is full of this:
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
2016-06-13 21:37:58.947 35 ERROR keystone.
Work-around: run flush_all in memcache (telnet localhost 11211) every few seconds during the upgrade
summary: |
- liberty to mitaka upgrade (stable) has broken caching + cached tokens break liberty to Mitaka upgrade |
summary: |
- cached tokens break liberty to Mitaka upgrade + cached tokens break Liberty to Mitaka upgrade |
description: | updated |
description: | updated |
tags: | added: fernet |
description: | updated |
Changed in keystone: | |
importance: | Undecided → High |
tags: | added: mitaka-backport-potential |
Changed in keystone: | |
status: | New → Triaged |
Changed in keystone: | |
milestone: | none → newton-3 |
So per some IRC discussion, I don't expect that cached Liberty tokens will be valid in Mitaka, however, I do expect that the mere presence of cached Liberty tokens won't break new token issuance in Mitaka. I think this is a fair expectation and this has worked in every release since Havana when I started working on openstack.