cached tokens prevent to validate modified assignments

[Andrey Grebennikov]

Observed on Train.
4 units of Keystone deployed, each backed by shared mysql, with individual memcached local daemon for caching.
The user is assigned the role "member" in the project, validating new token returns proper assignment from every keystone unit:

[
  {
    "id": "758f4dc35c7b4377bfcba1ae083c2808",
    "name": "member"
  },
  {
    "id": "d1ea6664378645a788f44b8a1a44e874",
    "name": "reader"
  }
]

Adding the role of "admin" within the same project, request succeeded, database is adjusted, query of new roles list in the project for the user works fine.

User issues new token, validation returns new role in the project only from one unit of keystone, other 3 still show old list. Eventually (around 10 minutes timeframe) they all pick up.

$ for i in 43 52 53 60; do curl --insecure -i -H "X-Auth-Token: <redacted>" -H "X-Subject-Token: <redacted>" https://172.31.248.$i:35347/v3/auth/tokens |grep \{ |jq .token.roles; done
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 12433 100 12433 0 0 311k 0 --:--:-- --:--:-- --:--:-- 311k
[
  {
    "id": "a2565b5e5bb440ac9a964f726ccd2f26",
    "name": "admin"
  },
  {
    "id": "758f4dc35c7b4377bfcba1ae083c2808",
    "name": "member"
  },
  {
    "id": "d1ea6664378645a788f44b8a1a44e874",
    "name": "reader"
  }
]
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 12372 100 12372 0 0 294k 0 --:--:-- --:--:-- --:--:-- 294k
[
  {
    "id": "d1ea6664378645a788f44b8a1a44e874",
    "name": "reader"
  },
  {
    "id": "758f4dc35c7b4377bfcba1ae083c2808",
    "name": "member"
  }
]
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 12372 100 12372 0 0 280k 0 --:--:-- --:--:-- --:--:-- 287k
[
  {
    "id": "758f4dc35c7b4377bfcba1ae083c2808",
    "name": "member"
  },
  {
    "id": "d1ea6664378645a788f44b8a1a44e874",
    "name": "reader"
  }
]
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 12372 100 12372 0 0 246k 0 --:--:-- --:--:-- --:--:-- 246k
[
  {
    "id": "d1ea6664378645a788f44b8a1a44e874",
    "name": "reader"
  },
  {
    "id": "758f4dc35c7b4377bfcba1ae083c2808",
    "name": "member"
  }
]

The root cause is the "expiration_time" in [cache] section to be set to 600 seconds. During this time the units of keystone which were not yet issuing the token for the user with new assignments are returning old information.

[Corey Bryant (corey.bryant)]

@Andrey Thanks for reporting this. Can I mark this as a duplicate of https://bugs.launchpad.net/charm-keystone/+bug/1899117 or was there a different intention for this bug?