Comment 1 for bug 1809469

So I believe the tokens Keystone hands out last 1 hour (not sure on that), and with three controllers the default behaviour is to rotate every 8 hours:

ssh ctrl1 sudo cat /etc/kolla/keystone-fernet/crontab
0 0 * * * /usr/bin/fernet-rotate.sh
ssh ctrl2 sudo cat /etc/kolla/keystone-fernet/crontab
0 8 * * * /usr/bin/fernet-rotate.sh
ssh ctrl3 sudo cat /etc/kolla/keystone-fernet/crontab
0 16 * * * /usr/bin/fernet-rotate.sh

For each of these, fernet-rotate is giving you roughly the behaviour noted here:
https://docs.openstack.org/keystone/pike/admin/identity-fernet-token-faq.html#how-should-i-approach-key-distribution

The logs show the correct things happening:
May 9th 2019, 09:00:02.000 INFO ctrl2 keystone Excess key to purge: /etc/keystone/fernet-keys/139
 May 9th 2019, 01:00:02.000 INFO ctrl1 keystone Excess key to purge: /etc/keystone/fernet-keys/138
 May 8th 2019, 17:00:03.000 INFO ctrl3 keystone Excess key to purge: /etc/keystone/fernet-keys/137

However, we still see these logs from keystone:
May 9th 2019, 09:06:34.000 WARNING ctrl1 keystone
 This is not a recognized Fernet token <snip> TokenNotFound

Which suggests some clients think they have a valid token, but they don't, after the above rotation.

Possibly we need to set keystone CONF.fernet_tokens.max_active_keys?

cfg.IntOpt(
    'max_active_keys',
    default=3,
    min=1,
    help=utils.fmt("""
This controls how many keys are held in rotation by `keystone-manage
fernet_rotate` before they are discarded. The default value of 3 means that
keystone will maintain one staged key (always index 0), one primary key (the
highest numerical index), and one secondary key (every other index). Increasing
this value means that additional secondary keys will be kept in the rotation.
"""))