Race during Keystone deploy (fernet)

[Maciej Kucia]

RedHat 7.6 OpenStack Ocata
Custom build Docker images using binary type.
Keystone configured to use fernet tokens.

When keystone container is started it expects directory and tokens to be present.
This is checked by the following code https://github.com/openstack/keystone/blob/3d2b293d7edfb0bd4bdec9b33abc63d1308e10bd/keystone/token/providers/fernet/core.py#L36

In some rare scenarios, keystone container fails with

 2019-05-31 17:26:39.620011 File "/usr/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py", line 45, in _init_
 2019-05-31 17:26:39.620106 'Fernet keys.') % subs)
 2019-05-31 17:26:39.620126 SystemExit: /etc/keystone/fernet-keys/ does not contain keys, use keystone-manage fernet_setup to create Fernet keys.

When inspecting directory, keys are there

 (keystone)[root@osc1 fernet-keys]# ls -la
 total 12
 drwxrwx---. 2 keystone keystone 33 May 31 17:26 .
 drwxr-x---. 1 root keystone 61 May 31 17:26 ..
 rw------. 1 keystone keystone 44 May 31 17:26 0
 rw------. 1 keystone keystone 44 May 31 17:26 1
 rw------. 1 keystone keystone 44 May 31 17:26 2

Please note that the files creation time is the same as error message time (17:26).

Upon inspection of the ansible/roles/keystone/tasks/deploy.yml one can find that
init_fernet.yml task is executed after flush_handlers. When handlers are run, containers are created or restarted.

The obvious option would be to move init_fernet before handlers, but this task does require keystone_ssh and keystone_fernet to be up and running.

The solutions could include:
 - Changes in keystone itself to retry initialization as long as the keys are missing
 - Changes in keystone to fail in a way that the container will restart
 - Changes in kolla-ansible to enforce fernet init before keystone container starts.

The bug is found on Ocata but upon Ansible manifests inspection it could happen on master as well.

Workaround:
Restart Keystone container.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/665326

[Colleen Murphy (krinkle)]

I'm going to mark this as invalid for keystone. Keystone does expect the setup to happen in a specific order, and it is up to the operator or deployment tool to meet those requirements. Keystone can't accommodate implementation differences between different deployment tools.

[Mark Goddard (mgoddard)]

Agree with Colleen here - nothing to do on the Keystone side.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maciej Kucia (<email address hidden>) on branch: master
Review: https://review.opendev.org/665326