Prevent openstack-upgrade to Rocky if not using Fernet tokens

[Jose Guedez]

It is documented in somewhat old release notes [0] that the token provider should be set to Fernet before the upgrade to Openstack Rocky. However, if the upgrade is attempted with the default token-provider for Queens (uuid), the charm attempts to perform the upgrade and fails, ending up in a broken state.

    traceback: |
      Traceback (most recent call last):
        File "/var/lib/juju/agents/unit-keystone-0/charm/charmhelpers/contrib/openstack/utils.py", line 1396, in do_action_openstack_upgrade
          upgrade_callback(configs=configs)
        File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 774, in do_openstack_upgrade
          bootstrap_keystone(configs=configs)
        File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 881, in bootstrap_keystone
          '--bootstrap-region-id', config('region').split()[0]),
        File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '('keystone-manage', 'bootstrap',

The traceback above happened because the path for fernet keys doesn't exist - /etc/keystone/fernet-keys/.

The keystone_error.log also has the following traceback since keystone is trying to import the uuid modules that are no longer available:

mod_wsgi (pid=965232): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-public'.
Traceback (most recent call last):
  File "/usr/bin/keystone-wsgi-public", line 54, in <module>
    application = initialize_public_application()
  File "/usr/lib/python2.7/dist-packages/keystone/server/wsgi.py", line 24, in initialize_public_application
    name='public', config_files=flask_core._get_config_files())
  File "/usr/lib/python2.7/dist-packages/keystone/server/flask/core.py", line 164, in initialize_application
    startup_application_fn=loadapp)
  File "/usr/lib/python2.7/dist-packages/keystone/server/__init__.py", line 46, in setup_backends
    drivers = backends.load_backends()
  File "/usr/lib/python2.7/dist-packages/keystone/server/backends.py", line 59, in load_backends
    drivers = {d._provides_api: d() for d in managers}
  File "/usr/lib/python2.7/dist-packages/keystone/server/backends.py", line 59, in <dictcomp>
    drivers = {d._provides_api: d() for d in managers}
  File "/usr/lib/python2.7/dist-packages/keystone/token/provider.py", line 93, in __init__
    super(Manager, self).__init__(CONF.token.provider)
  File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 181, in __init__
    self.driver = load_driver(self.driver_namespace, driver_name)
  File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 81, in load_driver
    raise ImportError(msg % {'name': driver_name, 'namespace': namespace})
ImportError: Unable to find 'uuid' driver in 'keystone.token.provider'.

I already created a documentation bug in the charm-guide to highlight the issue in the current documentation [1]. However, given it is a known issue (as documented in the 18.08 charm release notes), the charm should not attempt to perform the upgrade at all and instead short-circuit the process and return an error message.

Ideally, keystone should also check the token-provider configured and handle the mismatch, reporting an error instead of the unhandled exception above.

[0] https://docs.openstack.org/charm-guide/latest/1808.html
[1] https://bugs.launchpad.net/charm-deployment-guide/+bug/1901980

[Alex Kavanagh (ajkavanagh)]

In terms of the keystone charm it probably needs to be handled in the 'openstack-upgrade' action (to fail) if 'action-managed-upgrade' is True.

However, if 'action-managed-upgrade' is False, the the upgrade is triggered with a config-change hook, and so this could need to enter, say, blocked if the upgrade is triggered (openstack-origin=cloud:bionic-rocky) but the token-provider=uuid - however, I think this is problematic.

Then there is the further issue to check that "token-provider" can have an empty default, i.e. pick the default, which on <=queens is uuid, and on >=rocky is ignored. In this scenario, do we switch to fernet automatically during upgrade? Do we do that anyway?

I think my vote for a full fix (in the charm) is on the upgrade to rocky, pre-switch the token to fernet, and then do the upgrade.

[Aurelien Lourot (aurelien-lourot)]

We are able to consistently reproduce this failure with our OpenStack upgrade tests from Queens to Rocky. [1]

[1] https://github.com/openstack-charmers/charmed-openstack-tester/blob/master/tests/openstack-upgrade/tests/tests.yaml