config-changed ValueError: Unable to detect peer relation cluster

[Frode Nordahl]

There seems to be a race condition between the availability of the cluster relation and the usage of peer storage in the charm.

From time to time I am seeing this:
2018-07-26 10:28:05 INFO juju-log Migrating the keystone database.
2018-07-26 10:28:16 DEBUG config-changed Traceback (most recent call last):
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 818, in <module>
2018-07-26 10:28:16 DEBUG config-changed main()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 811, in main
2018-07-26 10:28:16 DEBUG config-changed hooks.execute(sys.argv)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/hookenv.py", line 823, in execute
2018-07-26 10:28:16 DEBUG config-changed self._hooks[hook_name]()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1453, in wrapped_f
2018-07-26 10:28:16 DEBUG config-changed restart_functions)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/host.py", line 730, in restart_on_change_helper
2018-07-26 10:28:16 DEBUG config-changed r = lambda_f()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1452, in <lambda>
2018-07-26 10:28:16 DEBUG config-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-07-26 10:28:16 DEBUG config-changed return f(*args, **kwargs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/config-changed", line 196, in config_changed
2018-07-26 10:28:16 DEBUG config-changed do_openstack_upgrade_reexec(configs=CONFIGS)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 605, in do_openstack_upgrade_reexec
2018-07-26 10:28:16 DEBUG config-changed do_openstack_upgrade(configs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 647, in do_openstack_upgrade
2018-07-26 10:28:16 DEBUG config-changed migrate_database()
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2018-07-26 10:28:16 DEBUG config-changed return f(*args, **kwargs)
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 695, in migrate_database
2018-07-26 10:28:16 DEBUG config-changed peer_store('db-initialised', 'True')
2018-07-26 10:28:16 DEBUG config-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/charmhelpers/contrib/peerstorage/__init__.py", line 201, in peer_store
2018-07-26 10:28:16 DEBUG config-changed 'peer relation {}'.format(relation_name))
2018-07-26 10:28:16 DEBUG config-changed ValueError: Unable to detect peer relation cluster
2018-07-26 10:28:16 ERROR juju.worker.uniter.operation runhook.go:114 hook "config-changed" failed: exit status 1

[Frode Nordahl (fnordahl)]

Review: https://review.openstack.org/#/c/586203/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/586203

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/586203
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=a4c39edc24c3557f9da93223d04c5a4e1ff58e66
Submitter: Zuul
Branch: master

commit a4c39edc24c3557f9da93223d04c5a4e1ff58e66
Author: Frode Nordahl <email address hidden>
Date: Thu Jul 26 14:34:10 2018 +0200

    Use leader storage for `db-initialised` flag

    There are no relation-level concerns for this flag.

    Any pre-existing deployments with recent charms (as in released
    2015 or later) will have already migrated the peer storage to
    leader storage, so this change can build on that work directly.

    Ref: https://github.com/juju/charm-helpers/blame/master/charmhelpers/contrib/peerstorage/__init__.py

    Change-Id: Ia7362d257428b102c452d0e91bcf9b3378a1e6e9
    Closes-bug: #1783747