I'm seeing this with the latest 20.02 charms on an openstack-on-lxd bundle modified to add barbican/octavia/vault/barbican-vault and no spaces support. All three units are showing secrets-storage-relation-changed failed hooks. After deploying the bundle below, I have initialized vault, unsealed it, authorized the vault charm, and then after running refresh-secrets on the vault leader, the log below resulted from all of the barbican-vault unit logs. In my case, I'm not using spaces or overlays and my bundle can be found here for reproducing this issue in a juju-on-lxd lab: https://github.com/afreiberger/openstack-on-lxd/blob/add-bionic-stein-ha-bundle/bundle-bionic-stein-ha.yaml unit-barbican-vault-1: 13:13:00 INFO unit.barbican-vault/1.juju-log secrets-storage:122: Reactive main running for hook secrets-storage-relation-changed unit-barbican-vault-1: 13:13:01 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer> tracer: set flag endpoint.secrets-storage.changed.barbican-vault/0_role_id tracer: ++ queue handler hooks/relations/juju-info/requires.py:24:broken:juju-info tracer: ++ queue handler hooks/relations/tls-certificates/requires.py:109:broken:certificates tracer: ++ queue handler reactive/barbican_vault_handlers.py:34:secret_backend_vault_request unit-barbican-vault-1: 13:13:01 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer: set flag endpoint.secrets-storage.changed.barbican-vault/0_token unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer: set flag endpoint.secrets-storage.changed.barbican-vault/1_role_id unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer: set flag endpoint.secrets-storage.changed.barbican-vault/1_token unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer: set flag endpoint.secrets-storage.changed.barbican-vault/2_role_id unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer: set flag endpoint.secrets-storage.changed.barbican-vault/2_token unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer> tracer: set flag secrets-storage.available tracer: ++ queue handler reactive/barbican_vault_handlers.py:71:plugin_info_barbican_publish tracer: -- dequeue handler reactive/barbican_vault_handlers.py:34:secret_backend_vault_request unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer> tracer: starting handler dispatch, 27 flags set tracer: set flag barbican-vault-installed tracer: set flag charm.installed tracer: set flag charms.openstack.do-default-charm.installed tracer: set flag charms.openstack.do-default-config.changed tracer: set flag charms.openstack.do-default-update-status tracer: set flag config.default.ssl_ca tracer: set flag config.default.ssl_cert tracer: set flag config.default.ssl_key tracer: set flag endpoint.secrets-storage.changed tracer: set flag endpoint.secrets-storage.changed.barbican-vault/0_role_id tracer: set flag endpoint.secrets-storage.changed.barbican-vault/0_token tracer: set flag endpoint.secrets-storage.changed.barbican-vault/1_role_id tracer: set flag endpoint.secrets-storage.changed.barbican-vault/1_token tracer: set flag endpoint.secrets-storage.changed.barbican-vault/2_role_id tracer: set flag endpoint.secrets-storage.changed.barbican-vault/2_token tracer: set flag endpoint.secrets-storage.changed.egress-subnets tracer: set flag endpoint.secrets-storage.changed.ingress-address tracer: set flag endpoint.secrets-storage.changed.private-address tracer: set flag endpoint.secrets-storage.changed.vault_url tracer: set flag endpoint.secrets-storage.joined tracer: set flag endpoint.secrets.changed tracer: set flag endpoint.secrets.changed.egress-subnets tracer: set flag endpoint.secrets.changed.ingress-address tracer: set flag endpoint.secrets.changed.private-address tracer: set flag endpoint.secrets.joined tracer: set flag secrets-storage.available tracer: set flag secrets-storage.connected unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer: hooks phase, 0 handlers queued unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer> tracer: main dispatch loop, 3 handlers queued tracer: ++ queue handler hooks/relations/juju-info/requires.py:24:broken:juju-info tracer: ++ queue handler hooks/relations/tls-certificates/requires.py:109:broken:certificates tracer: ++ queue handler reactive/barbican_vault_handlers.py:71:plugin_info_barbican_publish unit-barbican-vault-1: 13:13:03 INFO unit.barbican-vault/1.juju-log secrets-storage:122: Invoking reactive handler: reactive/barbican_vault_handlers.py:71:plugin_info_barbican_publish unit-barbican-vault-1: 13:13:03 INFO unit.barbican-vault/1.juju-log secrets-storage:122: Retrieving secret-id from vault (http://10.0.8.3:8200) unit-barbican-vault-1: 13:13:03 ERROR unit.barbican-vault/1.juju-log secrets-storage:122: Hook error: Traceback (most recent call last): File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main bus.dispatch(restricted=restricted_mode) File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch _invoke(other_handlers) File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke handler.invoke() File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke self._action(*args) File "/var/lib/juju/agents/unit-barbican-vault-1/charm/reactive/barbican_vault_handlers.py", line 94, in plugin_info_barbican_publish secret_id = get_secret_id(secrets_storage, current_secret_id) File "/var/lib/juju/agents/unit-barbican-vault-1/charm/reactive/barbican_vault_handlers.py", line 59, in get_secret_id secret_id = vault_utils.retrieve_secret_id(url, token) File "lib/charm/vault_utils.py", line 25, in retrieve_secret_id response = client._post('/v1/sys/wrapping/unwrap') File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/hvac/utils.py", line 174, in new_func return method(*args, **kwargs) File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2579, in _post return self._adapter.post(*args, **kwargs) File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 107, in post return self.request('post', url, **kwargs) File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 336, in request response = super(JSONAdapter, self).request(*args, **kwargs) File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 304, in request utils.raise_for_error(response.status_code, text, errors=errors) File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/hvac/utils.py", line 32, in raise_for_error raise exceptions.InvalidRequest(message, errors=errors) hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed lib/charm/vault_utils.py:25: DeprecationWarning: Call to deprecated function '_post'. This method will be removed in version '0.8.0' Please use the 'post' method on the 'hvac.adapters' class moving forward. unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed response = client._post('/v1/sys/wrapping/unwrap') unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed Traceback (most recent call last): unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-barbican-vault-1/charm/hooks/secrets-storage-relation-changed", line 22, in unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed main() unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed bus.dispatch(restricted=restricted_mode) unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed _invoke(other_handlers) unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed handler.invoke() unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed self._action(*args) unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-barbican-vault-1/charm/reactive/barbican_vault_handlers.py", line 94, in plugin_info_barbican_publish unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed secret_id = get_secret_id(secrets_storage, current_secret_id) unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-barbican-vault-1/charm/reactive/barbican_vault_handlers.py", line 59, in get_secret_id unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed secret_id = vault_utils.retrieve_secret_id(url, token) unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed File "lib/charm/vault_utils.py", line 25, in retrieve_secret_id unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed response = client._post('/v1/sys/wrapping/unwrap') unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/hvac/utils.py", line 174, in new_func unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed return method(*args, **kwargs) unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed File "/var/lib/juju/agents/unit-barbican-vault-1/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2579, in _post unit-barbican-vault-1: 13:13:03 DEBUG unit.barbican-vault/1.secrets-storage-relation-changed return self._adapter.post(*args, **kwargs)