OpenStack Barbican-Vault Charm

  1. Bug #1849323
  2. Comment #0

Comment 0 for bug 1849323

Revision history for this message
Rodrigo Barbieri (rodrigo-barbieri2010) wrote :

on a deployment using vault, every time refresh-secrets is issued, the tokens are refreshed and the leader sends the new tokens through relation-data.

If the vault is deployed in HA, upon switching vault leaders (let's say new leader is vault/2 and old leader is vault/0), the old token will remain in the relation data between the units that require secrets-storage (barbican-vault, ceph-osd, ...) and the old leader (vault/0). The new leader (vault/2) will issue new tokens on refresh-secrets action and provide them through relation (vault/2 <=> barbican-vault, ceph-osd), but the requiring units will read the old tokens from the relation-data of the old leader (vault/0 <=> barbican-vault, ceph-osd). Then, it causes the exception below.

The tokens should be read from the new leader (vault/2) instead. The code sections that are suspected to cause the problem are:

barbican-vault: https://github.com/juju-solutions/charms.reactive/blob/1ff9c476693d834bc0fcc284ff1a89302d084c91/charms/reactive/endpoints.py#L748

ceph-osd and other classic charms: https://github.com/openstack/charm-ceph-osd/blob/ec8bcd5f9de981f9bf262c08ffd4a5dd440a8e12/hooks/charmhelpers/contrib/openstack/vaultlocker.py#L44

2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined lib/charm/vault_utils.py:22: DeprecationWarning: Call to deprecated function '_post'. This method will be removed in version '0.8.0' Please use the 'post' method on the 'hvac.adapters' class moving forward.
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined response = client._post('/v1/sys/wrapping/unwrap')
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined Traceback (most recent call last):
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/charm/hooks/secrets-storage-relation-joined", line 22, in <module>
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined main()
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined bus.dispatch(restricted=restricted_mode)
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined _invoke(other_handlers)
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined handler.invoke()
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined self._action(*args)
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/charm/reactive/barbican_vault_handlers.py", line 63, in plugin_info_barbican_publish
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined secrets_storage.unit_token)
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "lib/charm/vault_utils.py", line 22, in retrieve_secret_id
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined response = client._post('/v1/sys/wrapping/unwrap')
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/utils.py", line 170, in new_func
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return method(*args, **kwargs)
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2577, in _post
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return self._adapter.post(*args, **kwargs)
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 103, in post
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return self.request('post', url, **kwargs)
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 272, in request
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined utils.raise_for_error(response.status_code, text, errors=errors)
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/utils.py", line 30, in raise_for_error
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined raise exceptions.InvalidRequest(message, errors=errors)
2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist
2019-10-10 14:10:35 ERROR juju.worker.uniter.operation runhook.go:132 hook "secrets-storage-relation-joined" failed: exit status 1