Activity log for bug #1849323

Date Who What changed Old value New value Message
2019-10-22 14:16:21 Rodrigo Barbieri bug added bug
2019-10-22 14:16:51 Rodrigo Barbieri charm-barbican-vault: assignee Rodrigo Barbieri (rodrigo-barbieri2010)
2019-10-22 14:18:08 Rodrigo Barbieri bug task added charm-helpers
2019-10-22 14:18:20 Rodrigo Barbieri charm-barbican-vault: status New In Progress
2019-10-22 14:18:43 Tiago Pasqualini da Silva charm-helpers: assignee Tiago Pasqualini da Silva (tiago.pasqualini)
2019-10-22 14:18:52 Tiago Pasqualini da Silva tags sts
2019-10-22 14:20:19 Rodrigo Barbieri description on a deployment using vault, every time refresh-secrets is issued, the tokens are refreshed and the leader sends the new tokens through relation-data. If the vault is deployed in HA, upon switching vault leaders (let's say new leader is vault/2 and old leader is vault/0), the old token will remain in the relation data between the units that require secrets-storage (barbican-vault, ceph-osd, ...) and the old leader (vault/0). The new leader (vault/2) will issue new tokens on refresh-secrets action and provide them through relation (vault/2 <=> barbican-vault, ceph-osd), but the requiring units will read the old tokens from the relation-data of the old leader (vault/0 <=> barbican-vault, ceph-osd). Then, it causes the exception below. The tokens should be read from the new leader (vault/2) instead. The code sections that are suspected to cause the problem are: barbican-vault: https://github.com/juju-solutions/charms.reactive/blob/1ff9c476693d834bc0fcc284ff1a89302d084c91/charms/reactive/endpoints.py#L748 ceph-osd and other classic charms: https://github.com/openstack/charm-ceph-osd/blob/ec8bcd5f9de981f9bf262c08ffd4a5dd440a8e12/hooks/charmhelpers/contrib/openstack/vaultlocker.py#L44 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined lib/charm/vault_utils.py:22: DeprecationWarning: Call to deprecated function '_post'. This method will be removed in version '0.8.0' Please use the 'post' method on the 'hvac.adapters' class moving forward. 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined response = client._post('/v1/sys/wrapping/unwrap') 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined Traceback (most recent call last): 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/charm/hooks/secrets-storage-relation-joined", line 22, in <module> 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined main() 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined bus.dispatch(restricted=restricted_mode) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined _invoke(other_handlers) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined handler.invoke() 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined self._action(*args) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/charm/reactive/barbican_vault_handlers.py", line 63, in plugin_info_barbican_publish 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined secrets_storage.unit_token) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "lib/charm/vault_utils.py", line 22, in retrieve_secret_id 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined response = client._post('/v1/sys/wrapping/unwrap') 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/utils.py", line 170, in new_func 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return method(*args, **kwargs) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2577, in _post 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return self._adapter.post(*args, **kwargs) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 103, in post 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return self.request('post', url, **kwargs) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 272, in request 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined utils.raise_for_error(response.status_code, text, errors=errors) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/utils.py", line 30, in raise_for_error 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined raise exceptions.InvalidRequest(message, errors=errors) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist 2019-10-10 14:10:35 ERROR juju.worker.uniter.operation runhook.go:132 hook "secrets-storage-relation-joined" failed: exit status 1 on a deployment using vault, every time refresh-secrets is issued, the tokens are refreshed and the leader sends the new tokens through relation-data. If the vault is deployed in HA, upon switching vault leaders (let's say new leader is vault/2 and old leader is vault/0), the old token will remain in the relation data between the units that require secrets-storage (barbican-vault, ceph-osd, ...) and the old leader (vault/0). The new leader (vault/2) will issue new tokens on refresh-secrets action and provide them through relation (vault/2 <=> barbican-vault, ceph-osd), but the requiring units will read the old tokens from the relation-data of the old leader (vault/0 <=> barbican-vault, ceph-osd). Then, it causes the exception below. The tokens should be read from the new leader (vault/2) instead. As a workaround, if the leader is switched back to vault/0, the problem goes away until vault leader is changed again. The code sections that are suspected to cause the problem are: barbican-vault: https://github.com/juju-solutions/charms.reactive/blob/1ff9c476693d834bc0fcc284ff1a89302d084c91/charms/reactive/endpoints.py#L748 ceph-osd and other classic charms: https://github.com/openstack/charm-ceph-osd/blob/ec8bcd5f9de981f9bf262c08ffd4a5dd440a8e12/hooks/charmhelpers/contrib/openstack/vaultlocker.py#L44 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined lib/charm/vault_utils.py:22: DeprecationWarning: Call to deprecated function '_post'. This method will be removed in version '0.8.0' Please use the 'post' method on the 'hvac.adapters' class moving forward. 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined response = client._post('/v1/sys/wrapping/unwrap') 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined Traceback (most recent call last): 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/charm/hooks/secrets-storage-relation-joined", line 22, in <module> 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined main() 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined bus.dispatch(restricted=restricted_mode) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined _invoke(other_handlers) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined handler.invoke() 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined self._action(*args) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/charm/reactive/barbican_vault_handlers.py", line 63, in plugin_info_barbican_publish 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined secrets_storage.unit_token) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "lib/charm/vault_utils.py", line 22, in retrieve_secret_id 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined response = client._post('/v1/sys/wrapping/unwrap') 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/utils.py", line 170, in new_func 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return method(*args, **kwargs) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2577, in _post 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return self._adapter.post(*args, **kwargs) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 103, in post 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return self.request('post', url, **kwargs) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 272, in request 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined utils.raise_for_error(response.status_code, text, errors=errors) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/utils.py", line 30, in raise_for_error 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined raise exceptions.InvalidRequest(message, errors=errors) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist 2019-10-10 14:10:35 ERROR juju.worker.uniter.operation runhook.go:132 hook "secrets-storage-relation-joined" failed: exit status 1
2019-10-23 20:25:37 Tiago Pasqualini da Silva description on a deployment using vault, every time refresh-secrets is issued, the tokens are refreshed and the leader sends the new tokens through relation-data. If the vault is deployed in HA, upon switching vault leaders (let's say new leader is vault/2 and old leader is vault/0), the old token will remain in the relation data between the units that require secrets-storage (barbican-vault, ceph-osd, ...) and the old leader (vault/0). The new leader (vault/2) will issue new tokens on refresh-secrets action and provide them through relation (vault/2 <=> barbican-vault, ceph-osd), but the requiring units will read the old tokens from the relation-data of the old leader (vault/0 <=> barbican-vault, ceph-osd). Then, it causes the exception below. The tokens should be read from the new leader (vault/2) instead. As a workaround, if the leader is switched back to vault/0, the problem goes away until vault leader is changed again. The code sections that are suspected to cause the problem are: barbican-vault: https://github.com/juju-solutions/charms.reactive/blob/1ff9c476693d834bc0fcc284ff1a89302d084c91/charms/reactive/endpoints.py#L748 ceph-osd and other classic charms: https://github.com/openstack/charm-ceph-osd/blob/ec8bcd5f9de981f9bf262c08ffd4a5dd440a8e12/hooks/charmhelpers/contrib/openstack/vaultlocker.py#L44 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined lib/charm/vault_utils.py:22: DeprecationWarning: Call to deprecated function '_post'. This method will be removed in version '0.8.0' Please use the 'post' method on the 'hvac.adapters' class moving forward. 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined response = client._post('/v1/sys/wrapping/unwrap') 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined Traceback (most recent call last): 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/charm/hooks/secrets-storage-relation-joined", line 22, in <module> 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined main() 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined bus.dispatch(restricted=restricted_mode) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined _invoke(other_handlers) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined handler.invoke() 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined self._action(*args) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/charm/reactive/barbican_vault_handlers.py", line 63, in plugin_info_barbican_publish 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined secrets_storage.unit_token) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "lib/charm/vault_utils.py", line 22, in retrieve_secret_id 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined response = client._post('/v1/sys/wrapping/unwrap') 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/utils.py", line 170, in new_func 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return method(*args, **kwargs) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2577, in _post 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return self._adapter.post(*args, **kwargs) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 103, in post 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined return self.request('post', url, **kwargs) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 272, in request 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined utils.raise_for_error(response.status_code, text, errors=errors) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined File "/var/lib/juju/agents/unit-barbican-vault-8/.venv/lib/python3.6/site-packages/hvac/utils.py", line 30, in raise_for_error 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined raise exceptions.InvalidRequest(message, errors=errors) 2019-10-10 14:10:34 DEBUG secrets-storage-relation-joined hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist 2019-10-10 14:10:35 ERROR juju.worker.uniter.operation runhook.go:132 hook "secrets-storage-relation-joined" failed: exit status 1 on a deployment using vault, every time refresh-secrets is issued, the tokens are refreshed and the leader sends the new tokens through relation-data. If the vault is deployed in HA, upon switching vault leaders (let's say new leader is vault/2 and old leader is vault/0), the old token will remain in the relation data between the units that require secrets-storage (barbican-vault, ceph-osd, ...) and the old leader (vault/0). The new leader (vault/2) will issue new tokens on refresh-secrets action and provide them through relation (vault/2 <=> barbican-vault, ceph-osd), but the requiring units will read the old tokens from the relation-data of the old leader (vault/0 <=> barbican-vault, ceph-osd). Then, it causes the exception below. The tokens should be read from the new leader (vault/2) instead. As a workaround, if the leader is switched back to vault/0, the problem goes away until vault leader is changed again. Steps to reproduce: 1) Force vault leadership to lowest numbered unit (Vault/0) 2) Issue new tokens 3) Units will grab tokens from Vault/0 and everything will work fine 4) Force change vault leadership to a higher numbered unit (Vault/2) 5) Issue new tokens 6) Units will grab tokens from lowest value units (Vault/0) and will fail to authenticate This happens because on both reactive and classic charms, the related unit will loop through the vault units in ascending order and will grab the token from the first unit that has one. reactive charms: https://github.com/juju-solutions/charms.reactive/blob/1ff9c476693d834bc0fcc284ff1a89302d084c91/charms/reactive/endpoints.py#L748 classic charms: https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/openstack/vaultlocker.py#L44 Sample stacktrace: https://pastebin.ubuntu.com/p/65RHKKWj6q/
2019-10-23 20:25:42 Tiago Pasqualini da Silva charm-barbican-vault: assignee Rodrigo Barbieri (rodrigo-barbieri2010) Tiago Pasqualini da Silva (tiago.pasqualini)
2019-10-23 20:25:45 Tiago Pasqualini da Silva charm-helpers: status New In Progress
2019-11-04 15:44:20 James Page bug added subscriber Canonical Field High
2019-12-02 16:23:57 Edward Hope-Morley bug task added charm-ceph-osd
2019-12-02 16:24:08 Edward Hope-Morley bug task added charm-nova-compute
2019-12-02 16:25:17 Edward Hope-Morley bug task added charm-swift-proxy
2019-12-02 16:25:35 Edward Hope-Morley affects charm-swift-proxy charm-swift-storage
2019-12-02 16:26:17 Edward Hope-Morley charm-ceph-osd: milestone 20.01
2019-12-02 16:26:35 Edward Hope-Morley charm-nova-compute: milestone 20.01
2019-12-02 16:26:46 Edward Hope-Morley charm-swift-storage: milestone 20.01
2019-12-02 16:27:00 Edward Hope-Morley charm-ceph-osd: assignee Edward Hope-Morley (hopem)
2019-12-02 16:27:03 Edward Hope-Morley charm-swift-storage: assignee Edward Hope-Morley (hopem)
2019-12-02 16:27:06 Edward Hope-Morley charm-nova-compute: assignee Edward Hope-Morley (hopem)
2019-12-02 16:27:10 Edward Hope-Morley charm-barbican-vault: assignee Tiago Pasqualini da Silva (tiago.pasqualini) Edward Hope-Morley (hopem)
2019-12-02 16:27:17 Edward Hope-Morley charm-barbican-vault: status In Progress New
2019-12-06 15:23:15 Edward Hope-Morley charm-helpers: assignee Tiago Pasqualini da Silva (tiago.pasqualini) Edward Hope-Morley (hopem)
2019-12-06 16:10:44 Edward Hope-Morley charm-helpers: importance Undecided High
2019-12-06 16:10:46 Edward Hope-Morley charm-barbican-vault: importance Undecided High
2019-12-06 16:10:47 Edward Hope-Morley charm-ceph-osd: importance Undecided High
2019-12-06 16:10:48 Edward Hope-Morley charm-nova-compute: importance Undecided High
2019-12-06 16:10:50 Edward Hope-Morley charm-swift-storage: importance Undecided High
2019-12-06 16:31:43 OpenStack Infra charm-barbican-vault: status New In Progress
2019-12-10 13:36:00 Edward Hope-Morley charm-helpers: status In Progress Fix Committed
2019-12-10 13:38:08 OpenStack Infra charm-ceph-osd: status New In Progress
2019-12-10 13:39:16 OpenStack Infra charm-nova-compute: status New In Progress
2019-12-10 13:40:01 OpenStack Infra charm-swift-storage: status New In Progress
2019-12-14 12:17:23 OpenStack Infra charm-swift-storage: status In Progress Fix Committed
2019-12-14 12:21:54 OpenStack Infra charm-ceph-osd: status In Progress Fix Committed
2019-12-14 12:22:02 OpenStack Infra charm-nova-compute: status In Progress Fix Committed
2019-12-18 17:23:15 Rodrigo Barbieri charm-barbican-vault: status In Progress Fix Committed
2020-02-25 10:58:16 Edward Hope-Morley charm-ceph-osd: milestone 20.01 20.02
2020-02-25 10:58:20 Edward Hope-Morley charm-nova-compute: milestone 20.01 20.02
2020-02-25 10:58:23 Edward Hope-Morley charm-swift-storage: milestone 20.01 20.02
2020-02-25 10:58:29 Edward Hope-Morley charm-barbican-vault: status Fix Committed Fix Released
2020-02-25 10:58:30 Edward Hope-Morley charm-ceph-osd: status Fix Committed Fix Released
2020-02-25 10:58:32 Edward Hope-Morley charm-nova-compute: status Fix Committed Fix Released
2020-02-25 10:58:33 Edward Hope-Morley charm-swift-storage: status Fix Committed Fix Released
2020-05-19 12:49:32 James Page charm-helpers: status Fix Committed Fix Released
2021-01-26 09:54:23 Aurelien Lourot charm-barbican-vault: status Fix Released New
2021-01-26 09:54:27 Aurelien Lourot charm-barbican-vault: assignee Edward Hope-Morley (hopem)
2021-01-26 14:45:27 Aurelien Lourot charm-barbican-vault: status New Fix Released