I was hit by a very similiar bug. Reproducer bundle attached.
Charm version: cs:vault-44
After vault initialization, unsealing and charm authorisation I get following vault status:
Model Controller Cloud/Region Version SLA Timestamp
lma foundation-openstack openstack_cloud/us-east 2.8.10 unsupported 07:21:09Z
App Version Status Scale Charm Store Rev OS Notes
hacluster-vault active 3 hacluster jujucharms 74 ubuntu
vault 1.5.4 error 3 vault jujucharms 44 ubuntu
vault-mysql-router 8.0.23 active 3 mysql-router jujucharms 6 ubuntu
Unit Workload Agent Machine Public address Ports Message
vault/0* active idle 3 10.24.100.190 8200/tcp Unit is ready (active: true, mlock: enabled)
hacluster-vault/0* active idle 10.24.100.190 Unit is ready and clustered
vault-mysql-router/0* active idle 10.24.100.190 Unit is ready
vault/1 error idle 4 10.24.100.179 8200/tcp hook failed: "leader-settings-changed"
hacluster-vault/2 active idle 10.24.100.179 Unit is ready and clustered
vault-mysql-router/2 active idle 10.24.100.179 Unit is ready
vault/2 error idle 5 10.24.100.26 8200/tcp hook failed: "leader-settings-changed"
hacluster-vault/1 active idle 10.24.100.26 Unit is ready and clustered
vault-mysql-router/1 active idle 10.24.100.26 Unit is ready
Machine State DNS Inst id Series AZ Message
3 started 10.24.100.190 934b66b1-aa29-49ed-8f5b-12d31f9bef28 focal AZ1 ACTIVE
4 started 10.24.100.179 d1e0f50c-5e4f-402b-a0e2-17de98eec6d1 focal AZ2 ACTIVE
5 started 10.24.100.26 7c099246-4b3a-4ffc-a9a1-b52eaab24592 focal AZ3 ACTIVE
Relevant log section:
2021-04-28 07:08:50 WARNING leader-settings-changed All snaps up to date.
2021-04-28 07:08:50 INFO juju-log Initializing Leadership Layer (is follower)
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:253:configure_vault_mysql
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:294:mysql_setup
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:325:database_not_ready
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:415:cluster_connected
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/vault_handlers.py:598:prime_assess_status
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: hooks/relations/tls-certificates/provides.py:63:broken:certificates
2021-04-28 07:08:51 WARNING leader-settings-changed Traceback (most recent call last):
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/charm/hooks/leader-settings-changed", line 22, in <module>
2021-04-28 07:08:51 WARNING leader-settings-changed main()
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 84,
in main
2021-04-28 07:08:51 WARNING leader-settings-changed hookenv._run_atexit()
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/charmhelpers/core/hookenv.py", line 135
4, in _run_atexit
2021-04-28 07:08:51 WARNING leader-settings-changed callback(*args, **kwargs)
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/charm/reactive/vault_handlers.py", line 759, in _assess_status
2021-04-28 07:08:51 WARNING leader-settings-changed if not client_approle_authorized():
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/charm/reactive/vault_handlers.py", line 789, in client_approle_authorized2021-04-28 07:08:51 WARNING leader-settings-changed vault.get_local_client()
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 333, in wrapped_f
2021-04-28 07:08:51 WARNING leader-settings-changed return self(f, *args, **kw)
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 423, in __c
all__
2021-04-28 07:08:51 WARNING leader-settings-changed do = self.iter(retry_state=retry_state)
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 360, in iter
2021-04-28 07:08:51 WARNING leader-settings-changed return fut.result()
2021-04-28 07:08:51 WARNING leader-settings-changed File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
2021-04-28 07:08:51 WARNING leader-settings-changed return self.__get_result()
2021-04-28 07:08:51 WARNING leader-settings-changed File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
2021-04-28 07:08:51 WARNING leader-settings-changed raise self._exception
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __c
all__
2021-04-28 07:08:51 WARNING leader-settings-changed result = fn(*args, **kwargs)
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/charm/lib/charm/vault.py", line 254, in get_local_client
2021-04-28 07:08:51 WARNING leader-settings-changed client.auth_approle(app_role_id)
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 2072, in aut
h_approle
2021-04-28 07:08:51 WARNING leader-settings-changed return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token)
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1726, in auth
2021-04-28 07:08:51 WARNING leader-settings-changed return self._adapter.auth(
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 159, in auth
2021-04-28 07:08:51 WARNING leader-settings-changed response = self.post(url, **kwargs).json()
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 103, in post
2021-04-28 07:08:51 WARNING leader-settings-changed return self.request('post', url, **kwargs)
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 233, in request
2021-04-28 07:08:51 WARNING leader-settings-changed utils.raise_for_error(response.status_code, text, errors=errors)
2021-04-28 07:08:51 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.8/site-packages/hvac/utils.py", line 29, in raise_for_error
2021-04-28 07:08:51 WARNING leader-settings-changed raise exceptions.InvalidRequest(message, errors=errors)
2021-04-28 07:08:51 WARNING leader-settings-changed hvac.exceptions.InvalidRequest: missing client token
2021-04-28 07:08:52 ERROR juju.worker.uniter.operation runhook.go:139 hook "leader-settings-changed" (via explicit, bespoke hook script) failed: exit status 1
I was hit by a very similiar bug. Reproducer bundle attached.
Charm version: cs:vault-44
After vault initialization, unsealing and charm authorisation I get following vault status: openstack openstack_ cloud/us- east 2.8.10 unsupported 07:21:09Z
Model Controller Cloud/Region Version SLA Timestamp
lma foundation-
App Version Status Scale Charm Store Rev OS Notes
hacluster-vault active 3 hacluster jujucharms 74 ubuntu
vault 1.5.4 error 3 vault jujucharms 44 ubuntu
vault-mysql-router 8.0.23 active 3 mysql-router jujucharms 6 ubuntu
Unit Workload Agent Machine Public address Ports Message vault/0* active idle 10.24.100.190 Unit is ready and clustered mysql-router/ 0* active idle 10.24.100.190 Unit is ready settings- changed" mysql-router/ 2 active idle 10.24.100.179 Unit is ready settings- changed" mysql-router/ 1 active idle 10.24.100.26 Unit is ready
vault/0* active idle 3 10.24.100.190 8200/tcp Unit is ready (active: true, mlock: enabled)
hacluster-
vault-
vault/1 error idle 4 10.24.100.179 8200/tcp hook failed: "leader-
hacluster-vault/2 active idle 10.24.100.179 Unit is ready and clustered
vault-
vault/2 error idle 5 10.24.100.26 8200/tcp hook failed: "leader-
hacluster-vault/1 active idle 10.24.100.26 Unit is ready and clustered
vault-
Machine State DNS Inst id Series AZ Message aa29-49ed- 8f5b-12d31f9bef 28 focal AZ1 ACTIVE 5e4f-402b- a0e2-17de98eec6 d1 focal AZ2 ACTIVE 4b3a-4ffc- a9a1-b52eaab245 92 focal AZ3 ACTIVE
3 started 10.24.100.190 934b66b1-
4 started 10.24.100.179 d1e0f50c-
5 started 10.24.100.26 7c099246-
Relevant log section: settings- changed All snaps up to date. vault_handlers. py:253: configure_ vault_mysql vault_handlers. py:294: mysql_setup vault_handlers. py:325: database_ not_ready vault_handlers. py:415: cluster_ connected vault_handlers. py:598: prime_assess_ status /tls-certificat es/provides. py:63:broken: certificates settings- changed Traceback (most recent call last): settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/charm/ hooks/leader- settings- changed" , line 22, in <module> settings- changed main() settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ charms/ reactive/ __init_ _.py", line 84, settings- changed hookenv. _run_atexit( ) settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ charmhelpers/ core/hookenv. py", line 135 settings- changed callback(*args, **kwargs) settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/charm/ reactive/ vault_handlers. py", line 759, in _assess_status settings- changed if not client_ approle_ authorized( ): settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/charm/ reactive/ vault_handlers. py", line 789, in client_ approle_ authorized2021- 04-28 07:08:51 WARNING leader- settings- changed vault.get_ local_client( ) settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ tenacity/ __init_ _.py", line 333, in wrapped_f settings- changed return self(f, *args, **kw) settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ tenacity/ __init_ _.py", line 423, in __c settings- changed do = self.iter( retry_state= retry_state) settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ tenacity/ __init_ _.py", line 360, in iter settings- changed return fut.result() settings- changed File "/usr/lib/ python3. 8/concurrent/ futures/ _base.py" , line 432, in result settings- changed return self.__get_result() settings- changed File "/usr/lib/ python3. 8/concurrent/ futures/ _base.py" , line 388, in __get_result settings- changed raise self._exception settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ tenacity/ __init_ _.py", line 426, in __c settings- changed result = fn(*args, **kwargs) settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/charm/ lib/charm/ vault.py" , line 254, in get_local_client settings- changed client. auth_approle( app_role_ id) settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ hvac/v1/ __init_ _.py", line 2072, in aut settings- changed return self.auth( '/v1/auth/ {0}/login' .format( mount_point) , json=params, use_token= use_token) settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ hvac/v1/ __init_ _.py", line 1726, in auth settings- changed return self._adapter.auth( settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ hvac/adapters. py", line 159, in auth settings- changed response = self.post(url, **kwargs).json() settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ hvac/adapters. py", line 103, in post settings- changed return self.request( 'post', url, **kwargs) settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ hvac/adapters. py", line 233, in request settings- changed utils.raise_ for_error( response. status_ code, text, errors=errors) settings- changed File "/var/lib/ juju/agents/ unit-vault- 2/.venv/ lib/python3. 8/site- packages/ hvac/utils. py", line 29, in raise_for_error settings- changed raise exceptions. InvalidRequest( message, errors=errors) settings- changed hvac.exceptions .InvalidRequest : missing client token uniter. operation runhook.go:139 hook "leader- settings- changed" (via explicit, bespoke hook script) failed: exit status 1
2021-04-28 07:08:50 WARNING leader-
2021-04-28 07:08:50 INFO juju-log Initializing Leadership Layer (is follower)
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: reactive/
2021-04-28 07:08:51 INFO juju-log Invoking reactive handler: hooks/relations
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
in main
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
4, in _run_atexit
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
all__
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
all__
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
h_approle
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:51 WARNING leader-
2021-04-28 07:08:52 ERROR juju.worker.