Hello,
I've been trying to set up a vault cluster, and have had troubles getting it up and running per documentation. I've run through things several times but was not able to get things to work without taking extra steps.
Specifically: the documented steps seem to work perfectly up until I get to the authorize-charm step. However, after I run the authorize-charm action, only the leader goes to a green state. The remaining two units hit an error and report the message: 'hook failed: "leader-settings-changed"'
The traceback encountered by the errored units looks like this:
2021-04-08 15:18:42 INFO juju-log Invoking reactive handler: hooks/relations/tls-certificates/provides.py:63:broken:certificates
2021-04-08 15:18:42 WARNING leader-settings-changed Traceback (most recent call last):
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/charm/hooks/leader-settings-changed", line 22, in <module>
2021-04-08 15:18:42 WARNING leader-settings-changed main()
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 84, in main
2021-04-08 15:18:42 WARNING leader-settings-changed hookenv._run_atexit()
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/charmhelpers/core/hookenv.py", line 1354, in _run_atexit
2021-04-08 15:18:42 WARNING leader-settings-changed callback(*args, **kwargs)
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/charm/reactive/vault_handlers.py", line 759, in _assess_status
2021-04-08 15:18:42 WARNING leader-settings-changed if not client_approle_authorized():
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/charm/reactive/vault_handlers.py", line 789, in client_approle_authorized
2021-04-08 15:18:42 WARNING leader-settings-changed vault.get_local_client()
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/tenacity/__init__.py", line 333, in wrapped_f
2021-04-08 15:18:42 WARNING leader-settings-changed return self(f, *args, **kw)
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/tenacity/__init__.py", line 423, in __call__
2021-04-08 15:18:42 WARNING leader-settings-changed do = self.iter(retry_state=retry_state)
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/tenacity/__init__.py", line 360, in iter
2021-04-08 15:18:42 WARNING leader-settings-changed return fut.result()
2021-04-08 15:18:42 WARNING leader-settings-changed File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
2021-04-08 15:18:42 WARNING leader-settings-changed return self.__get_result()
2021-04-08 15:18:42 WARNING leader-settings-changed File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-04-08 15:18:42 WARNING leader-settings-changed raise self._exception
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/tenacity/__init__.py", line 426, in __call__
2021-04-08 15:18:42 WARNING leader-settings-changed result = fn(*args, **kwargs)
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/charm/lib/charm/vault.py", line 254, in get_local_client
2021-04-08 15:18:42 WARNING leader-settings-changed client.auth_approle(app_role_id)
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2072, in auth_approle
2021-04-08 15:18:42 WARNING leader-settings-changed return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token)
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 1729, in auth
2021-04-08 15:18:42 WARNING leader-settings-changed **kwargs
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 159, in auth
2021-04-08 15:18:42 WARNING leader-settings-changed response = self.post(url, **kwargs).json()
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 103, in post
2021-04-08 15:18:42 WARNING leader-settings-changed return self.request('post', url, **kwargs)
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 233, in request
2021-04-08 15:18:42 WARNING leader-settings-changed utils.raise_for_error(response.status_code, text, errors=errors)
2021-04-08 15:18:42 WARNING leader-settings-changed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.6/site-packages/hvac/utils.py", line 29, in raise_for_error
2021-04-08 15:18:42 WARNING leader-settings-changed raise exceptions.InvalidRequest(message, errors=errors)
2021-04-08 15:18:42 WARNING leader-settings-changed hvac.exceptions.InvalidRequest: missing client token
2021-04-08 15:18:42 ERROR juju.worker.uniter.operation runhook.go:136 hook "leader-settings-changed" (via explicit, bespoke hook script) failed: exit status 1
I did find a workaround: if I restart vault on the 2 errored units, then unseal vault again, I'm then able to "juju resolved" the errored units and everything will go green.
This was tested with cs:vault-44. I'm also attaching the Juju bundle used to deploy the environment.
I hit this today but vault/0 only errored, vault/1 was ok and vault/2(leader) also OK. Restarted vault on vault/0, unsealed it, then it was fine.