Mojo specs which use the designate-next-ha.yaml will deploy multiple units of vault but not configure an HA manager like etcd. This results in all units thinking they are the master. I *think* this may be the cause of some mojo run failures where a vault hook fails due to an internal server error talking to the vault service which was inturn cause by deadlocks in the DB:
2019-10-15 16:56:05 DEBUG juju-log certificates:106: Rendering vault systemd configuation
2019-10-15 16:56:05 DEBUG juju-log certificates:106: Changing permissions on existing content: 33188 -> 420
2019-10-15 16:56:05 DEBUG juju-log certificates:106: Opening vault port
2019-10-15 16:56:06 INFO juju-log certificates:106: Invoking reactive handler: reactive/vault_handlers.py:733:publish_ca_info
2019-10-15 16:56:26 ERROR juju-log certificates:106: Hook error:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/__init__.py", line 74, in main bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-vault-0/charm/reactive/vault_handlers.py", line 739, in publish_ca_info
chain = vault_pki.get_chain()
File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault_pki.py", line 74, in get_chain
client = vault.get_local_client()
File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 249, in get_local_client
client.auth_approle(app_role_id)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/v1/__init__.py", line 2072, in auth_approle
return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/v1/__init__.py", line 1729, in auth
**kwargs
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/adapters.py", line 159, in auth
response = self.post(url, **kwargs).json()
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/adapters.py", line 103, in post
return self.request('post', url, **kwargs)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/adapters.py", line 233, in request
utils.raise_for_error(response.status_code, text, errors=errors)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/utils.py", line 39, in raise_for_error
raise exceptions.InternalServerError(message, errors=errors)
hvac.exceptions.InternalServerError: internal error
2019-10-15 16:56:26 DEBUG certificates-relation-changed Traceback (most recent call last):
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/charm/hooks/certificates-relation-changed", line 22, in <module>
2019-10-15 16:56:26 DEBUG certificates-relation-changed main()
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/__init__.py", line 74, in main
2019-10-15 16:56:26 DEBUG certificates-relation-changed bus.dispatch(restricted=restricted_mode)
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 390, in dispatch
2019-10-15 16:56:26 DEBUG certificates-relation-changed _invoke(other_handlers)
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 359, in _invoke
2019-10-15 16:56:26 DEBUG certificates-relation-changed handler.invoke()
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 181, in invoke
2019-10-15 16:56:26 DEBUG certificates-relation-changed self._action(*args)
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/charm/reactive/vault_handlers.py", line 739, in publish_ca_info
2019-10-15 16:56:26 DEBUG certificates-relation-changed chain = vault_pki.get_chain()
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault_pki.py", line 74, in get_chain
2019-10-15 16:56:26 DEBUG certificates-relation-changed client = vault.get_local_client()
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 249, in get_local_client
2019-10-15 16:56:26 DEBUG certificates-relation-changed client.auth_approle(app_role_id)
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/v1/__init__.py", line 2072, in auth_approle
2019-10-15 16:56:26 DEBUG certificates-relation-changed return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token)
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/v1/__init__.py", line 1729, in auth
2019-10-15 16:56:26 DEBUG certificates-relation-changed **kwargs
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/adapters.py", line 159, in auth
2019-10-15 16:56:26 DEBUG certificates-relation-changed response = self.post(url, **kwargs).json()
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/adapters.py", line 103, in post
2019-10-15 16:56:26 DEBUG certificates-relation-changed return self.request('post', url, **kwargs)
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/adapters.py", line 233, in request
2019-10-15 16:56:26 DEBUG certificates-relation-changed utils.raise_for_error(response.status_code, text, errors=errors)
2019-10-15 16:56:26 DEBUG certificates-relation-changed File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.5/site-packages/hvac/utils.py", line 39, in raise_for_error
2019-10-15 16:56:26 DEBUG certificates-relation-changed raise exceptions.InternalServerError(message, errors=errors)
2019-10-15 16:56:26 DEBUG certificates-relation-changed hvac.exceptions.InternalServerError: internal error
2019-10-15 16:56:26 ERROR juju.worker.uniter.operation runhook.go:132 hook "certificates-relation-changed" failed: exit status 1
