After unsealing the vault units; all 3 units went into an error during the certificates-relation-joined hook.
They timed out connecting to 127.0.0.1 on port 8220
2021-04-12 12:27:40 ERROR juju-log certificates:220: Hook error:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
six.raise_from(e, None)
File "<string>", line 3, in raise_from
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/usr/lib/python3.8/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/urllib3/packages/six.py", line 735, in reraise
raise value
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 428, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 335, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=8220): Read timed out. (read timeout=30)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-vault-0/charm/reactive/vault_handlers.py", line 845, in publish_ca_info
if not client_approle_authorized():
File "/var/lib/juju/agents/unit-vault-0/charm/reactive/vault_handlers.py", line 788, in client_approle_authorized
vault.get_local_client()
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 329, in wrapped_f
return self.call(f, *args, **kw)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 409, in call
do = self.iter(retry_state=retry_state)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 356, in iter
return fut.result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
raise self._exception
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/tenacity/__init__.py", line 412, in call
result = fn(*args, **kwargs)
File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 254, in get_local_client
client.auth_approle(app_role_id)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 2072, in auth_approle
return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1726, in auth
return self._adapter.auth(
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 159, in auth
response = self.post(url, **kwargs).json()
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 103, in post
return self.request('post', url, **kwargs)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 218, in request
response = self.session.request(method, url, headers=headers,
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=8220): Read timed out. (read timeout=30)
the crashdump can be found at https://oil-jenkins.canonical.com/artifacts/2d1c39ab-1d4a-46f5-901c-a4ad5a46bef4/generated/generated/openstack/juju-crashdump-openstack-2021-04-12-12.30.49.tar.gz
and this log was taken from vault_0; however all 3 units have a similar Traceback
Closing this due to inactivity (low number of occurrences, and no hit for more than one year)