hook failed: "certificates-relation-joined" due to InternalServerError

[Konstantinos Kaskavelis]

We have a failed test run where in kubernetes-aws, vault/0 is in an error state due to:

hook failed: "certificates-relation-joined"

From the logs, we get:

2022-07-19 18:43:54 ERROR unit.vault/0.juju-log server.go:319 certificates:82: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-vault-0/charm/reactive/vault_handlers.py", line 1030, in create_certs
    bundle = vault_pki.generate_certificate(cert_type,
  File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault_pki.py", line 126, in generate_certificate
    response = client.write('{}/issue/{}'.format(CHARM_PKI_MP, role),
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 189, in write
    response = self._adapter.post('/v1/{0}'.format(path), json=kwargs, wrap_ttl=wrap_ttl)
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 103, in post
    return self.request('post', url, **kwargs)
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 233, in request
    utils.raise_for_error(response.status_code, text, errors=errors)
  File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.8/site-packages/hvac/utils.py", line 39, in raise_for_error
    raise exceptions.InternalServerError(message, errors=errors)
hvac.exceptions.InternalServerError: 1 error occurred:
        * unable to store certificate locally: invalid connection

Test run:

https://solutions.qa.canonical.com/testruns/testRun/717b405e-d3c0-4349-b764-5c35ea4bea5d

Logs:

https://oil-jenkins.canonical.com/artifacts/717b405e-d3c0-4349-b764-5c35ea4bea5d/index.html

[Jeffrey Chang (modern911)]

This is another run on openstack, and vault/1 failed with same error.
https://solutions.qa.canonical.com/testruns/6706f1f1-daf2-4183-a226-86fc4830067f

Error logs, which repeat for 5 hr.
2024-04-26 12:24:12 DEBUG unit.vault/1.juju-log server.go:325 certificates:244: Opening vault port
2024-04-26 12:24:12 INFO unit.vault/1.juju-log server.go:325 certificates:244: Invoking reactive handler: reactive/vault_handlers.py:360:mysql_setup
2024-04-26 12:24:12 INFO unit.vault/1.juju-log server.go:325 certificates:244: Invoking reactive handler: reactive/vault_handlers.py:391:database_not_ready
2024-04-26 12:24:12 INFO unit.vault/1.juju-log server.go:325 certificates:244: Invoking reactive handler: reactive/vault_handlers.py:481:cluster_connected
2024-04-26 12:24:12 INFO unit.vault/1.juju-log server.go:325 certificates:244: Invoking reactive handler: reactive/vault_handlers.py:697:send_vault_url_and_ca
2024-04-26 12:24:12 WARNING unit.vault/1.juju-log server.go:325 certificates:244: Use of remote_binding in publish_url is deprecated. See LP Bug #1895185
2024-04-26 12:24:12 INFO unit.vault/1.juju-log server.go:325 certificates:244: Invoking reactive handler: reactive/vault_handlers.py:743:prime_assess_status
2024-04-26 12:24:12 INFO unit.vault/1.juju-log server.go:325 certificates:244: Invoking reactive handler: reactive/vault_handlers.py:1120:sync_cert_from_cache
2024-04-26 12:24:43 ERROR unit.vault/1.juju-log server.go:325 certificates:244: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
TimeoutError: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.10/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib...

[Jeffrey Chang (modern911)]

This happens more than 10 times since late June in SolQA tests and only on kubernetes on AWS.

[Alex Kavanagh (ajkavanagh)]

Unfortunately, you'll need to get the snap logs for the vault snap to find out what is going on. It looks like something is up with vault, and without those logs, it's a bit tricky to see what is happening.