kubernetes-control-plane in error state due to hook failed: "kube-control-relation-joined"

[Konstantinos Kaskavelis]

We have a test run where kubernetes-control-plane in aws is left in error state due to a hook failing with the message: "kube-control-relation-joined"

From the logs, the traceback we are getting is:

2022-07-08 09:58:14 DEBUG unit.kubernetes-control-plane/0.juju-log server.go:319 kube-control:65: vault-kv.log: Logging VaultAppKV in to http://172.31.37.51:8200
2022-07-08 09:58:15 ERROR unit.kubernetes-control-plane/0.juju-log server.go:319 kube-control:65: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 73, in main
    hookenv._run_atstart()
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charmhelpers/core/hookenv.py", line 1348, in _run_atstart
    callback(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/reactive/vault_kv.py", line 46, in manage_app_kv_flags
    app_kv = vault_kv.VaultAppKV()
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 33, in __call__
    cls._singleton_instance = super().__call__(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 131, in __init__
    super().__init__()
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 41, in __init__
    response = self._client.read(self._path)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 60, in _client
    client.auth_approle(self._config["role_id"], self._config["secret_id"])
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/utils.py", line 201, in new_func
    return method(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1805, in auth_approle
    return self.login(
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1495, in login
    return self._adapter.login(url=url, use_token=use_token, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 197, in login
    response = self.post(url, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 126, in post
    return self.request("post", url, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 364, in request
    response = super(JSONAdapter, self).request(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 330, in request
    utils.raise_for_error(
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/utils.py", line 49, in raise_for_error
    raise exceptions.InternalServerError(
hvac.exceptions.InternalServerError: internal error, on post http://172.31.37.51:8200/v1/auth/approle/login

Vault is also in an error state with the message: hook failed: "secrets-relation-changed"

Test run:

https://solutions.qa.canonical.com/testruns/testRun/fdc0abff-c18c-453b-8bb8-e837600e5c84

[Bas de Bruijne (basdbruijne)]

I see the same error in this testrun: https://solutions.qa.canonical.com/testruns/testRun/5f59b3db-eb2f-4887-bdd3-8c073facb064. Except the message is 'hook failed: "certificates-relation-changed"'.

Crashdumps for this run are here:
https://oil-jenkins.canonical.com/artifacts/5f59b3db-eb2f-4887-bdd3-8c073facb064/generated/generated/kubernetes-aws/juju-crashdump-kubernetes-aws-2022-09-11-03.22.27.tar.gz