Kubernetes-control-plane is stuck with `hook failed: "vault-kv-relation-changed"` and the logs show as follows:
unit-kubernetes-control-plane-0: 14:45:20 INFO unit.kubernetes-control-plane/0.juju-log vault-kv:75: Invoking reactive handler: reactive/kubernetes_control_plane.py:3227:generate_encryption_key
unit-kubernetes-control-plane-0: 14:45:20 DEBUG unit.kubernetes-control-plane/0.juju-log vault-kv:75: vault-kv.log: Logging VaultAppKV in to http://172.31.43.113:8200
unit-kubernetes-control-plane-0: 14:45:20 ERROR unit.kubernetes-control-plane/0.juju-log vault-kv:75: Hook error:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
bus.dispatch(restricted=restricted_mode)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
_invoke(other_handlers)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
handler.invoke()
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
self._action(*args)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/reactive/kubernetes_control_plane.py", line 3230, in generate_encryption_key
app_kv = vault_kv.VaultAppKV()
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 33, in __call__
cls._singleton_instance = super().__call__(*args, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 132, in __init__
super().__init__()
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 41, in __init__
response = self._client.read(self._path)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 60, in _client
client.auth_approle(self._config["role_id"], self._config["secret_id"])
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/utils.py", line 201, in new_func
return method(*args, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1805, in auth_approle
return self.login(
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1495, in login
return self._adapter.login(url=url, use_token=use_token, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 197, in login
response = self.post(url, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 126, in post
return self.request("post", url, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 364, in request
response = super(JSONAdapter, self).request(*args, **kwargs)
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 330, in request
utils.raise_for_error(
File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/utils.py", line 37, in raise_for_error
raise exceptions.InvalidRequest(message, errors=errors, method=method, url=url)
hvac.exceptions.InvalidRequest: missing secret_id, on post http://172.31.43.113:8200/v1/auth/approle/login
On run: https://solutions.qa.canonical.com/v2/testruns/0359a572-3c92-4092-8866-081216cbbcb0/
Artifacts: https://oil-jenkins.canonical.com/artifacts/0359a572-3c92-4092-8866-081216cbbcb0/index.html
I see. If _get_secret_id ever raises VaultNotReady[1], then the next time it is called, the data_changed condition[2] will evaluate to False because data_changed has been called with that token before. This prevents the token from being processed.
Instead, _get_secret_id returns a cached secret ID from unitdata[3]. In this case, the cached secret ID is None because no secret ID has ever been set.
Recommended fix: replace that data_changed with a smarter condition that only advances *after* a secret ID has been successfully obtained with the token.
[1]: https:/ /github. com/charmed- kubernetes/ layer-vault- kv/blob/ 13d3ae371259ae4 b5a14c72f03f66e 2f8e83f001/ lib/charms/ layer/vault_ kv.py#L266 /github. com/charmed- kubernetes/ layer-vault- kv/blob/ 13d3ae371259ae4 b5a14c72f03f66e 2f8e83f001/ lib/charms/ layer/vault_ kv.py#L251 /github. com/charmed- kubernetes/ layer-vault- kv/blob/ 13d3ae371259ae4 b5a14c72f03f66e 2f8e83f001/ lib/charms/ layer/vault_ kv.py#L274
[2]: https:/
[3]: https:/