[vault-kv-relationship-changed] InvalidRequest: missing secret_id

[Moises Emilio Benzan Mora]

Kubernetes-control-plane is stuck with `hook failed: "vault-kv-relation-changed"` and the logs show as follows:

unit-kubernetes-control-plane-0: 14:45:20 INFO unit.kubernetes-control-plane/0.juju-log vault-kv:75: Invoking reactive handler: reactive/kubernetes_control_plane.py:3227:generate_encryption_key
unit-kubernetes-control-plane-0: 14:45:20 DEBUG unit.kubernetes-control-plane/0.juju-log vault-kv:75: vault-kv.log: Logging VaultAppKV in to http://172.31.43.113:8200
unit-kubernetes-control-plane-0: 14:45:20 ERROR unit.kubernetes-control-plane/0.juju-log vault-kv:75: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/reactive/kubernetes_control_plane.py", line 3230, in generate_encryption_key
    app_kv = vault_kv.VaultAppKV()
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 33, in __call__
    cls._singleton_instance = super().__call__(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 132, in __init__
    super().__init__()
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 41, in __init__
    response = self._client.read(self._path)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/charm/lib/charms/layer/vault_kv.py", line 60, in _client
    client.auth_approle(self._config["role_id"], self._config["secret_id"])
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/utils.py", line 201, in new_func
    return method(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1805, in auth_approle
    return self.login(
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1495, in login
    return self._adapter.login(url=url, use_token=use_token, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 197, in login
    response = self.post(url, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 126, in post
    return self.request("post", url, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 364, in request
    response = super(JSONAdapter, self).request(*args, **kwargs)
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 330, in request
    utils.raise_for_error(
  File "/var/lib/juju/agents/unit-kubernetes-control-plane-0/.venv/lib/python3.8/site-packages/hvac/utils.py", line 37, in raise_for_error
    raise exceptions.InvalidRequest(message, errors=errors, method=method, url=url)
hvac.exceptions.InvalidRequest: missing secret_id, on post http://172.31.43.113:8200/v1/auth/approle/login

On run: https://solutions.qa.canonical.com/v2/testruns/0359a572-3c92-4092-8866-081216cbbcb0/
Artifacts: https://oil-jenkins.canonical.com/artifacts/0359a572-3c92-4092-8866-081216cbbcb0/index.html

[George Kraft (cynerva)]

I see. If _get_secret_id ever raises VaultNotReady[1], then the next time it is called, the data_changed condition[2] will evaluate to False because data_changed has been called with that token before. This prevents the token from being processed.

Instead, _get_secret_id returns a cached secret ID from unitdata[3]. In this case, the cached secret ID is None because no secret ID has ever been set.

Recommended fix: replace that data_changed with a smarter condition that only advances *after* a secret ID has been successfully obtained with the token.

[1]: https://github.com/charmed-kubernetes/layer-vault-kv/blob/13d3ae371259ae4b5a14c72f03f66e2f8e83f001/lib/charms/layer/vault_kv.py#L266
[2]: https://github.com/charmed-kubernetes/layer-vault-kv/blob/13d3ae371259ae4b5a14c72f03f66e2f8e83f001/lib/charms/layer/vault_kv.py#L251
[3]: https://github.com/charmed-kubernetes/layer-vault-kv/blob/13d3ae371259ae4b5a14c72f03f66e2f8e83f001/lib/charms/layer/vault_kv.py#L274

[Adam Dyess (addyess)]

Proposed Fix
https://github.com/juju-solutions/layer-vault-kv/pull/19

[Adam Dyess (addyess)]

Backport of layer-vault-kv is here:
https://github.com/charmed-kubernetes/layer-vault-kv/commit/27b38232674136c084fd4e1e8f86b3abbf2a2a43