hook failed: "vault-kv-relation-changed"

[Konstantinos Kaskavelis]

We have a test run (SKU: fkb-master-kubernetes-focal-keystone-aws) failing with the following error:

ERROR:root:kubernetes-master/1 failed: workload status is error

and the message for the Error status of the kubernetes master is:

hook failed: "vault-kv-relation-changed"

Test run:
https://solutions.qa.canonical.com/testruns/testRun/7c56304e-6065-4e45-9540-2c28b7bc79cd

Artifacts:
https://oil-jenkins.canonical.com/artifacts/7c56304e-6065-4e45-9540-2c28b7bc79cd/index.html

[Moises Emilio Benzan Mora (moisesbenzan)]

unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed Created symlink /etc/systemd/system/[e-mail]/[e-mail] → /etc/systemd/system/vaultlocker-loop@.service.
unit-kubernetes-control-plane-1: 09:05:11 DEBUG unit.kubernetes-control-plane/1.juju-log vault-kv:59: Writing file /var/snap/kube-apiserver/common/encryption/encryption_config.yaml root:root 600
unit-kubernetes-control-plane-1: 09:05:11 DEBUG unit.kubernetes-control-plane/1.juju-log vault-kv:59: tracer: cleared flag layer.vault-kv.app-kv.changed.encryption_key
unit-kubernetes-control-plane-1: 09:05:11 DEBUG unit.kubernetes-control-plane/1.juju-log vault-kv:59: tracer>
tracer: set flag kubernetes-control-plane.secure-storage.created
tracer: -- dequeue handler reactive/kubernetes_control_plane.py:3270:create_secure_storage
unit-kubernetes-control-plane-1: 09:05:11 DEBUG unit.kubernetes-control-plane/1.juju-log vault-kv:59: vault-kv.log: Writing hashes to charm-e34c99d9-06c4-445c-8f56-3d5547baaff2-kubernetes-control-plane/kv/app-hashes/1
unit-kubernetes-control-plane-1: 09:05:11 DEBUG unit.kubernetes-control-plane/1.juju-log vault-kv:59: vault-kv.log: Logging VaultAppKV in to http://172.31.36.61:8200
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed Traceback (most recent call last):
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-control-plane-1/charm/hooks/vault-kv-relation-changed", line 22, in <module>
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed main()
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-control-plane-1/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 84, in main
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed hookenv._run_atexit()
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-control-plane-1/.venv/lib/python3.10/site-packages/charmhelpers/core/hookenv.py", line 1357, in _run_atexit
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed callback(*args, **kwargs)
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-control-plane-1/charm/reactive/vault_kv.py", line 71, in update_app_kv_hashes
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed app_kv.update_hashes()
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-control-plane-1/charm/lib/charms/layer/vault_kv.py", line 224, ...

[Moises Emilio Benzan Mora (moisesbenzan)]

Latest occurrence: https://solutions.qa.canonical.com/testruns/1e2f8642-198c-4119-8220-19586c0810cd
Artifacts: https://oil-jenkins.canonical.com/artifacts/1e2f8642-198c-4119-8220-19586c0810cd/index.html

Future occurrences: https://solutions.qa.canonical.com/bugs/1975500

[Alex Kavanagh (ajkavanagh)]

The cause is here:

unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed hvac.exceptions.InternalServerError: 1 error occurred:
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed * failed to write: dial tcp 127.0.0.1:3306: connect: connection refused
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed
unit-kubernetes-control-plane-1: 09:05:11 WARNING unit.kubernetes-control-plane/1.vault-kv-relation-changed , on post http://172.31.36.61:8200/v1/charm-e34c99d9-06c4-445c-8f56-3d5547baaff2-kubernetes-control-plane/kv/app-hashes/1
unit-kubernetes-control-plane-1: 09:05:11 DEBUG juju.worker.uniter.remotestate got leader settings change for kubernetes-control-plane/1: ok=true

i.e. the local mysql router (…:3306) is not available.

Whilst there is an issue with robustness in the vault charm (e.g. it error's out), this (almost certainly) would recover if the hook was retried when the local mysql-router instance is available.

It would be very useful if you could dig into this and work out what was happening with the mysql-router at the time of the hook error in vault.