Waiting for vault to be available after a reboot

[Marian Gasparovic]

After a reboot all units recovered except vault

vault/0* waiting idle 10.1.106.177 Waiting for vault to be available

Logs and artifacts - https://solutions.qa.canonical.com/testruns/2f74435c-6c5d-47ae-ab16-f4b601d55275

[Matt Verran (mv-2112)]

This looks to be the same issue I'm hitting on 2023.2/candidate (331).

Note that when we skip cert verification it does appear to be up and running

$ vault status -tls-skip-verify
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 1
Threshold 1
Unseal Progress 0/1
Unseal Nonce n/a
Version 1.15.2
Storage Type raft
HA Enabled true

[Matt Verran (mv-2112)]

As mitigation...

Get the juju secret id for vault, something like:-

juju show-unit vault/0 | yq ' .vault/0.relation-info[0].application-data.vault-initialization-secret-id '

Use that value to to query for the actual secret:

juju show-secret --reveal secret://d92fbccd-e7b6-43fe-8145-860bf3bdad19/cloqodhtpiev5hkf0p2g | yq ' .*.content.unsealkeys '

Extract the key from the output above and fire into the container:

kubectl exec -n openstack vault-0 -c vault -- vault operator unseal -tls-skip-verify 67e5bb1676ae564a67a7b1bddfc1f459c3b72db0765875f7270c662af87cee1d

Confirm its unsealed:

kubectl exec -n openstack vault-0 -c vault -- vault status -tls-skip-verify
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.15.3
Build Date n/a
Storage Type raft
Cluster Name vault-cluster-45284784
Cluster ID c0817a4e-d73c-2698-f2a4-7641f9afae71
HA Enabled true
HA Cluster https://10.1.191.173:8201
HA Mode active
Active Since 2023-12-08T11:59:40.958655679Z
Raft Committed Index 588
Raft Applied Index 588

Note: juju status vault/0 still shows it as 'Waiting for vault to be available'

[Matt Verran (mv-2112)]

Source to working out the above gleaned from https://github.com/canonical/vault-k8s-operator/blob/main/src/charm.py.

It should trigger unseal here https://github.com/canonical/vault-k8s-operator/blob/main/src/charm.py#L324.

[Matt Verran (mv-2112)]

I note the vault charm is latest/edge 44, current is 61 which does at least report active/idle in juju status when unsealed. Will confirm if this fixes reboot.

Possible futher mitigation, juju refresh vault

[Matt Verran (mv-2112)]

Can confirm, 2023.2/edge (339) with vault-k8s charm latest/edge 61 is fully 'green' in juju status post reboot.

[James Page (james-page)]

Thanks Matt

I'll get that revision updated in the tf plans for 2023.{1,2} - however we won't be able to squeeze in a release before christmas now.

[James Page (james-page)]

https://github.com/canonical/sunbeam-terraform/pull/45