Vault HA does not work as documented

[Cory Johns]

In lp:1833595 [1] we added instructions for using Vault in HA without EasyRSA using a manual step to transition from non-HA to HA. This was later changed [2] to just use an overlay, stating that testing had shown that the manual transition was no longer necessary. However, it seems that the situation may have regressed or there was a mistake in the original testing, because following the current instructions leads to the secondary Vault unit going into an errored state, as well as showing misleading info in the status (though I already proposed a fix for this [3]). It seems that our Vault CI test also does not test it in HA, so we didn't catch this.

Ideally, the Vault charm would not go into an errored state, but in the meantime, we may need to revert the overlay change to the docs.

[1]: https://bugs.launchpad.net/charm-kubernetes-master/+bug/1833595
[2]: https://github.com/charmed-kubernetes/kubernetes-docs/pull/285
[3]: https://review.opendev.org/#/c/667234/

[Cory Johns (johnsca)]

I also saw k8s-master go into an errored state when doing the manual HA transition due to a connection error which we could handle more gracefully.

[Cory Johns (johnsca)]

Doc revert: https://github.com/charmed-kubernetes/kubernetes-docs/pull/482

CI PR for single-step Vault HA: https://github.com/charmed-kubernetes/jenkins/pull/641 (will need to modify this in the meantime to do the multi-step, per the doc PR above)

[Cory Johns (johnsca)]

This updates the Jenkins test to actually use Vault in HA mode without EasyRSA, and works around the status issues in both the Vault and k8s-master charm: https://github.com/charmed-kubernetes/jenkins/pull/641

This fixes the hook error in the k8s-master charm during the Vault restart: https://github.com/juju-solutions/layer-vault-kv/pull/11

This fixes the libjuju unit.resolved() issue (which would be used in the workaround for the hook error in the k8s-master charm): https://github.com/juju/python-libjuju/pull/485

The status issues in the Vault charm were fixed in https://review.opendev.org/c/openstack/charm-vault/+/782585 and https://review.opendev.org/c/openstack/charm-vault/+/782585

[Cory Johns (johnsca)]

Also created https://review.opendev.org/c/openstack/charm-vault/+/784608 so that Vault will report when a manual restart is needed to pick up the HA config.