Comment 5 for bug 1949913

The control-plane charms uses vault-kv in the following situations to store state encrypted in vault:

Diagnosis
----------

1) The Kubernetes-Control-Plane charm stores the "encryption_key" in a vault secret named charm-{app-name} at the path charm-{app-name}/kv/app. (Source from 1.27_release branch [1])

2) There are no uses of VaultUnitKV (individualized unit KV store)

Reactive Flag Events (per 1.27/stable release)
--------------

1) Each unit stores a hash at the path charm-{app-name}/kv/app-hashes/{unit-num} which is an md5 hexdigest of each key data in */kv/app/. Its purpose is to update reactive flags for other units in the event one unit changes the main /kv/app/* item, the other units can react to changes by watching for flags like
* layer.vault-kv.app-kv.changed
* layer.vault-kv.app-kv.changed.{key}
* layer.vault-kv.app-kv.set.{key}

2) Only leader units generate an encryption_key when vault relation is ready, and "layer.vault-kv.app-kv.set.encryption_key" is cleared.

3) Each unit writes the encryption_key to disk when vault relation is ready, "layer.vault-kv.app-kv.set.encryption_key" is set and "kubernetes-control-plane.secure-storage.created" is unset

4) Each unit clears "kubernetes-control-plane.secure-storage.created" if the vault relation is no longer ready and "kubernetes-control-plane.secure-storage.created" is currently set.

Assurances
----
* A charm-upgrade must confirm each unit maintains the correct encryption key located in /var/snap/kube-apiserver/common/encryption/encryption_config.yaml so that secrets can be unencrypted

* A charm-upgrade in one cluster must not disturb another cluster (upgraded or not)

* A charm-upgrade should store the encryption_key to a new secrets store named charm-{model-uuid}-{app-name} at /kv/app

* A charm-upgrade should NOT generate a new encryption_key when the key is unset in charm-{model-uuid}-{app-name}/kv/app, but is available in the encryption_config.yaml. Rather it should try to read from the yaml, and push this as the restored encryption_key value for this secret store.

Links
------------------
[1]: https://github.com/charmed-kubernetes/charm-kubernetes-control-plane/blob/release_1.27/reactive/kubernetes_control_plane.py#L3307