Comment 8 for bug 1940549

In addition to your suggested solutions @johnsca I think there's also an option to implement "leader storage" as a cache. Leader unit can use "leader-set" to store cert/keys when they are issued and when leadership changes, and new leader unit processes the requests from "clients", it first looks into cache if certificate was previously issued for that particular request, and if so, it will reuse it.

This approach is bit of a spin on your suggestion #3 except it does require synchronization between all the vault (provider) units via peer relations when new certificate is issued.

The current approach of re-issuing of certificates on leader change is unnecessary, as the root CA is shared between vault (provider) units and does not change with leadership change. Imo we should aim to remove the re-issuing as part of this fix.

Potential downside of using the leader storage is that all the secrets are stored in the controller database, but I checked with @aluria and he said that even the data from regular relations (provide-require) are stored on the controller, so it should not make much difference security-wise. (As the secrets are currently stored in the provide-require relationship data anyway)