Comment 3 for bug 1888900

This appears to be caused by a combination of how vault is bootstrapping tls and how cert-manager is currently configured to manage the k8s secrets it creates. I'll provide a bit of background on how vault TLS is set up and propose a solution.

Vault currently bootstraps its TLS by generating a key pair during the helm template stage, and then using that key pair to create a cert-manager issuer resource. See:
kubectl get issuers.cert-manager.io -n vault

Vault then provisions a certificate from that cert-manager issuer, which creates a certificate resource and a secret containing the certificate data. This secret is what is consumed by the vault components to run TLS.
[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get certificate -n vault
NAME READY SECRET AGE
vault-server-tls True vault-server-tls 39m

[sysadmin@controller-0 ~(keystone_admin)]$ kubectl get secrets -n vault vault-server-tls
NAME TYPE DATA AGE
vault-server-tls kubernetes.io/tls 3 39m

When vault is deleted with system application-remove vault, the issuer and certificate resources are deleted, but the secret is left behind because it is not owned by the certificate resource. This is intended behaviour for how cert-manager is configured. This orphaned secret does not get updated on a subsequent reapply and is then invalid when vault attempts to use it.

Cert-manager can be configured to clean up secrets when the corresponding certificate resource is removed by enabling this option via its helm chart:
extraArgs: []
  # When this flag is enabled, secrets will be automatically removed when the certificate resource is deleted
  # - --enable-certificate-owner-ref=true

This can be tested using an override like this:
[sysadmin@controller-0 ~(keystone_admin)]$ cat cm.yaml
extraArgs:
  - --enable-certificate-owner-ref=true

system helm-override-update --values cm.yaml cert-manager cert-manager cert-manager
system application-apply cert-manager

With this enabled, vault can be removed and reapplied properly.

The alternative to this is to manually delete the orphaned secret before applying vault.