barbican-vault is stuck with waiting - 'secrets-storage' incomplete

[Nobuto Murata]

There is a similar bug report already but closed with Invalid so filing a new one.
https://bugs.launchpad.net/charm-barbican-vault/+bug/1842058
Also it's worth noting that the previous bug was focused on HA setup including vip and such, but this time it's a non HA deployment on a test bed. It's not a one-off issue but sometimes happens.

And with only one network space:
$ juju spaces
Name Space ID Subnets
alpha 0
space-first 1 192.168.151.0/24
undefined 2 10.0.9.0/24

cs:barbican-vault-27

Steps to reproduce:
1. juju deploy ./bundle.yaml
2. juju-wait -w --max_wait 4500 \
    --exclude vault \
    --exclude neutron-api-plugin-ovn \
    --exclude ovn-central \
    --exclude ovn-chassis \
    --exclude octavia \
    --exclude octavia-ovn-chassis \
    --exclude barbican-vault

3. juju run-action vault/leader --wait generate-root-ca

4. juju-wait -w --max_wait 1800 \
    --exclude octavia

Expected:
Step 4 completes without an error

Actual:
(Sometimes) Step 4 fails with timeout, and even after that barbican-vault is stuck indefinitely.

[Nobuto Murata (nobuto)]

$ juju run -u barbican-vault/0 -- charms.reactive get_flags
barbican-vault-installed
charm.installed
charm.is-subordinate
charms.openstack.do-default-charm.installed
charms.openstack.do-default-config.changed
charms.openstack.do-default-update-status
endpoint.secrets-storage.changed
endpoint.secrets-storage.changed.egress-subnets
endpoint.secrets-storage.changed.ingress-address
endpoint.secrets-storage.changed.private-address
endpoint.secrets-storage.changed.vault_url
endpoint.secrets-storage.joined
endpoint.secrets.changed
endpoint.secrets.changed.egress-subnets
endpoint.secrets.changed.ingress-address
endpoint.secrets.changed.private-address
endpoint.secrets.joined
secrets-storage.connected

$ juju run -u vault/0 -- charms.reactive get_flags
certificates.available
charm.vault.ca.ready
charm.vault.global-client-cert.created
configured
endpoint.certificates.changed
endpoint.certificates.changed.cert_requests
endpoint.certificates.changed.certificate_name
endpoint.certificates.changed.common_name
endpoint.certificates.changed.egress-subnets
endpoint.certificates.changed.ingress-address
endpoint.certificates.changed.private-address
endpoint.certificates.changed.sans
endpoint.certificates.changed.unit_name
endpoint.certificates.joined
endpoint.secrets.changed
endpoint.secrets.changed.egress-subnets
endpoint.secrets.changed.ingress-address
endpoint.secrets.changed.private-address
endpoint.secrets.changed.unit_name
endpoint.secrets.joined
leadership.is_leader
leadership.set.keys
leadership.set.local-charm-access-id
leadership.set.root-ca
leadership.set.root_token
pki.backend.tuned
secrets.connected
shared-db.available
shared-db.connected
snap.installed.core
snap.installed.vault
snap.refresh.set
started
vault.ssl.configured

[Nobuto Murata (nobuto)]

$ juju run -u barbican-vault/0 -- relation-get -r secrets-storage:94 - vault/0
egress-subnets: 192.168.151.120/32
ingress-address: 192.168.151.120
private-address: 192.168.151.120
vault_url: '"http://192.168.151.120:8200"'

[Nobuto Murata (nobuto)]

$ juju run -u vault/0 -- relation-get -r secrets:94 - barbican-vault/0
access_address: '"192.168.151.121"'
egress-subnets: 192.168.151.121/32
hostname: '"juju-5b4300-1-lxd-0"'
ingress-address: 192.168.151.121
isolated: "false"
private-address: 192.168.151.121
secret_backend: '"charm-barbican-vault"'
unit_name: '"barbican-vault/0"'

[Nobuto Murata (nobuto)]

Hmm, it might be on the Vault charm side such as:
https://bugs.launchpad.net/vault-charm/+bug/1874059
https://bugs.launchpad.net/vault-charm/+bug/1870910