secrets-storage relation stuck waiting

[Edward Hope-Morley]

A freshly deployed nova-compute+vault is remaining stuck in waiting status saying there are no remote units on the secrets-storage relation despite that not being the case:

$ juju status nova-compute --relations
Model Controller Cloud/Region Version SLA Timestamp
octavia-ovn-testing hopem stsstack/stsstack 2.7.0 unsupported 12:15:37Z

App Version Status Scale Charm Store Rev OS Notes
nova-compute 20.0.1 waiting 1 nova-compute jujucharms 489 ubuntu
ovn-chassis 2.12.0 active 1 ovn-chassis jujucharms 23 ubuntu

Unit Workload Agent Machine Public address Ports Message
nova-compute/0* waiting idle 6 10.5.0.86 Incomplete relations: vault
  ovn-chassis/0* active idle 10.5.0.86 Unit is ready

Machine State DNS Inst id Series AZ Message
6 started 10.5.0.86 395c174e-ca73-41f7-99bd-f9ef15a6c1fd bionic nova ACTIVE

Relation provider Requirer Interface Type Message
glance:image-service nova-compute:image-service glance regular
nova-compute:cloud-compute nova-cloud-controller:cloud-compute nova-compute regular
nova-compute:compute-peer nova-compute:compute-peer nova peer
ovn-central:ovsdb ovn-chassis:ovsdb ovsdb regular
ovn-chassis:nova-compute nova-compute:neutron-plugin neutron-plugin subordinate
ovn-chassis:ovsdb-subordinate octavia:ovsdb-subordinate ovsdb-subordinate subordinate
rabbitmq-server:amqp nova-compute:amqp rabbitmq regular
vault:certificates ovn-chassis:certificates tls-certificates regular
vault:secrets nova-compute:secrets-storage vault-kv regular
$ juju run -a nova-compute -- relation-ids secrets-storage
secrets-storage:51
$ juju run -a nova-compute -- relation-list -r secrets-storage:51
vault/0
$ juju run -a nova-compute -- relation-get -r secrets-storage:51 - vault/0
egress-subnets: 10.5.0.8/32
ingress-address: 10.5.0.8
private-address: 10.5.0.8
vault_url: '"http://10.5.0.8:8200"'

And in the logs I have:

# tail /var/log/juju/unit-nova-compute-0.log
2020-04-05 12:11:52 INFO juju-log Updating status.
2020-04-05 12:11:54 INFO juju-log Vault dependencies met so including in status check
2020-04-05 12:11:55 DEBUG juju-log Generating template context for amqp
2020-04-05 12:11:55 DEBUG update-status ERROR no relation id specified
2020-04-05 12:11:55 DEBUG juju-log Generating template context for image-service.
2020-04-05 12:11:57 INFO juju-log Generated config context for neutron network manager.
2020-04-05 12:11:57 DEBUG juju-log 0 section(s) found
2020-04-05 12:11:58 DEBUG update-status none
2020-04-05 12:11:58 DEBUG juju-log Generating template context for cloud-credentials
2020-04-05 12:11:58 INFO juju-log vault relations's interface, secrets-storage, is related but has no units in the relation.

[Edward Hope-Morley (hopem)]

Further analsyis shows that the message "vault relations's interface, secrets-storage, is related but has no units in the relation." is misleading and is being show as a result of charm-helpers.contrib.openstack.vaultlocker.VaultKVContext().complete being false due to the fact that the related vault charm has not provided a token. I assume this a race issue on the vault charm side since a token should have been provided when vault was unsealed.

[Edward Hope-Morley (hopem)]

removing then adding the relation again fixed the problem i.e.

$ juju run -a nova-compute -- relation-get -r secrets-storage:59 - vault/0
egress-subnets: 10.5.0.8/32
ingress-address: 10.5.0.8
nova-compute/0_role_id: '"e1d47084-579a-afe1-326e-ec643b5a1b87"'
nova-compute/0_token: '"s.B3Ynbrb2hz1omo5BPQUH2oAF"'
private-address: 10.5.0.8
vault_url: '"http://10.5.0.8:8200"'

[Cory Johns (johnsca)]

This seems likely related to https://bugs.launchpad.net/vault-charm/+bug/1874059