Comment 0 for bug 1970888

Revision history for this message
Liam Young (gnuoy) wrote :

If the certificates-relation-joined hook runs before vault has been configured
then the hook fails. This is because the charm tries to access the running
vault service but at this point vault is not configured or running. This
regression appears to have been introduced by 1159e547
( https://review.opendev.org/c/openstack/charm-vault/+/828885 ). This patch
seems to incorrectly gate on the `certificates.available` flag. Despite the
name, `certificates.available` only indicates that certificate have
been requested i.e. it means "a certificate is available to be processed" it
does not mean that vault is ready.

The issue can be reproduced with this bundle:

series: focal
applications:

  keystone-mysql-router:
    charm: ch:mysql-router
    channel: latest/edge
  vault-mysql-router:
    charm: ch:mysql-router
    channel: latest/edge

  mysql-innodb-cluster:
    charm: ch:mysql-innodb-cluster
    constraints: mem=3072M
    num_units: 3
    channel: latest/edge

  vault:
    num_units: 3
    charm: ch:vault
    channel: latest/edge

  keystone:
    charm: ch:keystone
    num_units: 1
    options:
      admin-password: openstack
    channel: latest/edge

relations:
  - - 'vault:shared-db'
    - 'vault-mysql-router:shared-db'

  - - 'keystone:shared-db'
    - 'keystone-mysql-router:shared-db'
  - - 'keystone-mysql-router:db-router'
    - 'mysql-innodb-cluster:db-router'

  - - 'vault:certificates'
    - 'keystone:certificates'

Note that in the bundle the relation between vault-mysql-router and
mysql-innodb-cluster is missing. This simulates the situation where
a `certificates-relation-joined` fires before vault has been setup
because the initial configuration of vault is gated on
`shared-db.available` flag being set.

This bug can present itself in subtly different ways that may initially
appear like the db-router/shared-db relations are at fault. In the
output below vault/0 and vault/2 are both hitting this bug and in the
case of vault/0 the bug was hit before the unit sent its db access request
to vault-mysql-router/2 which is why vault-mysql-router/2 is reporting it
has missing data.

Unit Workload Message
vault/0 error hook failed: "certificates-relation-joined"
  vault-mysql-router/2 waiting shared-db' incomplete, Waiting for proxied
                                   DB creation from cluster
vault/1* blocked Vault needs to be initialized
  vault-mysql-router/1 active Unit is ready
vault/2 error hook failed: "certificates-relation-joined"
  vault-mysql-router/0* active Unit is ready