postgresql:db relation fails if vault is not ready

Bug #2044298 reported by Max Asnaashari
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
vault-charm
New
Undecided
Unassigned

Bug Description

When running on the LXD Juju cloud provider with Juju 3.1 to 3.3:

```
juju deploy vault vault --model lxd-cloud/cell0-ovn-central --config auto-generate-root-ca-cert=true --config totally-unsecure-auto-unlock=true --channel 1.8/stable --num-units 1

juju deploy postgresql postgresql --model lxd-cloud/cell0-ovn-central --channel latest/stable --num-units 1

# Wait here

juju integrate vault:db postgresql:db --model lxd-cloud/cell0-ovn-central
```

Without sleeping for several minutes until vault reports `Unit is ready` before making the relation, the relation will fail with the error: `hook failed: "db-relation-changed" for postgresql:db

```
unit-vault-2: 21:39:56 INFO unit.vault/2.juju-log db:13: Reactive main running for hook db-relation-changed
unit-vault-2: 21:39:56 ERROR unit.vault/2.juju-log db:13: Unable to find implementation for relation: peers of vault-ha
unit-vault-2: 21:39:56 INFO unit.vault/2.juju-log db:13: Initializing Leadership Layer (is leader)
unit-vault-2: 21:39:56 INFO unit.vault/2.juju-log db:13: Initializing Snap Layer
unit-vault-2: 21:39:56 INFO unit.vault/2.juju-log db:13: Invoking reactive handler: reactive/vault_handlers.py:260:configure_vault_psql
unit-vault-2: 21:39:56 INFO unit.vault/2.juju-log db:13: Invoking reactive handler: reactive/vault_handlers.py:348:request_db
unit-vault-2: 21:39:56 INFO unit.vault/2.juju-log db:13: Invoking reactive handler: reactive/vault_handlers.py:737:prime_assess_status
unit-vault-2: 21:39:56 INFO unit.vault/2.juju-log db:13: Invoking reactive handler: reactive/vault_handlers.py:1165:tune_pki_backend_config_changed
unit-vault-2: 21:39:56 ERROR unit.vault/2.juju-log db:13: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-vault-2/charm/reactive/vault_handlers.py", line 1188, in tune_pki_backend_config_changed
    vault_pki.update_roles(max_ttl=max_ttl)
  File "/var/lib/juju/agents/unit-vault-2/charm/lib/charm/vault_pki.py", line 361, in update_roles
    local = client.secrets.pki.read_role(
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.10/site-packages/hvac/api/secrets_engines/pki.py", line 467, in read_role
    return self._adapter.get(
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.10/site-packages/hvac/adapters.py", line 113, in get
    return self.request("get", url, **kwargs)
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.10/site-packages/hvac/adapters.py", line 364, in request
    response = super(JSONAdapter, self).request(*args, **kwargs)
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.10/site-packages/hvac/adapters.py", line 330, in request
    utils.raise_for_error(
  File "/var/lib/juju/agents/unit-vault-2/.venv/lib/python3.10/site-packages/hvac/utils.py", line 43, in raise_for_error
    raise exceptions.InvalidPath(message, errors=errors, method=method, url=url)
hvac.exceptions.InvalidPath: no handler for route 'charm-pki-local/roles/local', on get http://127.0.0.1:8220/v1/charm-pki-local/roles/local

```

Tags: lxd-cloud
Revision history for this message
Max Asnaashari (masnax) wrote (last edit ):

Additionally, this also happens if you try to make 2 simultaneous relations to vault, even if you wait until all charms are happy by sleeping for 15 mins first.

```
juju deploy postgresql postgresql --model lxd-cloud/cell0-ovn-central --channel latest/stable --num-units 1
juju deploy vault vault --model lxd-cloud/cell0-ovn-central --config auto-generate-root-ca-cert=true --config totally-unsecure-auto-unlock=true --channel 1.8/stable --num-units 1
juju deploy ovn-central ovn-central --model lxd-cloud/cell0-ovn-central --channel 23.03/stable --num-units 3

# A long 15 minute sleep here

juju integrate ovn-central:certificates vault:certificates --model lxd-cloud/cell0-ovn-central
juju integrate vault:db postgresql:db --model lxd-cloud/cell0-ovn-central
```

Revision history for this message
Max Asnaashari (masnax) wrote (last edit ):
Download full text (3.5 KiB)

I'm also seeing a failure to establish the `vault:certificates` relation as a CMR:
```
unit-vault-29: 17:59:16 WARNING unit.vault/29.certificates-relation-created hvac.exceptions.InvalidPath: no handler for route 'charm-pki-local/cert/ca_chain', on get http://127.0.0.1:8220/v1/charm-pki-local/cert/ca_chain
unit-vault-29: 17:59:16 ERROR juju.worker.uniter.operation hook "certificates-relation-created" (via explicit, bespoke hook script) failed: exit status 1
unit-vault-29: 17:59:58 ERROR unit.vault/29.juju-log certificates:165: Unable to find implementation for relation: peers of vault-ha
unit-vault-29: 17:59:59 ERROR unit.vault/29.juju-log certificates:165: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.10/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.10/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-vault-29/charm/reactive/vault_handlers.py", line 1055, in publish_ca_info
    chain = vault_pki.get_chain()
  File "/var/lib/juju/agents/unit-vault-29/charm/lib/charm/vault_pki.py", line 86, in get_chain
    response = client.secrets.pki.read_certificate('ca_chain',
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.10/site-packages/hvac/api/secrets_engines/pki.py", line 76, in read_certificate
    return self._adapter.get(
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.10/site-packages/hvac/adapters.py", line 113, in get
    return self.request("get", url, **kwargs)
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.10/site-packages/hvac/adapters.py", line 364, in request
    response = super(JSONAdapter, self).request(*args, **kwargs)
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.10/site-packages/hvac/adapters.py", line 330, in request
    utils.raise_for_error(
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.10/site-packages/hvac/utils.py", line 43, in raise_for_error
    raise exceptions.InvalidPath(message, errors=errors, method=method, url=url)
hvac.exceptions.InvalidPath: no handler for route 'charm-pki-local/cert/ca_chain', on get http://127.0.0.1:8220/v1/charm-pki-local/cert/ca_chain
```

I'm running the following:

```
juju deploy ovn-central ovn-central --model lxd-cloud/cell0-ovn-central --channel 23.03/stable --num-units 3
juju deploy postgresql postgresql --model lxd-cloud/cell0-ovn-central --channel latest/stable --num-units 1
juju deploy vault vault --model lxd-cloud/cell0-ovn-central --config auto-generate-root-ca-cert=true --config totally-unsecure-auto-unlock=true --channel 1.8/stable --num-units 1

while ! juju status -m lxd-cloud/cell0-ovn-central | grep vault/ | grep -q "Unit is ready" ; do
  sleep 2
done

sleep 200

juju in...

Read more...

Revision history for this message
James Page (james-page) wrote :

Its worth noting that the gates for the vault charm do not test with pgsql; only with MySQL or with the native raft based database that vault 1.8 and later provides.

I'd recommend using the embedded raft based DB - it avoids the need for any other charms for vault.

I suspect that the pgsql integration in simply broken (this was inherited from and earlier owner of the vault charm) - I'd prefer to actually drop this support rather than fix it.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.