hook failed: "start" with failed to validate CIDR blocks: invalid CIDR address: None/32 error

Bug #1978542 reported by jukito
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
vault-charm
New
Undecided
Unassigned

Bug Description

My Leader of Vault, with an HA config, fails always saying this:

```
unit-vault-29: 08:40:28 ERROR juju.worker.uniter.operation hook "start" (via explicit, bespoke hook script) failed: exit status 1
unit-vault-29: 08:40:28 INFO juju.worker.uniter awaiting error resolution for "start" hook
unit-vault-29: 08:41:47 INFO juju.worker.uniter awaiting error resolution for "start" hook
unit-vault-29: 08:41:48 INFO unit.vault/29.juju-log Reactive main running for hook start
unit-vault-29: 08:41:48 ERROR unit.vault/29.juju-log Unable to find implementation for relation: peers of vault-ha
unit-vault-29: 08:41:48 INFO unit.vault/29.juju-log Initializing Snap Layer
unit-vault-29: 08:41:48 INFO unit.vault/29.juju-log Initializing Leadership Layer (is leader)
unit-vault-29: 08:41:49 INFO unit.vault/29.juju-log Invoking reactive handler: reactive/vault_handlers.py:254:configure_vault_mysql
unit-vault-29: 08:41:49 INFO unit.vault/29.juju-log Etcd detected, setting api_addr to http://xxxxxxxxxx:8200
unit-vault-29: 08:41:49 INFO unit.vault/29.juju-log Invoking reactive handler: reactive/vault_handlers.py:296:mysql_setup
unit-vault-29: 08:41:50 INFO unit.vault/29.juju-log Invoking reactive handler: reactive/vault_handlers.py:327:database_not_ready
unit-vault-29: 08:41:50 INFO unit.vault/29.juju-log Invoking reactive handler: reactive/vault_handlers.py:417:cluster_connected
unit-vault-29: 08:41:50 INFO unit.vault/29.juju-log Invoking reactive handler: reactive/vault_handlers.py:469:configure_secrets_backend
unit-vault-29: 08:41:50 ERROR unit.vault/29.juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-vault-29/charm/reactive/vault_handlers.py", line 545, in configure_secrets_backend
    approle_id = vault.configure_approle(
  File "/var/lib/juju/agents/unit-vault-29/charm/lib/charm/vault.py", line 390, in configure_approle
    client.create_role(
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.8/site-packages/hvac/v1/__init__.py", line 1854, in create_role
    return self._adapter.post('/v1/auth/{0}/role/{1}'.format(mount_point, role_name), json=kwargs)
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 103, in post
    return self.request('post', url, **kwargs)
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.8/site-packages/hvac/adapters.py", line 233, in request
    utils.raise_for_error(response.status_code, text, errors=errors)
  File "/var/lib/juju/agents/unit-vault-29/.venv/lib/python3.8/site-packages/hvac/utils.py", line 39, in raise_for_error
    raise exceptions.InternalServerError(message, errors=errors)
hvac.exceptions.InternalServerError: 1 error occurred:
        * failed to validate CIDR blocks: invalid CIDR address: None/32
```

The vault juju config are this:

```
  vault:
    charm: cs:vault
    num_units: 3
    to:
    - lxd:0
    - lxd:1
    - lxd:2
    options:
      vip: xxxxxxxxx xxxxxxxxxx
    bindings:
      "": *internal
      ha: *admin
  vault-mysql-router:
    charm: cs:mysql-router
    bindings:
      db-router: *internal
      shared-db: *internal
relations:
- - vault-mysql-router:db-router
  - mysql-innodb-cluster:db-router
- - vault:shared-db
  - vault-mysql-router:shared-db
- - vault:etcd
  - etcd:db
- - ha-vault:ha
  - vault:ha
- - nova-compute-amd:secrets-storage
  - vault:secrets
- - vault:certificates
  - neutron-api-plugin-ovn:certificates
- - ovn-central:certificates
  - vault:certificates
- - ovn-chassis:certificates
  - vault:certificates
```

My rev of Vault is 54. Any hint welcomed!

Revision history for this message
Vern Hart (vern) wrote :

I've been seeing this with vault revision 82 (channel 1.7/stable).

When running the hooks, I added some debug code and see that two relations in the requests list have ingress_address=None; Units ceph-osd/2 and ceph-osd/8. All the other ceph-osd units seem to have worked as expected.

I was able to work around the issue by using request['access_address'] if request['ingress_address'] is None. (Simple patch attached.)

I am not sure if this is a good general solution but in my situation, where the ingress_address is set, it's the same as the access_address.

Revision history for this message
Shunde Zhang (shunde-zhang) wrote :

I hit this issue when I deployed openstack ussuri in my lab.

nova-compute 21.2.4 waiting 1/2 nova-compute ussuri/stable 669 no Incomplete relations: vault
vault 1.7.9 error 3 vault 1.7/stable 107 no hook failed: "secrets-relation-changed"

I have set encrypt to true in nova-compute.

$ juju config nova-compute encrypt
true

So it requires a relation with vault.

vault:secrets nova-compute:secrets-storage vault-kv regular

However there is no ingress-address in relation data, it has access-address only.

$ juju show-unit vault/6
......
  - relation-id: 118
    endpoint: secrets
    related-endpoint: secrets-storage
    application-data: {}
    related-units:
      nova-compute/4:
        in-scope: true
        data:
          access_address: 192.168.1.87
          hostname: helped-orca
          isolated: "True"
          secret_backend: charm-vaultlocker

Revision history for this message
Shunde Zhang (shunde-zhang) wrote :

It turns out that the issue is caused by a missing network interface.
I have vault in a network space but nova-compute doesn't have an interface attached to that network space.
Hence relation data doesn't have a valid ingress address since they are not in the same subnet.
Adding an interface in the same network space to nova-compute solved the issue.

Changed in vault-charm:
status: New → Invalid
status: Invalid → New
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.