Barbican Vault - "secrets-storage-relation-joined" error with Barbican HA cfg

[Lorenzo Cavassa]

Ubuntu 18.04
Openstack Rocky
Barbican/Barbican Vault ~next charms

Description
===========

When Barbican is deployed using an HA configuration, 2 Barbican Vault units on of 3 show an error.
Here attached an extract of the unit log file.

When Barbican is deployed without HA (1 service unit only) it looks good.

[James Page (james-page)]

Please provide details of the bundle being deployed; as discussed on IRC I believe this is due to lack of explicit network space binding in the barbican-vault charm which results in random binding of the spaces by Juju.

[Lorenzo Cavassa (lorenzo-cavassa)]

Here is a bundle extract for the Barbican/Barbican Vault service:

......

hacluster-barbican:
    charm: cs:hacluster
    options:
      cluster_count: 3
  hacluster-octavia:
    charm: cs:hacluster
    options:
      cluster_count: 3
  barbican:
    charm: cs:barbican
    num_units: 1
    bindings:
     "": *oam-space
     public: *public-space
     admin: *admin-space
     internal: *internal-space
     shared-db: *internal-space
    options:
      worker-multiplier: *worker-multiplier
      openstack-origin: *openstack-origin
      region: *openstack-region
      vip: *barbican-vip
      use-internal-endpoints: true
      #os-admin-hostname: *barbican-admin-hostname
      #os-internal-hostname: *barbican-internal-hostname
      #os-public-hostname: *barbican-public-hostname
    to:
    - lxd:0
    - lxd:2
    - lxd:4
  barbican-vault:
    charm: cs:barbican-vault

........

[Lorenzo Cavassa (lorenzo-cavassa)]

There is a typo in the 'barbican' app section I pasted.
Here is the right line:

num_units: 3

Thank you,

Lorenzo

[Lorenzo Cavassa (lorenzo-cavassa)]

I modified the bundle to add an additional barbican-vault binding for
the secrets-storage relation:

applications:
  hacluster-barbican:
    charm: cs:hacluster
    options:
      cluster_count: 3
      #cluster_count: 1
  hacluster-octavia:
    charm: cs:hacluster
    options:
      cluster_count: 3
  barbican:
    charm: cs:barbican
    num_units: 3
    bindings:
     "": *oam-space
     public: *public-space
     admin: *admin-space
     internal: *internal-space
     shared-db: *internal-space
    options:
      worker-multiplier: *worker-multiplier
      openstack-origin: *openstack-origin
      region: *openstack-region
      vip: *barbican-vip
      use-internal-endpoints: true
      #os-admin-hostname: *barbican-admin-hostname
      #os-internal-hostname: *barbican-internal-hostname
      #os-public-hostname: *barbican-public-hostname
    to:
    - lxd:0
    - lxd:2
    - lxd:4
  barbican-vault:
    charm: cs:barbican-vault
    bindings:
     "": *oam-space
     secrets-storage: *internal-space

I got errors on all the 3 barbican-vault units.
Looks like all those try to reach Vault on the public API network (172.16.255/24) instead of on the internal API network (192.168.10/24) as by the binding definition.
Vault application isn't exposed on the public API network but it's connected with the internal API network only.

juju run --unit barbican-vault/9 "network-get --primary-address secrets-storage"
172.16.255.66

juju run --unit barbican-vault/8 "network-get --primary-address secrets-storage"
172.16.255.70

juju run --unit barbican-vault/7 "network-get --primary-address secrets-storage"
172.16.255.64

[James Page (james-page)]

Unable to reproduce - I've tested with barbican-vault bound to a specific space for the secrets-storage relation and its working just fine.

[Lorenzo Cavassa (lorenzo-cavassa)]

Here is the output of Juju status barbican-vault --format=yaml

https://pastebin.canonical.com/p/3NgTXBk3YQ/

This is the Vault one:

https://pastebin.canonical.com/p/f8xxS5Wvyg/

and this is the overlay bundle I'm using to deploy Octavia/Barbican:

https://pastebin.canonical.com/p/g6CSDb3sP7/

As you can see the barbican-vault endpoint-bindings:

secrets: ""
secrets-storage: ""

are empty even if defined in the bundle:

 barbican-vault:
    charm: cs:barbican-vault
    bindings:
     "": *oam-space
     secrets-storage: *internal-space
     secrets: *internal-space

Might it be a Juju bug when it has to deploy overlay bundles?

[Lorenzo Cavassa (lorenzo-cavassa)]

This is the whole Octavia overlay bundle:

https://pastebin.canonical.com/p/X4hNtMCYqr/

[Heather Lanigan (hmlanigan)]

Provided by @lorenzo-cavassa: https://pastebin.canonical.com/p/qQSmt3hdc6/

juju debug-log --include unit-barbican-vault-10

The add relation is failing on:

unit-barbican-vault-10: 15:37:05 INFO unit.barbican-vault/10.juju-log secrets-storage:353: Reactive main running for hook secrets-storage-relation-joined
unit-barbican-vault-10: 15:37:06 INFO unit.barbican-vault/10.juju-log secrets-storage:353: Invoking reactive handler: reactive/barbican_vault_handlers.py:47:plugin_info_barbican_publish
unit-barbican-vault-10: 15:37:06 INFO unit.barbican-vault/10.juju-log secrets-storage:353: Retrieving secret-id from vault (http://x.x.x.x:8200)
unit-barbican-vault-10: 15:37:06 ERROR unit.barbican-vault/10.juju-log secrets-storage:353: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-barbican-vault-10/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-barbican-vault-10/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-barbican-vault-10/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-barbican-vault-10/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-barbican-vault-10/charm/reactive/barbican_vault_handlers.py", line 62, in plugin_info_barbican_publish
    secrets_storage.unit_token)
  File "lib/charm/vault_utils.py", line 22, in retrieve_secret_id
    response = client._post('/v1/sys/wrapping/unwrap')
  File "/var/lib/juju/agents/unit-barbican-vault-10/.venv/lib/python3.6/site-packages/hvac/utils.py", line 169, in new_func
    return method(*args, **kwargs)
  File "/var/lib/juju/agents/unit-barbican-vault-10/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2515, in _post
    return self._adapter.post(*args, **kwargs)
  File "/var/lib/juju/agents/unit-barbican-vault-10/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 106, in post
    return self.request('post', url, **kwargs)
  File "/var/lib/juju/agents/unit-barbican-vault-10/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 265, in request
    utils.raise_for_error(response.status_code, text, errors=errors)
  File "/var/lib/juju/agents/unit-barbican-vault-10/.venv/lib/python3.6/site-packages/hvac/utils.py", line 29, in raise_for_error
    raise exceptions.InvalidRequest(message, errors=errors)
hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist

We can check what juju set the bindings to after deploy with "juju config barbican-vault"

[Tim Penhey (thumper)]

The problem is with the overlay. The variables defined in the primary bundle are not available to the overlay. The overlay is parsed independently of the primary bundle, so I'm guessing that the YAML parser is translating the variables to be empty.

[Drew Freiberger (afreiberger)]

I'm seeing this with the latest 20.02 charms on an openstack-on-lxd bundle modified to add barbican/octavia/vault/barbican-vault and no spaces support. All three units are showing secrets-storage-relation-changed failed hooks.

After deploying the bundle below, I have initialized vault, unsealed it, authorized the vault charm, and then after running refresh-secrets on the vault leader, the log below resulted from all of the barbican-vault unit logs.

In my case, I'm not using spaces or overlays and my bundle can be found here for reproducing this issue in a juju-on-lxd lab:

https://github.com/afreiberger/openstack-on-lxd/blob/add-bionic-stein-ha-bundle/bundle-bionic-stein-ha.yaml

unit-barbican-vault-1: 13:13:00 INFO unit.barbican-vault/1.juju-log secrets-storage:122: Reactive main running for hook secrets-storage-relation-changed
unit-barbican-vault-1: 13:13:01 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer>
tracer: set flag endpoint.secrets-storage.changed.barbican-vault/0_role_id
tracer: ++ queue handler hooks/relations/juju-info/requires.py:24:broken:juju-info
tracer: ++ queue handler hooks/relations/tls-certificates/requires.py:109:broken:certificates
tracer: ++ queue handler reactive/barbican_vault_handlers.py:34:secret_backend_vault_request
unit-barbican-vault-1: 13:13:01 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer: set flag endpoint.secrets-storage.changed.barbican-vault/0_token
unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer: set flag endpoint.secrets-storage.changed.barbican-vault/1_role_id
unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer: set flag endpoint.secrets-storage.changed.barbican-vault/1_token
unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer: set flag endpoint.secrets-storage.changed.barbican-vault/2_role_id
unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer: set flag endpoint.secrets-storage.changed.barbican-vault/2_token
unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer>
tracer: set flag secrets-storage.available
tracer: ++ queue handler reactive/barbican_vault_handlers.py:71:plugin_info_barbican_publish
tracer: -- dequeue handler reactive/barbican_vault_handlers.py:34:secret_backend_vault_request
unit-barbican-vault-1: 13:13:02 DEBUG unit.barbican-vault/1.juju-log secrets-storage:122: tracer>
tracer: starting handler dispatch, 27 flags set
tracer: set flag barbican-vault-installed
tracer: set flag charm.installed
tracer: set flag charms.openstack.do-default-charm.installed
tracer: set flag charms.openstack.do-default-config.changed
tracer: set flag charms.openstack.do-default-update-status
tracer: set flag config.default.ssl_ca
tracer: set flag config.default.ssl_cert
tracer: set flag config.default.ssl_key
tracer: set flag endpoint.secrets-storage.changed
tracer: set flag endpoint.secrets-storage.changed.barbican-vault/0_role_id
tracer: set flag endpoint.secrets-storage.changed.barbican-vault/0_token
tracer: set flag endpoint.secrets-sto...

[Drew Freiberger (afreiberger)]

My reproducer may be having issues because my bundle had two barbican applications. Testing further.

[Michael Skalka (mskalka)]

Drew,

Could this be a duplicate of https://bugs.launchpad.net/charm-barbican-vault/+bug/1871981 ?

[David Ames (thedac)]

OK, I think this bug has multiple duplicates.

The original report is a duplicate of https://bugs.launchpad.net/charm-barbican-vault/+bug/1849323 which was resolved.

Just today the fix for https://bugs.launchpad.net/charm-barbican-vault/+bug/1871981 landed in master.

And it seems Drew's bundle may have been incorrect.

Drew can you test with cs:~openstack-charmers-next/barbican-vault and let me know?

[Vladimir Grevtsev (vlgrevtsev)]

Reproduced today with cs:barbican-vault-1 with the same "hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist" error; will try to upgrade to the -next version and see if that helps or not.

[Vladimir Grevtsev (vlgrevtsev)]

cs:barbican-vault-13, of course*

[Vladimir Grevtsev (vlgrevtsev)]

@thedac, -next version worked fine without any hook failures.

[David Ames (thedac)]

@Vladimir, Thanks for the feedback. I will mark this as a duplicate officially.