config-changed hooks fail for many charms with error: "wrapping token is not valid or does not exist"

[Giuseppe Petralia]

In bootstack, we have many units (charms are nova-compute, ceph-osd and barbican-vault) in error state.

Looking at the debug-log we can see the following traceback:

2019-12-04 16:01:56 DEBUG compute-peer-relation-joined Traceback (most recent call last):
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/compute-peer-relation-joined", line 648, in <module>
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined main()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/compute-peer-relation-joined", line 644, in main
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined assess_status(CONFIGS)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/nova_compute_utils.py", line 844, in assess_status
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined assess_status_func(configs)()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1422, in _assess_status_func
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined state, message = _determine_os_workload_status(*args, **kwargs)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 812, in _determine_os_workload_status
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined configs, required_interfaces)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 879, in _ows_check_generic_interfaces
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined required_interfaces)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1160, in incomplete_relation_data
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined complete_ctxts = configs.complete_contexts()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 351, in complete_contexts
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined for i in six.itervalues(self.templates)]
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 351, in <listcomp>
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined for i in six.itervalues(self.templates)]
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 127, in complete_contexts
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined self.context()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 112, in context
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined _ctxt = context()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 60, in __call__
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined token=token
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 123, in retrieve_secret_id
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined response = client._post('/v1/sys/wrapping/unwrap')
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1239, in _post
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined return self.__request('post', url, **kwargs)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1278, in __request
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined self.__raise_error(response.status_code, text, errors=errors)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1284, in __raise_error
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined raise exceptions.InvalidRequest(message, errors=errors)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist
2019-12-04 16:01:56 ERROR juju.worker.uniter.operation runhook.go:132 hook "compute-peer-relation-joined" failed: exit status 1

Even running the vault action "refresh-secrets" didn't fix the issue.

[Alex Kavanagh (ajkavanagh)]

Setting to incomplete. It's not clear how the system got into this state. i.e. were there any events prior to the error condition occurring; events being "rebooting of units", "power outage", "upgrade of something", "hardware failure", etc. Please could you provide a juju crashdump of the system (or if not possible), a sanitized juju model, and the debug-logs (--replay) for the vault unit and an units-in-error nova-compute, ceph-osd and barbican-vault.

[David O Neill (dmzoneill)]

hi,

for me, I resolved this by issuing

juju resolve <broken-barbian-unit> --no-retry
juju run-action vault/leader refresh-secrets

it resolved itself from there and the token was unwrapped