| 2019-12-04 16:12:27 |
Giuseppe Petralia |
description |
In bootstack, we have many units in error state.
Looking at the debug-log we can see the following traceback:
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined Traceback (most recent call last):
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/compute-peer-relation-joined", line 648, in <module>
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined main()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/compute-peer-relation-joined", line 644, in main
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined assess_status(CONFIGS)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/nova_compute_utils.py", line 844, in assess_status
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined assess_status_func(configs)()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1422, in _assess_status_func
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined state, message = _determine_os_workload_status(*args, **kwargs)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 812, in _determine_os_workload_status
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined configs, required_interfaces)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 879, in _ows_check_generic_interfaces
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined required_interfaces)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1160, in incomplete_relation_data
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined complete_ctxts = configs.complete_contexts()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 351, in complete_contexts
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined for i in six.itervalues(self.templates)]
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 351, in <listcomp>
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined for i in six.itervalues(self.templates)]
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 127, in complete_contexts
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined self.context()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 112, in context
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined _ctxt = context()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 60, in __call__
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined token=token
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 123, in retrieve_secret_id
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined response = client._post('/v1/sys/wrapping/unwrap')
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1239, in _post
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined return self.__request('post', url, **kwargs)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1278, in __request
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined self.__raise_error(response.status_code, text, errors=errors)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1284, in __raise_error
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined raise exceptions.InvalidRequest(message, errors=errors)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist
2019-12-04 16:01:56 ERROR juju.worker.uniter.operation runhook.go:132 hook "compute-peer-relation-joined" failed: exit status 1
Even running the vault action "refresh-secrets" didn't fix the issue. |
In bootstack, we have many units (charms are nova-compute, ceph-osd and barbican-vault) in error state.
Looking at the debug-log we can see the following traceback:
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined Traceback (most recent call last):
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/compute-peer-relation-joined", line 648, in <module>
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined main()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/compute-peer-relation-joined", line 644, in main
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined assess_status(CONFIGS)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/nova_compute_utils.py", line 844, in assess_status
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined assess_status_func(configs)()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1422, in _assess_status_func
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined state, message = _determine_os_workload_status(*args, **kwargs)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 812, in _determine_os_workload_status
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined configs, required_interfaces)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 879, in _ows_check_generic_interfaces
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined required_interfaces)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1160, in incomplete_relation_data
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined complete_ctxts = configs.complete_contexts()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 351, in complete_contexts
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined for i in six.itervalues(self.templates)]
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 351, in <listcomp>
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined for i in six.itervalues(self.templates)]
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 127, in complete_contexts
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined self.context()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 112, in context
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined _ctxt = context()
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 60, in __call__
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined token=token
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/var/lib/juju/agents/unit-nova-compute-kvm-4/charm/hooks/charmhelpers/contrib/openstack/vaultlocker.py", line 123, in retrieve_secret_id
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined response = client._post('/v1/sys/wrapping/unwrap')
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1239, in _post
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined return self.__request('post', url, **kwargs)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1278, in __request
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined self.__raise_error(response.status_code, text, errors=errors)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined File "/usr/lib/python3/dist-packages/hvac/v1/__init__.py", line 1284, in __raise_error
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined raise exceptions.InvalidRequest(message, errors=errors)
2019-12-04 16:01:56 DEBUG compute-peer-relation-joined hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist
2019-12-04 16:01:56 ERROR juju.worker.uniter.operation runhook.go:132 hook "compute-peer-relation-joined" failed: exit status 1
Even running the vault action "refresh-secrets" didn't fix the issue. |
|
