Hello
I faced issue when deploying k8s with vault.
This symptom was not happened regularly but intermittently.
I've analyzed this but no clue where is the exact point this is happening.
actually generated token seems to be not proper.
Could you please advice me to analyze this or fix this?
#######################################################
error msg in kubernetes-master
#######################################################
2019-09-16 05:53:18 DEBUG vault-kv-relation-changed /var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charmhelpers/contrib/openstack/vaultlocker.py:127: DeprecationWarning: Call to deprecated function '_post'. This method will be removed in version '0.8.0' Please use the 'post' method on the 'hvac.adapters' class moving forward. 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed response = client._post('/v1/sys/wrapping/unwrap') 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed Traceback (most recent call last): 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/charm/hooks/vault-kv-relation-changed", line 22, in <module> 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed main() 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed bus.dispatch(restricted=restricted_mode) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed _invoke(other_handlers) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed handler.invoke() 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed self._action(*args) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/charm/reactive/vault_kv.py", line 22, in set_ready 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed vault_kv.get_vault_config() 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "lib/charms/layer/vault_kv.py", line 212, in get_vault_config 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed 'secret_id': _get_secret_id(vault), 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "lib/charms/layer/vault_kv.py", line 228, in _get_secret_id 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed secret_id = retrieve_secret_id(vault_url, token) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charmhelpers/contrib/openstack/vaultlocker.py", line 127, in retrieve_secret_id 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed response = client._post('/v1/sys/wrapping/unwrap') 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/hvac/utils.py", line 170, in new_func 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed return method(*args, **kwargs) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2525, in _post 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed return self._adapter.post(*args, **kwargs) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 103, in post 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed return self.request('post', url, **kwargs) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 272, in request 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed utils.raise_for_error(response.status_code, text, errors=errors) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/hvac/utils.py", line 30, in raise_for_error 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed raise exceptions.InvalidRequest(message, errors=errors) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist
#######################################################
Some analysis
#######################################################
juju run --unit kubernetes-master/0 -- "relation-get -r vault-kv:19 - vault/0" egress-subnets: 10.0.0.23/32 ingress-address: 10.0.0.23 kubernetes-master/0_role_id: '"72a89a04-6aea-db8d-2102-48694e0dde68"' kubernetes-master/0_token: '"s.wtHvWWnEDdjvWNuIgyflHQz4"' private-address: 10.0.0.23 vault_url: '"http://10.0.0.254:8200"'
vault token lookup s.wtHvWWnEDdjvWNuIgyflHQz4 Error looking up token: Error making API request. URL: POST http://10.0.0.23:8200/v1/auth/token/lookup Code: 403. Errors: * bad token
The token in the relation data is a one-shot token which is used to request a secret ID from Vault which is then saved and used for future requests. This is failing on the retrieve_secret_id call; the most likely reason why this would fail is if it was called more than once, possibly due to a previous hook error.
Can you check the logs and confirm whether this failure was the first error in the logs? Also, can you see if the secret_id already has a value:
juju run --unit kubernetes-master/0 -- chlp unitdata get layer.vault- kv.secret_ id