hook failed: "vault-kv-relation-changed" when deploying master with vault

Bug #1844092 reported by Seyeong Kim on 2019-09-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Kubernetes Master Charm
Undecided
Unassigned

Bug Description

Hello

I faced issue when deploying k8s with vault.
This symptom was not happened regularly but intermittently.

I've analyzed this but no clue where is the exact point this is happening.

actually generated token seems to be not proper.

Could you please advice me to analyze this or fix this?

#######################################################
error msg in kubernetes-master
#######################################################

2019-09-16 05:53:18 DEBUG vault-kv-relation-changed /var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charmhelpers/contrib/openstack/vaultlocker.py:127: DeprecationWarning: Call to deprecated function '_post'. This method will be removed in version '0.8.0' Please use the 'post' method on the 'hvac.adapters' class moving forward. 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed response = client._post('/v1/sys/wrapping/unwrap') 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed Traceback (most recent call last): 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/charm/hooks/vault-kv-relation-changed", line 22, in <module> 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed main() 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed bus.dispatch(restricted=restricted_mode) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed _invoke(other_handlers) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed handler.invoke() 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed self._action(*args) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/charm/reactive/vault_kv.py", line 22, in set_ready 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed vault_kv.get_vault_config() 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "lib/charms/layer/vault_kv.py", line 212, in get_vault_config 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed 'secret_id': _get_secret_id(vault), 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "lib/charms/layer/vault_kv.py", line 228, in _get_secret_id 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed secret_id = retrieve_secret_id(vault_url, token) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charmhelpers/contrib/openstack/vaultlocker.py", line 127, in retrieve_secret_id 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed response = client._post('/v1/sys/wrapping/unwrap') 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/hvac/utils.py", line 170, in new_func 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed return method(*args, **kwargs) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/hvac/v1/__init__.py", line 2525, in _post 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed return self._adapter.post(*args, **kwargs) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 103, in post 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed return self.request('post', url, **kwargs) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/hvac/adapters.py", line 272, in request 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed utils.raise_for_error(response.status_code, text, errors=errors) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/hvac/utils.py", line 30, in raise_for_error 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed raise exceptions.InvalidRequest(message, errors=errors) 2019-09-16 05:53:18 DEBUG vault-kv-relation-changed hvac.exceptions.InvalidRequest: wrapping token is not valid or does not exist

#######################################################
Some analysis
#######################################################

juju run --unit kubernetes-master/0 -- "relation-get -r vault-kv:19 - vault/0" egress-subnets: 10.0.0.23/32 ingress-address: 10.0.0.23 kubernetes-master/0_role_id: '"72a89a04-6aea-db8d-2102-48694e0dde68"' kubernetes-master/0_token: '"s.wtHvWWnEDdjvWNuIgyflHQz4"' private-address: 10.0.0.23 vault_url: '"http://10.0.0.254:8200"'

vault token lookup s.wtHvWWnEDdjvWNuIgyflHQz4 Error looking up token: Error making API request. URL: POST http://10.0.0.23:8200/v1/auth/token/lookup Code: 403. Errors: * bad token

Tags: sts Edit Tag help
Cory Johns (johnsca) wrote :

The token in the relation data is a one-shot token which is used to request a secret ID from Vault which is then saved and used for future requests. This is failing on the retrieve_secret_id call; the most likely reason why this would fail is if it was called more than once, possibly due to a previous hook error.

Can you check the logs and confirm whether this failure was the first error in the logs? Also, can you see if the secret_id already has a value:

juju run --unit kubernetes-master/0 -- chlp unitdata get layer.vault-kv.secret_id

Seyeong Kim (xtrusia) wrote :

@johnsca

It was the only error I could see before.
In my test env secret_id was None. but I think my test env was contaminated with several tests.

In order to confirm this, I'll collect more information and I'll attach it.

Cory Johns (johnsca) wrote :

Another possibility that comes to mind would be that the unit token was regenerated on the Vault side for some reason but then also didn't make it over the relation in time before k8s-master saw and tried to use the old, now invalid value.

You could check for this by seeing if there were multiple instances of "Invoking reactive handler: .*configure_secrets_backend" in the Vault charm log.

Cory Johns (johnsca) wrote :

I haven't been able to replicate this in my testing.

Changed in charm-kubernetes-master:
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers