removing relation between kubernetes-master:vault-kv and vault:secrets doesn't work properly

[Seyeong Kim]

I deployed k8s with vault,

then removed relation between kubernets-master and vault

juju remove-relation kubernetes-master:vault-kv vault:secrets

then i faced error hook failed: "vault-kv-relation-departed"

error inside unit is below [1]

so I analyzed a bit then found that layer.vault-kv.ready is remained.

I removed it and juju resolved kubernetes-master/0 then it went stable status

juju run --unit kubernetes-master/0 -- "charms.reactive clear_flag layer.vault-kv.ready"

I think removing relation is not supported properly.

or I missed something else?

Thanks.

[1] ##########################################################################
2019-09-16 08:40:20 ERROR juju-log vault-kv:19: Hook error: Traceback (most recent call last): File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main bus.dispatch(restricted=restricted_mode) File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch _invoke(other_handlers) File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke handler.invoke() File "/var/lib/juju/agents/unit-kubernetes-master-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke self._action(*args) File "/var/lib/juju/agents/unit-kubernetes-master-0/charm/reactive/kubernetes_master.py", line 2251, in generate_encryption_key app_kv = vault_kv.VaultAppKV() File "lib/charms/layer/vault_kv.py", line 25, in __call__ cls._singleton_instance = super().__call__(*args, **kwargs) File "lib/charms/layer/vault_kv.py", line 96, in __init__ self._path = '{}/kv/app'.format(self._config['secret_backend']) File "lib/charms/layer/vault_kv.py", line 43, in _config _VaultBaseKV._config = get_vault_config() File "lib/charms/layer/vault_kv.py", line 201, in get_vault_config raise VaultNotReady() charms.layer.vault_kv.VaultNotReady

[Cory Johns (johnsca)]

There is a handler to clear the ready flag [1] but I think there is a timing issue with it due to the fact that other handlers might run between when the connected flag is removed and when that handler would run. I think the solution to this would be to use triggers [2], but we may also need to include the available flag in addition to connected. Also, the interface layer should probably also be migrated to use manage_flags [3] to improve the timing of the connected and available flags themselves, as well.

[1]: https://github.com/juju-solutions/layer-vault-kv/blob/master/reactive/vault_kv.py#L40-L43
[2]: https://charmsreactive.readthedocs.io/en/latest/triggers.html
[3]: https://charmsreactive.readthedocs.io/en/latest/charms.reactive.relations.html#charms.reactive.endpoints.Endpoint.manage_flags

[Cory Johns (johnsca)]

https://github.com/openstack-charmers/charm-interface-vault-kv/pull/7

[Cory Johns (johnsca)]

The interface PR has not been merged yet. Not sure why this got changed to released.

[George Kraft (cynerva)]

We're hitting this in CI. In addition to the open PR against interface-vault-kv, it looks to me like we'll need a PR against layer-vault-kv to fix the race condition on the clear_ready handler.

[Cory Johns (johnsca)]

PR for the layer-vault-kv portion: https://github.com/juju-solutions/layer-vault-kv/pull/9