juju remove-application vault stuck at unresolvable hook error for secrets-relation-departed

[John George]

Removed vault with: juju remove-application vault

Two of the three units were removed without issues. The final unit went into an error state for the secrets-relation-departed hook. juju resolved with and without --no-retry did not allow the removal to proceed. In the end the workaround was to use juju remove-machine --force to clean up the model.

The vault unit log has this traceback:

2019-03-21 13:37:08 DEBUG secrets-relation-departed Traceback (most recent call last):
2019-03-21 13:37:08 DEBUG secrets-relation-departed File "/var/lib/juju/agents/unit-vault-1/charm/hooks/secrets-relation-departed", line 22, in <module>
2019-03-21 13:37:08 DEBUG secrets-relation-departed main()
2019-03-21 13:37:08 DEBUG secrets-relation-departed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.5/site-packages/charms/reactive/__init__.py", line 73, in main
2019-03-21 13:37:08 DEBUG secrets-relation-departed bus.dispatch(restricted=restricted_mode)
2019-03-21 13:37:08 DEBUG secrets-relation-departed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 390, in dispatch
2019-03-21 13:37:08 DEBUG secrets-relation-departed _invoke(other_handlers)
2019-03-21 13:37:08 DEBUG secrets-relation-departed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 359, in _invoke
2019-03-21 13:37:08 DEBUG secrets-relation-departed handler.invoke()
2019-03-21 13:37:08 DEBUG secrets-relation-departed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 181, in invoke
2019-03-21 13:37:08 DEBUG secrets-relation-departed self._action(*args)
2019-03-21 13:37:08 DEBUG secrets-relation-departed File "/var/lib/juju/agents/unit-vault-1/charm/reactive/vault_handlers.py", line 221, in configure_vault_mysql
2019-03-21 13:37:08 DEBUG secrets-relation-departed configure_vault(context)
2019-03-21 13:37:08 DEBUG secrets-relation-departed File "/var/lib/juju/agents/unit-vault-1/charm/reactive/vault_handlers.py", line 169, in configure_vault
2019-03-21 13:37:08 DEBUG secrets-relation-departed ca=context['etcd_tls_ca_file'])
2019-03-21 13:37:08 DEBUG secrets-relation-departed File "/var/lib/juju/agents/unit-vault-1/charm/reactive/vault_handlers.py", line 109, in save_etcd_client_credentials
2019-03-21 13:37:08 DEBUG secrets-relation-departed write_file(key, credentials['client_key'], perms=0o600)
2019-03-21 13:37:08 DEBUG secrets-relation-departed File "/var/lib/juju/agents/unit-vault-1/.venv/lib/python3.5/site-packages/charmhelpers/core/host.py", line 560, in write_file
2019-03-21 13:37:08 DEBUG secrets-relation-departed target.write(content)
2019-03-21 13:37:08 DEBUG secrets-relation-departed TypeError: a bytes-like object is required, not 'NoneType'
2019-03-21 13:37:08 ERROR juju.worker.uniter.operation runhook.go:132 hook "secrets-relation-departed" failed: exit status 1

[Cory Johns (johnsca)]

I think the issue actually lies in the etcd interface layer. The Vault code already has the failing code behind a check for the etcd.tls.available flag, which shouldn't be set if any of those values are None.

It looks like there need to be some else conditions added to the following to remove the flag if any of the fields are cleared before the relation is broken: https://github.com/juju-solutions/interface-etcd/blob/master/requires.py#L30-L36

[Chris MacNaughton (chris.macnaughton)]

https://github.com/juju-solutions/interface-etcd/issues/13

[Chris MacNaughton (chris.macnaughton)]

I raised the above issue on the interface-etcd repo to track this issue there as well