cannot remove deployed etcd application

[Tim Van Steenburgh]

Opened by kwmonroe on 2018-10-29 15:28:56+00:00 at https://github.com/juju-solutions/layer-etcd/issues/142

------------------------------------------------------------

This is pretty simple to reproduce using juju-2.4.4:
$ juju deploy etcd etcd-test
Located charm "cs:etcd-209".
Deploying charm "cs:etcd-209".

## wait for deployment to settle

$ juju remove-application etcd-test
removing application etcd-test
This results in the following traceback:
unit-etcd-test-0: 10:19:49 INFO unit.etcd-test/0.juju-log cluster:18: Reactive main running for hook cluster-relation-broken
unit-etcd-test-0: 10:19:49 INFO unit.etcd-test/0.juju-log cluster:18: Initializing Leadership Layer (is leader)
unit-etcd-test-0: 10:19:49 INFO unit.etcd-test/0.juju-log cluster:18: Initializing Snap Layer
unit-etcd-test-0: 10:19:49 DEBUG unit.etcd-test/0.cluster-relation-broken none
unit-etcd-test-0: 10:19:49 INFO unit.etcd-test/0.juju-log cluster:18: Invoking reactive handler: reactive/etcd.py:461:perform_self_unregistration
unit-etcd-test-0: 10:19:49 DEBUG unit.etcd-test/0.cluster-relation-broken open /var/snap/etcd/common/server.crt: no such file or directory
unit-etcd-test-0: 10:19:49 INFO unit.etcd-test/0.juju-log cluster:18: b''
unit-etcd-test-0: 10:19:49 ERROR unit.etcd-test/0.juju-log cluster:18: Hook error:
Traceback (most recent call last):
  File "lib/etcdctl.py", line 157, in run
    return check_output(split(command)).decode('ascii')
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/snap/bin/etcd.etcdctl', 'member', 'list']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-etcd-test-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-etcd-test-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 379, in dispatch
    _invoke(hook_handlers)
  File "/var/lib/juju/agents/unit-etcd-test-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-etcd-test-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-etcd-test-0/charm/reactive/etcd.py", line 467, in perform_self_unregistration
    members = etcdctl.member_list()
  File "lib/etcdctl.py", line 88, in member_list
    out = self.run("{} member list".format(etcdctl_command()))
  File "lib/etcdctl.py", line 160, in run
    raise EtcdCtl.CommandFailed() from e
etcdctl.EtcdCtl.CommandFailed

This may be related to #138 as it also involves removing an etcd unit, but in that case, we failed with:
Command '['/snap/bin/etcd.etcdctl', '--endpoint', 'https://172.31.69.41:2379', 'member', 'remove', '3040be6455feb4e9']' returned non-zero exit status 1

Here we fail with:
Command '['/snap/bin/etcd.etcdctl', 'member', 'list']' returned non-zero exit status 1.