The charm does not remove a mon with juju destroy-unit

[Johan Ehnberg]

Juju 1.20.14-amd64. Tested on MaaS and local providers. Tested on a clean install. Charm: cs:trusty/ceph-31

The commands run (after bootstrapping) are:
juju deploy -n3 --config local.yaml ceph
<wait to stabilize>
juju add-unit ceph
<wait for completion>
juju destroy-unit ceph/0
<wait for completion>
juju ssh 2 sudo ceph status

local.yaml has
fsid: <something>
monitor-count: 3
monitor-secret: <something>
osd-devices: '/dev/sdb /dev/sdc /dev/sdd'
osd-reformat: yes

After the commands are run, the mon is removed from ceph.conf but you'll see ceph still shows 4 running monitors.

Logically after the commands, one typically destroys the machine also:
juju destroy-machine <insert the destroyed unit's machine here>
juju ssh 3 sudo ceph status

This results in ceph warning about missing one mon
HEALTH_WARN 1 mons down

You have to remove it manually with the following command on a ceph node:
sudo ceph mon remove <monitor name>

[Chris Holcombe (xfactor973)]

This is just a matter of implementing the remove relationship for ceph and having it properly clean up the services. I've run into this also on bare metal