support removal of osd

[Wouter van Bommel]

The current charm does not support removal & cleanup of a current disk.

This means that at the moment there is a physical disk failure the charm cannot be used to remove it from the existing ceph cluster, cleanup wal logical volume and db logical volume if bluestore is used etc.

It would save a lot of time, if this could be automated, versus manually having to match id's from lsblk to figure out which id of the bluestore is no longer in use.

And if vault is used, to manually remove the lv for the osd, the vg for the osd and the pv for the osd, etc.

[Alex Kavanagh (ajkavanagh)]

Note, "wishlist" doesn't mean it's unimportant, just that it's a feature request and not a bug.

[Paul Goins (vultaire)]

I'd like to add regarding vault specifically: we encountered an issue on a customer cloud where we rebooted after removing an OSD, and the system hung for nearly 3 hours (10000 seconds) because the vaultlocker systemctl job uses 10000 seconds as its retry threshold. I would have thought it was just completely hung if it weren't for it hanging at my EOD and then things were up and running when I came back the next morning.

While I've updated the associated internal playbook regarding this so we can hopefully avoid this in the future, it's another reason why it'd be good to have the charm handle this cleanup if possible.