[OPS] Removing and adding ceph-osd units causes osd.id to be reused without cleaning

[Pedro Guimarães]

Hi,

On a long-time running cluster, if we remove and readd ceph-osds several times, we will inevitably start seeing Ceph reusing osd.ids and their corresponding keys.

In this case, charm will fail at ceph-volume lvm creation with: https://pastebin.canonical.com/p/FMcsPP9WXW/

To clean-up, we must manually run:
ceph osd rm osd.<id>
ceph auth del osd.<id>

For each of the OSDs present on ceph-osd unit.

The charm should deal with that at -departed hook context.