We ran into something similar when performing upgrade from queens to rocky UCA on ceph-osd when one of the OSDs was dead, hardware replaced, but the old /var/lib/ceph/osd/ceph-XX directory existed still on one of the ceph-osd units.
We worked around this by doing the following (in our case, the osd was 97, so replace XX with the dead OSD number.)
systemctl stop jujud-unit-ceph-osd-X
cd /var/lib/ceph/osd
cd ceph-XX
ls -al block
# Ensure that the referenced block device does not exist on the system or in lvm, etc before continuing.
rm -rf *
cd /var/lib/ceph/osd
umount /var/lib/ceph/osd/ceph-XX
rmdir ceph-XX
systemctl start jujud-unit-ceph-osd-X
We ran into something similar when performing upgrade from queens to rocky UCA on ceph-osd when one of the OSDs was dead, hardware replaced, but the old /var/lib/ ceph/osd/ ceph-XX directory existed still on one of the ceph-osd units.
We worked around this by doing the following (in our case, the osd was 97, so replace XX with the dead OSD number.)
systemctl stop jujud-unit- ceph-osd- X ceph/osd/ ceph-XX ceph-osd- X
cd /var/lib/ceph/osd
cd ceph-XX
ls -al block
# Ensure that the referenced block device does not exist on the system or in lvm, etc before continuing.
rm -rf *
cd /var/lib/ceph/osd
umount /var/lib/
rmdir ceph-XX
systemctl start jujud-unit-