On one unit that was performing a luminous->mimic upgrade on charm version 21.04, experienced an infinite loop of the following in the unit debug-logs:
unit-ceph-osd-13: 20:42:16 DEBUG unit.ceph-osd/13.juju-log Command '['ceph', 'daemon', '/var/run/ceph/ceph-osd.51.asok', 'status']' returned non-zero exit status 22.
unit-ceph-osd-13: 20:42:16 DEBUG unit.ceph-osd/13.config-changed admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
This exit status 22 is indicating the OSD doesn't exist or isn't running.
When investigating, I found the following /var/lib/ceph/osd/ directories and mounts:
myhost:/var/lib/ceph/osd# ls -al
total 16
drwxr-xr-x 9 ceph ceph 4096 Jul 2 2020 .
drwxr-x--- 11 ceph ceph 4096 Sep 9 2019 ..
drwxr-xr-x 2 ceph ceph 4096 Sep 9 2019 ceph-51
drwxrwxrwt 2 ceph ceph 200 Jun 28 02:35 ceph-56
drwxrwxrwt 2 ceph ceph 200 Jun 28 02:35 ceph-60
drwxrwxrwt 2 ceph ceph 200 Jun 28 02:35 ceph-64
drwxrwxrwt 2 ceph ceph 200 Jun 28 02:35 ceph-68
drwxrwxrwt 2 ceph ceph 200 Jun 28 02:35 ceph-74
drwxrwxrwt 2 ceph ceph 200 Jun 28 02:35 ceph-89
-rw------- 1 ceph ceph 69 Sep 9 2019 ceph.client.osd-upgrade.keyring
myhost:/var/lib/ceph/osd# mount|grep ceph
tmpfs on /var/lib/ceph/osd/ceph-74 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-56 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-68 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-64 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-60 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-89 type tmpfs (rw,relatime)
running rmdir /var/lib/ceph/osd/ceph-51 killing, and restarting the agent allowed the command to continue. I think somehow there needs to be better logic of configured/available OSDs to cover poor cleanup after replacing an OSD.
Fix proposed to branch: master
Review: https:/