add-osd action fails to add the osd disk

[Kamal Bhaskar]

Action "add-osd" fails to add the disk device as osd to the microceph cluster and returns below error:

$ juju run-action microceph/3 add-osd device-id=/dev/vdb --wait
unit-microceph-3:
  UnitId: microceph/3
  id: "26"
  message: "Error: Failed adding new disk: Failed to bootstrap OSD: Failed to run:
    ceph-osd --mkfs --no-mon-config -i 0: exit status 250 (2023-10-03T09:01:08.389+0000
    7fb8383fa800 -1 bluestore(/var/lib/ceph/osd/ceph-0) _open_db_and_around failed
    to load os-type: (2) No such file or directory\n2023-10-03T09:01:08.389+0000 7fb8383fa800
    -1 bluestore(/var/lib/ceph/osd/ceph-0) mkfs fsck found fatal error: (2) No such
    file or directory\n2023-10-03T09:01:08.389+0000 7fb8383fa800 -1 OSD::mkfs: ObjectStore::mkfs
    failed with error (2) No such file or directory\n2023-10-03T09:01:08.389+0000
    7fb8383fa800 -1 \e[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-0:
    (2) No such file or directory\e[0m)\n"
  results: {}
  status: failed
  timing:
    completed: 2023-10-03 09:01:09 +0000 UTC
    enqueued: 2023-10-03 09:01:07 +0000 UTC
    started: 2023-10-03 09:01:07 +0000 UTC
$

microceph/3 was also the leader unit.

No such directory existed actually:

# ls -lrt /var/lib/ceph/osd/ceph-0
ls: cannot access '/var/lib/ceph/osd/ceph-0': No such file or directory
#

Whereas the microceph cli inside the unit worked absolutely fine, used below command after ssh-ing into the same unit, which completed successfully:

microceph disk add /dev/vdb --wipe