In testrun https://solutions.qa.canonical.com/v2/testruns/9c1ef2de-b611-45f1-95a5-c92882df08b0, ceph-osd fails with:
```
ceph-osd/0 error idle 0 10.246.167.73 hook failed: "mon-relation-changed"
ceph-osd/1 active idle 1 10.246.165.146 Unit is ready (2 OSD)
```
In the logs we see:
```
unit-ceph-osd-0: 17:12:23 INFO unit.ceph-osd/0.juju-log mon:19: osdize cmd: ['ceph-volume', 'lvm', 'create', '--osd-fsid', 'de4b0935-8a01-4542-92d3-2887316c19e1', '--bluestore', '--data', 'ceph-de4b0935-8a01-4542-92d3-2887316c19e1/osd-block-de4b0935-8a01-4542-92d3-2887316c19e1']
unit-ceph-osd-0: 17:12:23 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/ceph-authtool --gen-print-key
unit-ceph-osd-0: 17:12:23 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new de4b0935-8a01-4542-92d3-2887316c19e1
unit-ceph-osd-0: 17:12:23 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/ceph-authtool --gen-print-key
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-4
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed --> Executable selinuxenabled not in PATH: /var/lib/juju/tools/unit-ceph-osd-0:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-de4b0935-8a01-4542-92d3-2887316c19e1/osd-block-de4b0935-8a01-4542-92d3-2887316c19e1
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/ln -s /dev/ceph-de4b0935-8a01-4542-92d3-2887316c19e1/osd-block-de4b0935-8a01-4542-92d3-2887316c19e1 /var/lib/ceph/osd/ceph-4/block
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-4/activate.monmap
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 2022-11-18T17:12:24.094+0000 7fd150963640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed 2022-11-18T17:12:24.094+0000 7fd150963640 -1 AuthRegistry(0x7fd14c05f280) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed stderr: got monmap epoch 2
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-4/keyring --create-keyring --name osd.4 --add-key AQD3vHdjmcRfJBAAC/6ORavKd0qR6NNLp3OVVA==
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed stdout: creating /var/lib/ceph/osd/ceph-4/keyring
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed added entity osd.4 auth(key=AQD3vHdjmcRfJBAAC/6ORavKd0qR6NNLp3OVVA==)
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4/keyring
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4/
unit-ceph-osd-0: 17:12:24 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 4 --monmap /var/lib/ceph/osd/ceph-4/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-4/ --osd-uuid de4b0935-8a01-4542-92d3-2887316c19e1 --setuser ceph --setgroup ceph
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 2022-11-18T17:12:24.266+0000 7faeb1e355c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/) _read_fsid unparsable uuid
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 2022-11-18T17:12:24.582+0000 7faeb1e355c0 -1 bluefs _replay 0x0: stop: uuid 00000000-0000-0000-0000-000000000000 != super.uuid 8ded2b97-7f0e-4efa-9bfa-13bc85b45f76, block dump:
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: *
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 00000ff0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 00001000
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 2022-11-18T17:12:25.062+0000 7faeb1e355c0 -1 rocksdb: verify_sharding unable to list column families: NotFound:
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 2022-11-18T17:12:25.062+0000 7faeb1e355c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/) _open_db erroring opening db:
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 2022-11-18T17:12:25.602+0000 7faeb1e355c0 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 2022-11-18T17:12:25.602+0000 7faeb1e355c0 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-4/: (5) Input/output error[0m
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed --> Was unable to complete a new OSD, will rollback changes
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.4 --yes-i-really-mean-it
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 2022-11-18T17:12:25.722+0000 7f9b169b5640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed 2022-11-18T17:12:25.722+0000 7f9b169b5640 -1 AuthRegistry(0x7f9b1005f280) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx
unit-ceph-osd-0: 17:12:25 WARNING unit.ceph-osd/0.mon-relation-changed stderr: purged osd.4
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed --> Zapping: /dev/ceph-de4b0935-8a01-4542-92d3-2887316c19e1/osd-block-de4b0935-8a01-4542-92d3-2887316c19e1
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed --> Unmounting /var/lib/ceph/osd/ceph-4
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-4
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed stderr: umount: /var/lib/ceph/osd/ceph-4 unmounted
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-de4b0935-8a01-4542-92d3-2887316c19e1/osd-block-de4b0935-8a01-4542-92d3-2887316c19e1 bs=1M count=10 conv=fsync
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed stderr: 10+0 records in
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed 10+0 records out
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed 10485760 bytes (10 MB, 10 MiB) copied, 0.0365643 s, 287 MB/s
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed stderr:
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed --> Only 1 LV left in VG, will proceed to destroy volume group ceph-de4b0935-8a01-4542-92d3-2887316c19e1
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed Running command: vgremove -v -f ceph-de4b0935-8a01-4542-92d3-2887316c19e1
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed stderr: Removing ceph--de4b0935--8a01--4542--92d3--2887316c19e1-osd--block--de4b0935--8a01--4542--92d3--2887316c19e1 (253:0)
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed stderr: Archiving volume group "ceph-de4b0935-8a01-4542-92d3-2887316c19e1" metadata (seqno 3).
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed Releasing logical volume "osd-block-de4b0935-8a01-4542-92d3-2887316c19e1"
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed stderr: Creating volume group backup "/etc/lvm/backup/ceph-de4b0935-8a01-4542-92d3-2887316c19e1" (seqno 4).
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed stdout: Logical volume "osd-block-de4b0935-8a01-4542-92d3-2887316c19e1" successfully removed
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed stderr: Removing physical volume "/dev/sda5" from volume group "ceph-de4b0935-8a01-4542-92d3-2887316c19e1"
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed stdout: Volume group "ceph-de4b0935-8a01-4542-92d3-2887316c19e1" successfully removed
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed --> Zapping successful for OSD: 4
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.mon-relation-changed --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 4 --monmap /var/lib/ceph/osd/ceph-4/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-4/ --osd-uuid de4b0935-8a01-4542-92d3-2887316c19e1 --setuser ceph --setgroup ceph
unit-ceph-osd-0: 17:12:26 ERROR unit.ceph-osd/0.juju-log mon:19: Unable to initialize device: /dev/sda5
unit-ceph-osd-0: 17:12:26 WARNING unit.ceph-osd/0.juju-log mon:19: lsblk output: NAME="loop0" MAJ:MIN="7:0" RM="0" SIZE="63.2M" RO="1" TYPE="loop" MOUNTPOINTS="/snap/core20/1623"
NAME="loop1" MAJ:MIN="7:1" RM="0" SIZE="103M" RO="1" TYPE="loop" MOUNTPOINTS="/snap/lxd/23541"
NAME="loop2" MAJ:MIN="7:2" RM="0" SIZE="48M" RO="1" TYPE="loop" MOUNTPOINTS="/snap/snapd/17029"
NAME="loop3" MAJ:MIN="7:3" RM="0" SIZE="115M" RO="1" TYPE="loop" MOUNTPOINTS="/snap/core/13886"
NAME="loop4" MAJ:MIN="7:4" RM="0" SIZE="55.6M" RO="1" TYPE="loop" MOUNTPOINTS="/snap/core18/2620"
NAME="loop5" MAJ:MIN="7:5" RM="0" SIZE="10.9M" RO="1" TYPE="loop" MOUNTPOINTS="/snap/kubectl/2731"
NAME="loop6" MAJ:MIN="7:6" RM="0" SIZE="21.9M" RO="1" TYPE="loop" MOUNTPOINTS="/snap/kubelet/2711"
NAME="loop7" MAJ:MIN="7:7" RM="0" SIZE="9.4M" RO="1" TYPE="loop" MOUNTPOINTS="/snap/kube-proxy/2803"
NAME="loop8" MAJ:MIN="7:8" RM="0" SIZE="9M" RO="1" TYPE="loop" MOUNTPOINTS="/snap/canonical-livepatch/146"
NAME="sda" MAJ:MIN="8:0" RM="0" SIZE="223.6G" RO="0" TYPE="disk" MOUNTPOINTS=""
NAME="sda1" MAJ:MIN="8:1" RM="0" SIZE="1M" RO="0" TYPE="part" MOUNTPOINTS=""
NAME="sda2" MAJ:MIN="8:2" RM="0" SIZE="476M" RO="0" TYPE="part" MOUNTPOINTS="/boot/efi"
NAME="sda3" MAJ:MIN="8:3" RM="0" SIZE="1.9G" RO="0" TYPE="part" MOUNTPOINTS="/boot"
NAME="sda4" MAJ:MIN="8:4" RM="0" SIZE="167.6G" RO="0" TYPE="part" MOUNTPOINTS="/"
NAME="sda5" MAJ:MIN="8:5" RM="0" SIZE="53.1G" RO="0" TYPE="part" MOUNTPOINTS=""
NAME="sdb" MAJ:MIN="8:16" RM="0" SIZE="223.6G" RO="0" TYPE="disk" MOUNTPOINTS=""
NAME="sdb1" MAJ:MIN="8:17" RM="0" SIZE="1M" RO="0" TYPE="part" MOUNTPOINTS=""
NAME="sdb2" MAJ:MIN="8:18" RM="0" SIZE="223.5G" RO="0" TYPE="part" MOUNTPOINTS=""
```
Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/9c1ef2de-b611-45f1-95a5-c92882df08b0/index.html
