Where /usr/lib/ceph/ceph-osd-prestart.sh has some logic that exits with an error code when certain things aren't ready. I think we might be able to add something in there. For example it currently has:
data="/var/lib/ceph/osd/${cluster:-ceph}-$id"
if [ -L "$journal" -a ! -e "$journal" ]; then
udevadm settle --timeout=5 || :
if [ -L "$journal" -a ! -e "$journal" ]; then
echo "ceph-osd(${cluster:-ceph}-$id): journal not present, not starting yet." 1>&2
exit 0
fi
fi
The 'udevadm settle' watches the udev event queue and exists if all current events are handled or if it's been 5 seconds. Perhaps we can do something similar for this issue.
Here's what I see in /var/log/ceph/ceph-osd.0.log during a system reboot:
-------------------------------------------------------------------------
2019-05-29 19:04:25.800237 7fa6940d1700 1 freelist shutdown
...
2019-05-29 19:04:25.800548 7fa6940d1700 1 bdev(0x557eca7a1680 /var/lib/ceph/osd/ceph-0/block.wal) close
2019-05-29 19:04:26.079227 7fa6940d1700 1 bdev(0x557eca7a1200 /var/lib/ceph/osd/ceph-0/block.db) close
2019-05-29 19:04:26.266085 7fa6940d1700 1 bdev(0x557eca7a1440 /var/lib/ceph/osd/ceph-0/block) close
2019-05-29 19:04:26.474086 7fa6940d1700 1 bdev(0x557eca7a0fc0 /var/lib/ceph/osd/ceph-0/block) close
...
2019-05-29 19:04:53.601570 7fdd2ec17e40 1 bdev create path /var/lib/ceph/osd/ceph-0/block.db type kernel
2019-05-29 19:04:53.601581 7fdd2ec17e40 1 bdev(0x561e50583200 /var/lib/ceph/osd/ceph-0/block.db) open path /var/lib/ceph/osd/ceph-0/block.db
2019-05-29 19:04:53.601855 7fdd2ec17e40 1 bdev(0x561e50583200 /var/lib/ceph/osd/ceph-0/block.db) open size 1073741824 (0x40000000, 1GiB) block_size 4096 (4KiB) rotational
2019-05-29 19:04:53.601867 7fdd2ec17e40 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-0/block.db size 1GiB
2019-05-29 19:04:53.602131 7fdd2ec17e40 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2019-05-29 19:04:53.602143 7fdd2ec17e40 1 bdev(0x561e50583440 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2019-05-29 19:04:53.602464 7fdd2ec17e40 1 bdev(0x561e50583440 /var/lib/ceph/osd/ceph-0/block) open size 10733223936 (0x27fc00000, 10.0GiB) block_size 4096 (4KiB) rotational
2019-05-29 19:04:53.602480 7fdd2ec17e40 1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-0/block size 10.0GiB
2019-05-29 19:04:53.602499 7fdd2ec17e40 1 bdev create path /var/lib/ceph/osd/ceph-0/block.wal type kernel
2019-05-29 19:04:53.602502 7fdd2ec17e40 1 bdev(0x561e50583680 /var/lib/ceph/osd/ceph-0/block.wal) open path /var/lib/ceph/osd/ceph-0/block.wal
2019-05-29 19:04:53.602709 7fdd2ec17e40 1 bdev(0x561e50583680 /var/lib/ceph/osd/ceph-0/block.wal) open size 100663296 (0x6000000, 96MiB) block_size 4096 (4KiB) rotational
2019-05-29 19:04:53.602717 7fdd2ec17e40 1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-0/block.wal size 96MiB
...
I didn't recreate this but I did get a deployment on serverstack with bluestore WAL and DB devices. That's done with:
1) juju deploy --series bionic --num-units 1 --constraints mem=2G --config expected- osd-count= 1 --config monitor-count=1 cs:ceph-mon ceph-mon
2) juju deploy --series bionic --num-units 1 --constraints mem=2G --storage osd-devices= cinder, 10G --storage bluestore- wal=cinder, 1G --storage bluestore- db=cinder, 1G cs:ceph-osd ceph-osd
3) juju add-relation ceph-osd ceph-mon
James Page mentioned taking a look at the systemd bits.
ceph-osd systemd unit ------- ------- system/ ceph-osd@ .service calls: /usr/lib/ ceph/ceph- osd-prestart. sh --cluster ${CLUSTER} --id %i
-------
/lib/systemd/
ExecStartPre=
Where /usr/lib/ ceph/ceph- osd-prestart. sh has some logic that exits with an error code when certain things aren't ready. I think we might be able to add something in there. For example it currently has:
data="/ var/lib/ ceph/osd/ ${cluster: -ceph}- $id"
if [ -L "$journal" -a ! -e "$journal" ]; then ${cluster: -ceph}- $id): journal not present, not starting yet." 1>&2
udevadm settle --timeout=5 || :
if [ -L "$journal" -a ! -e "$journal" ]; then
echo "ceph-osd(
exit 0
fi
fi
The 'udevadm settle' watches the udev event queue and exists if all current events are handled or if it's been 5 seconds. Perhaps we can do something similar for this issue.
Here's what I see in /var/log/ ceph/ceph- osd.0.log during a system reboot: ------- ------- ------- ------- ------- ------- ------- ------- ------- --- ceph/osd/ ceph-0/ block.wal) close ceph/osd/ ceph-0/ block.db) close ceph/osd/ ceph-0/ block) close ceph/osd/ ceph-0/ block) close ceph/osd/ ceph-0/ block.db type kernel ceph/osd/ ceph-0/ block.db) open path /var/lib/ ceph/osd/ ceph-0/ block.db ceph/osd/ ceph-0/ block.db) open size 1073741824 (0x40000000, 1GiB) block_size 4096 (4KiB) rotational ceph/osd/ ceph-0/ block.db size 1GiB ceph/osd/ ceph-0/ block type kernel ceph/osd/ ceph-0/ block) open path /var/lib/ ceph/osd/ ceph-0/ block ceph/osd/ ceph-0/ block) open size 10733223936 (0x27fc00000, 10.0GiB) block_size 4096 (4KiB) rotational ceph/osd/ ceph-0/ block size 10.0GiB ceph/osd/ ceph-0/ block.wal type kernel ceph/osd/ ceph-0/ block.wal) open path /var/lib/ ceph/osd/ ceph-0/ block.wal ceph/osd/ ceph-0/ block.wal) open size 100663296 (0x6000000, 96MiB) block_size 4096 (4KiB) rotational ceph/osd/ ceph-0/ block.wal size 96MiB
-------
2019-05-29 19:04:25.800237 7fa6940d1700 1 freelist shutdown
...
2019-05-29 19:04:25.800548 7fa6940d1700 1 bdev(0x557eca7a1680 /var/lib/
2019-05-29 19:04:26.079227 7fa6940d1700 1 bdev(0x557eca7a1200 /var/lib/
2019-05-29 19:04:26.266085 7fa6940d1700 1 bdev(0x557eca7a1440 /var/lib/
2019-05-29 19:04:26.474086 7fa6940d1700 1 bdev(0x557eca7a0fc0 /var/lib/
...
2019-05-29 19:04:53.601570 7fdd2ec17e40 1 bdev create path /var/lib/
2019-05-29 19:04:53.601581 7fdd2ec17e40 1 bdev(0x561e50583200 /var/lib/
2019-05-29 19:04:53.601855 7fdd2ec17e40 1 bdev(0x561e50583200 /var/lib/
2019-05-29 19:04:53.601867 7fdd2ec17e40 1 bluefs add_block_device bdev 1 path /var/lib/
2019-05-29 19:04:53.602131 7fdd2ec17e40 1 bdev create path /var/lib/
2019-05-29 19:04:53.602143 7fdd2ec17e40 1 bdev(0x561e50583440 /var/lib/
2019-05-29 19:04:53.602464 7fdd2ec17e40 1 bdev(0x561e50583440 /var/lib/
2019-05-29 19:04:53.602480 7fdd2ec17e40 1 bluefs add_block_device bdev 2 path /var/lib/
2019-05-29 19:04:53.602499 7fdd2ec17e40 1 bdev create path /var/lib/
2019-05-29 19:04:53.602502 7fdd2ec17e40 1 bdev(0x561e50583680 /var/lib/
2019-05-29 19:04:53.602709 7fdd2ec17e40 1 bdev(0x561e50583680 /var/lib/
2019-05-29 19:04:53.602717 7fdd2ec17e40 1 bluefs add_block_device bdev 0 path /var/lib/
...