ceph charm doesn't create all bcache backed osds

[Chris Procter]

We have nodes with 11 disks for use with ceph-osd, however on deployment sometimes only some of the disks are actually created as osds. e.g. (with 5 identical nodes)

sas-ceph-osd/0* active idle 31 x.x.x.x Unit is ready (11 OSD)
sas-ceph-osd/1 active idle 32 x.x.x.x Unit is ready (3 OSD)
sas-ceph-osd/2 active idle 33 x.x.x.x Unit is ready (3 OSD)
sas-ceph-osd/3 active idle 34 x.x.x.x Unit is ready (11 OSD)
sas-ceph-osd/4 active idle 35 x.x.x.x Unit is ready (11 OSD)

lsblk shows that the bcache devices have not had the required crypt and ceph devices created on top

sdk 8:160 0 5.5T 0 disk
└─bcache5 252:640 0 5.5T 0 disk
  └─crypt-8d447d6b-5b0a-4f47-97b0-15eb5933fb79 253:0 0 5.5T 0 crypt
    └─ceph--8d447d6b--5b0a--4f47--97b0--15eb5933fb79-osd--block--8d447d6b--5b0a--4f47--97b0--15eb5933fb79 253:1 0 5.5T 0 lvm
sdi 8:128 0 5.5T 0 disk
└─bcache4 252:512 0 5.5T 0 disk
sdg 8:96 0 5.5T 0 disk
└─bcache9 252:1152 0 5.5T 0 disk
sde 8:64 0 5.5T 0 disk
└─bcache7 252:896 0 5.5T 0 disk

the logs show:

root@osd-1:/var/log/juju# grep bcache4 unit-sas-ceph-osd-1.log
2018-07-24 09:24:17 INFO juju-log secrets-storage:224: Path /dev/disk/by-dname/bcache4 does not exist - bailing
2018-07-24 09:24:18 DEBUG juju-log secrets-storage:224: block device /dev/disk/by-dname/bcache4 uuid is None. Unable to save to hdparm.conf
root@osd-1:/var/log/juju# grep bcache7 unit-sas-ceph-osd-1.log
2018-07-24 09:24:18 INFO juju-log secrets-storage:224: Path /dev/disk/by-dname/bcache7 does not exist - bailing
2018-07-24 09:24:19 DEBUG juju-log secrets-storage:224: block device /dev/disk/by-dname/bcache7 uuid is None. Unable to save to hdparm.conf

but these devices do exist

root@osd-1:/var/log/juju# ls -l /dev/disk/by-dname/bcache*
lrwxrwxrwx 1 root root 13 Jul 24 09:29 /dev/disk/by-dname/bcache1 -> ../../bcache3
lrwxrwxrwx 1 root root 14 Jul 24 09:29 /dev/disk/by-dname/bcache10 -> ../../bcache10
lrwxrwxrwx 1 root root 14 Jul 24 09:29 /dev/disk/by-dname/bcache11 -> ../../bcache11
lrwxrwxrwx 1 root root 13 Jul 24 09:29 /dev/disk/by-dname/bcache12 -> ../../bcache0
lrwxrwxrwx 1 root root 13 Jul 24 09:29 /dev/disk/by-dname/bcache2 -> ../../bcache8
lrwxrwxrwx 1 root root 13 Jul 24 09:29 /dev/disk/by-dname/bcache3 -> ../../bcache7
lrwxrwxrwx 1 root root 13 Jul 24 09:29 /dev/disk/by-dname/bcache4 -> ../../bcache2
lrwxrwxrwx 1 root root 13 Jul 24 09:29 /dev/disk/by-dname/bcache5 -> ../../bcache9
lrwxrwxrwx 1 root root 13 Jul 24 09:29 /dev/disk/by-dname/bcache6 -> ../../bcache6
lrwxrwxrwx 1 root root 13 Jul 24 09:29 /dev/disk/by-dname/bcache7 -> ../../bcache4
lrwxrwxrwx 1 root root 13 Jul 24 09:29 /dev/disk/by-dname/bcache8 -> ../../bcache1
lrwxrwxrwx 1 root root 13 Jul 24 09:29 /dev/disk/by-dname/bcache9 -> ../../bcache5

root@osd-1:/var/log/juju# grep bcache /var/log/syslog
root@fnos-sas01:/var/log/juju# grep bcache ../syslog
Jul 24 08:39:25 fnos-sas01 kernel: [ 6.778545] bcache: bch_journal_replay() journal replay done, 0 keys in 2 entries, seq 61
Jul 24 08:39:25 fnos-sas01 kernel: [ 6.778652] bcache: register_cache() registered cache device nvme0n1p2
Jul 24 08:39:25 fnos-sas01 kernel: [ 6.779003] bcache: register_bdev() registered backing device md3
Jul 24 08:39:25 fnos-sas01 kernel: [ 6.799664] bcache: bch_cached_dev_attach() Caching md3 as bcache0 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.134760] bcache: register_bdev() registered backing device sdj
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.135887] bcache: bch_cached_dev_attach() Caching sdj as bcache1 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.136131] bcache: register_bdev() registered backing device sdf
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.137269] bcache: bch_cached_dev_attach() Caching sdf as bcache2 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.137471] bcache: register_bdev() registered backing device sdc
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.138554] bcache: bch_cached_dev_attach() Caching sdc as bcache3 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.138743] bcache: register_bdev() registered backing device sdi
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.139625] bcache: bch_cached_dev_attach() Caching sdi as bcache4 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.139822] bcache: register_bdev() registered backing device sdk
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.141545] bcache: bch_cached_dev_attach() Caching sdk as bcache5 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.141887] bcache: register_bdev() registered backing device sdh
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.146830] bcache: bch_cached_dev_attach() Caching sdh as bcache6 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.147120] bcache: register_bdev() registered backing device sde
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.148142] bcache: bch_cached_dev_attach() Caching sde as bcache7 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.148350] bcache: register_bdev() registered backing device sdd
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.149230] bcache: bch_cached_dev_attach() Caching sdd as bcache8 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.149464] bcache: register_bdev() registered backing device sdg
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.150392] bcache: bch_cached_dev_attach() Caching sdg as bcache9 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.241520] bcache: register_bdev() registered backing device sdl
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.242399] bcache: bch_cached_dev_attach() Caching sdl as bcache10 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.250523] bcache: register_bdev() registered backing device sdm
Jul 24 08:39:25 fnos-sas01 kernel: [ 7.251433] bcache: bch_cached_dev_attach() Caching sdm as bcache11 on set 516de282-52e7-4a0b-ad2c-769851d0a73a
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.102267] bcache: register_bcache() error /dev/sdj: device already registered (emitting change event)
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.134227] bcache: register_bcache() error /dev/sde: device already registered (emitting change event)
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.140876] bcache: register_bcache() error /dev/sdi: device already registered (emitting change event)
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.141938] bcache: register_bcache() error /dev/sdg: device already registered (emitting change event)
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.142135] bcache: register_bcache() error /dev/sdd: device already registered (emitting change event)
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.143213] bcache: register_bcache() error /dev/sdc: device already registered (emitting change event)
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.144269] bcache: register_bcache() error /dev/sdk: device already registered (emitting change event)
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.146892] bcache: register_bcache() error /dev/sdh: device already registered (emitting change event)
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.151928] bcache: register_bcache() error /dev/sdm: device already registered (emitting change event)
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.152894] bcache: register_bcache() error /dev/sdf: device already registered (emitting change event)
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.166031] bcache: register_bcache() error /dev/sdl: device already registered (emitting change event)
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.193388] bcache: register_bcache() error /dev/nvme0n1p2: device already registered
Jul 24 08:39:25 fnos-sas01 kernel: [ 16.417310] bcache: register_bcache() error /dev/md3: device already registered (emitting change event)

Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdi: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdc: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdg: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/sde: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdd: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdh: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdl: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdk: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdj: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdm: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdf: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/md3: device already registered (emitting change event)
Jul 24 09:24:32 fnos-sas01 kernel: bcache: register_bcache() error /dev/nvme0n1p2: device already registered
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdh: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdl: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdj: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdf: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdg: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdc: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdk: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdi: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdh: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdl: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdj: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdf: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdg: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdc: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdk: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdi: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdd: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdm: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/sde: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/md3: device already registered (emitting change event)
Jul 24 09:24:56 fnos-sas01 kernel: bcache: register_bcache() error /dev/nvme0n1p2: device already registered
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdh: device already registered (emitting change event)
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdd: device already registered (emitting change event)
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdf: device already registered (emitting change event)
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdg: device already registered (emitting change event)
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/sde: device already registered (emitting change event)
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdk: device already registered (emitting change event)
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdl: device already registered (emitting change event)
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdi: device already registered (emitting change event)
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdm: device already registered (emitting change event)
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdj: device already registered (emitting change event)
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdc: device already registered (emitting change event)
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/nvme0n1p2: device already registered
Jul 24 09:25:19 fnos-sas01 kernel: bcache: register_bcache() error /dev/md3: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdc: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdm: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdi: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdf: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdd: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdh: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdj: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/sde: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdk: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdl: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdg: device already registered (emitting change event)
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/nvme0n1p2: device already registered
Jul 24 09:25:37 fnos-sas01 kernel: bcache: register_bcache() error /dev/md3: device already registered (emitting change event)
Jul 24 09:29:57 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdc: device already registered (emitting change event)
Jul 24 09:29:57 fnos-sas01 kernel: bcache: register_bcache() error /dev/sde: device already registered (emitting change event)
Jul 24 09:29:57 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdd: device already registered (emitting change event)
Jul 24 09:29:57 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdj: device already registered (emitting change event)
Jul 24 09:29:57 fnos-sas01 kernel: bcache: register_bcache() error /dev/sdm: device already registered (emitting change event)

Our suspicion is the symlinks in /dev/disk/by-dname are being recreated every time udev trigger is run by the charm and this leads to a race condition if they are not recreated in time for the charm to try and create an osd from them

[Dmitrii Shcherbakov (dmitriis)]

Example of how his manifests itself:

unit-sas-ceph-osd-1.log:2018-07-24 09:24:17 INFO juju-log secrets-storage:224: Path /dev/disk/by-dname/bcache2 does not exist - bailing

stat /dev/disk/by-dname/bcache2 | grep Modify
Modify: 2018-07-24 09:29:57.220399555 +0000

/dev/disk/by-dname/bcache has obviously been recreated after cold plug (which happens when systemd-udevd starts replaying uvents from initramfs stage)

vaultlocker effectively triggers cold plug for block devices via udevadm

https://github.com/openstack-charmers/vaultlocker/blob/1.0.2/vaultlocker/dmcrypt.py#L89-L119

    command = [
        'udevadm',
        'trigger',
        '--subsystem-match=block',
        '--action=add'
    ]

http://man7.org/linux/man-pages/man8/udevadm.8.html
   udevadm trigger [options] [devpath|file...]
       Request device events from the kernel. Primarily used to replay
       events at system coldplug time.
       -c, --action=ACTION
           Type of event to be triggered. The default value is change.

Settle is used but only on a specific luks device:

https://github.com/openstack-charmers/vaultlocker/blob/1.0.2/vaultlocker/dmcrypt.py#L105-L119
    command = [
        'udevadm',
        'settle',
        '--exit-if-exists=/dev/disk/by-uuid/{}'.format(uuid),
    ]
   udevadm settle [options]
       Watches the udev event queue, and exits if all current events are
       handled.
       -E, --exit-if-exists=FILE
           Stop waiting if file exists.

This makes it a good candidate for a race if a node has a lot to do. We had about 3/18 nodes hitting this condition repeatedly.

[Dmitrii Shcherbakov (dmitriis)]

Subscribed ~field-critical as we cannot get consistent redeployments with all active OSDs with the race condition described above.

[Dmitrii Shcherbakov (dmitriis)]

We also consistently get this issue on nodes without bcache devices but with partition UUID-based symlinks: /dev/disk/by-dname/osd0-part0, /dev/disk/by-dname/osd1-part0, ...

[James Page (james-page)]

I've hacked in a udevadm settle call into the ceph-osd vaultlocker code:

  juju upgrade-charm --switch cs:~james-page/ceph-osd-bug1783308 ceph-osd

This will prove if @dmitriis hypothesis about udev racing is correct; we can then investigate a more complete solution.

[Dmitrii Shcherbakov (dmitriis)]

The non-bcache node issue was actually related to apparmor config: https://bugs.launchpad.net/charm-ceph-osd/+bug/1783373

So, this bug more likely affects only bcache-based setups.

[Chris Procter (chrisp262)]

We have redeployed with cs:~james-page/ceph-osd-bug1783308 but once again have 3 OSDs out of 11:

root@osd-1:/var/log/juju# grep bailing unit-sas-ceph-osd-1.log
2018-07-24 22:33:20 INFO juju-log secrets-storage:224: Path /dev/disk/by-dname/bcache1 does not exist - bailing
2018-07-24 22:33:20 INFO juju-log secrets-storage:224: Path /dev/disk/by-dname/bcache2 does not exist - bailing
2018-07-24 22:33:20 INFO juju-log secrets-storage:224: Path /dev/disk/by-dname/bcache3 does not exist - bailing
2018-07-24 22:33:21 INFO juju-log secrets-storage:224: Path /dev/disk/by-dname/bcache4 does not exist - bailing
2018-07-24 22:33:21 INFO juju-log secrets-storage:224: Path /dev/disk/by-dname/bcache5 does not exist - bailing
2018-07-24 22:33:21 INFO juju-log secrets-storage:224: Path /dev/disk/by-dname/bcache6 does not exist - bailing
2018-07-24 22:33:22 INFO juju-log secrets-storage:224: Path /dev/disk/by-dname/bcache7 does not exist - bailing
2018-07-24 22:33:22 INFO juju-log secrets-storage:224: Path /dev/disk/by-dname/bcache8 does not exist - bailing

root@osd-1:/var/log/juju# ceph-disk list
/dev/bcache0 other, unknown
/dev/bcache1 other, unknown
/dev/bcache10 other, crypto_LUKS
/dev/bcache11 other, crypto_LUKS
/dev/bcache2 other, unknown
/dev/bcache3 other, unknown
/dev/bcache4 other, unknown
/dev/bcache5 other, unknown
/dev/bcache6 other, unknown
/dev/bcache7 other, unknown
/dev/bcache8 other, crypto_LUKS
/dev/bcache9 other, crypto_LUKS

[Dmitrii Shcherbakov (dmitriis)]

Seems like letting a model to settle before activating vault may have an effect on that as nodes have less work to do, in another redeploy all osds are active:

juju status | grep -P '(nvme|sas)-ceph-osd'
nvme-ceph-osd 12.2.4 active 14 ceph-osd-bug1783308 jujucharms 0 ubuntu
sas-ceph-osd 12.2.4 active 5 ceph-osd-bug1783308 jujucharms 0 ubuntu
nvme-ceph-osd/0* active idle 0 192.0.2.11 Unit is ready (10 OSD)
nvme-ceph-osd/1 active idle 1 192.0.2.12 Unit is ready (10 OSD)
nvme-ceph-osd/2 active idle 2 192.0.2.13 Unit is ready (10 OSD)
nvme-ceph-osd/3 active idle 3 192.0.2.27 Unit is ready (10 OSD)
nvme-ceph-osd/4 active idle 4 192.0.2.15 Unit is ready (10 OSD)
nvme-ceph-osd/5 active idle 5 192.0.2.26 Unit is ready (10 OSD)
nvme-ceph-osd/6 active idle 6 192.0.2.18 Unit is ready (10 OSD)
nvme-ceph-osd/7 active idle 7 192.0.2.24 Unit is ready (10 OSD)
nvme-ceph-osd/8 active idle 8 192.0.2.25 Unit is ready (10 OSD)
nvme-ceph-osd/9 active idle 9 192.0.2.21 Unit is ready (10 OSD)
nvme-ceph-osd/10 active idle 10 192.0.2.23 Unit is ready (10 OSD)
nvme-ceph-osd/11 active idle 11 192.0.2.29 Unit is ready (10 OSD)
nvme-ceph-osd/12 active idle 12 192.0.2.51 Unit is ready (10 OSD)
nvme-ceph-osd/13 active idle 13 192.0.2.28 Unit is ready (10 OSD)
sas-ceph-osd/0 active idle 14 192.0.2.22 Unit is ready (11 OSD)
sas-ceph-osd/1 active executing 15 192.0.2.17 Unit is ready (11 OSD)
sas-ceph-osd/2 active executing 16 192.0.2.52 Unit is ready (11 OSD)
sas-ceph-osd/3 active idle 17 192.0.2.20 Unit is ready (11 OSD)
sas-ceph-osd/4* active idle 18 192.0.2.16 Unit is ready (11 OSD)

[James Page (james-page)]

I've put up:

  https://github.com/openstack-charmers/vaultlocker/pull/3

for review; testing now from the vaultlocker testing PPA.

[James Page (james-page)]

Tested OK in env with bcache enabled where we reproduce the original missing by-uuid behaviour.

vaultlocker change merged and patched in the vaultlocker tools PPA - Marking Fix Released.

[Chris MacNaughton (chris.macnaughton)]

Given that this is resolved in vaultlocker, I'm marking this invalid on ceph-osd