No block devices detected- permission denied when using ceph encryption on luminous+

[Chris MacNaughton]

There appears to be a race with udev setting ownership on things when using ceph Luminous+ with encryption

2018-11-15 14:23:46.480 7fbce106d240 0 set uid:gid to 64045:64045 (ceph:ceph)
2018-11-15 14:23:46.480 7fbce106d240 0 ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable), process ceph-osd, pid 14901
2018-11-15 14:23:46.480 7fbce106d240 0 pidfile_write: ignore empty --pid-file
2018-11-15 14:23:46.480 7fbce106d240 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied
2018-11-15 14:23:46.480 7fbce106d240 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied
2018-11-15 14:23:46.480 7fbce106d240 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied
2018-11-15 14:23:46.480 7fbce106d240 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied
2018-11-15 14:23:46.488 7fbce106d240 0 load: jerasure load: lrc load: isa
2018-11-15 14:23:46.488 7fbce106d240 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2018-11-15 14:23:46.488 7fbce106d240 1 bdev(0x55a3f5294700 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2018-11-15 14:23:46.488 7fbce106d240 -1 bdev(0x55a3f5294700 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied
2018-11-15 14:23:46.492 7fbce106d240 1 bluestore(/var/lib/ceph/osd/ceph-0) _mount path /var/lib/ceph/osd/ceph-0
2018-11-15 14:23:46.492 7fbce106d240 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied
2018-11-15 14:23:46.492 7fbce106d240 1 bdev create path /var/lib/ceph/osd/ceph-0/block type kernel
2018-11-15 14:23:46.492 7fbce106d240 1 bdev(0x55a3f5294700 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2018-11-15 14:23:46.492 7fbce106d240 -1 bdev(0x55a3f5294700 /var/lib/ceph/osd/ceph-0/block) open open got: (13) Permission denied
2018-11-15 14:23:46.492 7fbce106d240 -1 osd.0 0 OSD:init: unable to mount object store
2018-11-15 14:23:46.492 7fbce106d240 -1 ESC[0;31m ** ERROR: osd init failed: (13) Permission deniedESC[0m

[Chris MacNaughton (chris.macnaughton)]

This is reliably seen on the ceph-encrypt mojo specs for bionic-queens(L) and bionic-rocky(M)

[James Page (james-page)]

Worth noting that I think this is using ceph for key storage, so is not using vaultlocker.

So there may be a udev race - However I'm going to mark this as a low as this is not the 'default' key management choice.

[Chris Sanders (chris.sanders)]

I've triggered this today with Luminous on Bionic, ceph-osd-285.

For anyone that runs across this you can recover by running 'sudo chown ceph:ceph /dev/dm-*' and then restart the [e-mail].

After a crashed osd process this triggered was causing osds to fail to start for me. All of my osds are affected. This appears to be reliably happening (4 machines in my case all exhibit it) on bionic + luminous. There should really be a warning on the charmstore not to use this combination of configuration options if it's known to cause osds to fail, or maybe a higher priority to get it fixed?

Charm configurations should all be safe, deviating from a default shouldn't cause this sever of a failure.