[18.11] wal and db LVs are created only on one utility device although multiple are specified

[Dmitrii Shcherbakov]

Although the code path in the charm clearly uses find_least_used_utility_device

https://github.com/openstack/charm-ceph-osd/blob/stable/18.11/lib/ceph/utils.py#L1636-L1651

and

juju config ceph-osd bluestore-db
/dev/disk/by-dname/nvme0n1-part1 /dev/disk/by-dname/nvme1n1-part1

juju config ceph-osd bluestore-wal
/dev/disk/by-dname/nvme0n1-part1 /dev/disk/by-dname/nvme1n1-part1

I still managed to get into a situation where all wal and db LVs are created on one device only (in this case, an NVMe partition):

tree /dev/disk/by-dname/
/dev/disk/by-dname/
├── md0 -> ../../md0
├── nvme0n1 -> ../../nvme0n1
├── nvme0n1-part1 -> ../../nvme0n1p1
├── nvme0n1-part2 -> ../../nvme0n1p2
├── nvme1n1 -> ../../nvme1n1
├── nvme1n1-part1 -> ../../nvme1n1p1
├── nvme1n1-part2 -> ../../nvme1n1p2
├── osddata0 -> ../../bcache5
├── osddata1 -> ../../bcache3
├── osddata2 -> ../../bcache2
├── osddata3 -> ../../bcache4
├── osddata4 -> ../../bcache0
├── osddata5 -> ../../bcache1
├── sda -> ../../sda
├── sda-part1 -> ../../sda1
├── sdb -> ../../sdb
├── sdc -> ../../sdc
├── sdd -> ../../sdd
├── sde -> ../../sde
├── sdf -> ../../sdf
├── sdf-part1 -> ../../sdf1
├── sdf-part2 -> ../../sdf2
├── sdf-part3 -> ../../sdf3
├── sdg -> ../../sdg
├── sdh -> ../../sdh
└── sdh-part1 -> ../../sdh1

unit log:

2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed Created symlink /etc/systemd/system/multi-user.target.wants/vaultlocker-decrypt@39c0b52e-46cd-4562-84b9-58d4
23e5495d.service → /lib/systemd/system/vaultlocker-decrypt@.service.
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed 1+0 records in
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed 1+0 records out
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed 512 bytes copied, 0.000341123 s, 1.5 MB/s
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed Failed to find physical volume "/dev/mapper/crypt-39c0b52e-46cd-4562-84b9-58d423e5495d".
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed Physical volume "/dev/mapper/crypt-39c0b52e-46cd-4562-84b9-58d423e5495d" successfully created.
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed Volume group "ceph-39c0b52e-46cd-4562-84b9-58d423e5495d" successfully created
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed Logical volume "osd-block-39c0b52e-46cd-4562-84b9-58d423e5495d" created.
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed Failed to find physical volume "/dev/nvme1n1p1".
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed Failed to find physical volume "/dev/nvme0n1p1".
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed Rounding up size to full physical extent 1.86 GiB
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed Logical volume "osd-wal-39c0b52e-46cd-4562-84b9-58d423e5495d" created.
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed Failed to find physical volume "/dev/nvme1n1p1".
2019-03-21 22:53:40 DEBUG secrets-storage-relation-changed Failed to find physical volume "/dev/nvme0n1p1".
2019-03-21 22:53:41 DEBUG secrets-storage-relation-changed Rounding up size to full physical extent <54.95 GiB
2019-03-21 22:53:41 DEBUG secrets-storage-relation-changed Logical volume "osd-db-39c0b52e-46cd-4562-84b9-58d423e5495d" created.
2019-03-21 22:53:41 INFO juju-log secrets-storage:237: osdize cmd: ['ceph-volume', 'lvm', 'create', '--osd-fsid', '39c0b52e-46cd-4562-84b9-58d423e5495d', '--bluestore', '--data', 'ceph-39c0b52e-46cd-4562-84b9-58d423e5495d/osd-block-39c0b52e-46cd-4562-84b9-58d423e5495d', '--block.wal', 'ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1/osd-wal-39c0b52e-46cd-4562-84b9-58d423e5495d', '--block.db', 'ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1/osd-db-39c0b52e-46cd-4562-84b9-58d423e5495d']

lsblk

# ...

nvme1n1 259:0 0 2.9T 0 disk
├─nvme1n1p1 259:1 0 676.1G 0 part
│ └─crypt-d3b03da9-e8d0-49aa-b844-122f29c68d91 253:2 0 676.1G 0 crypt
│ ├─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--wal--28b00e6c--842d--4adf--9723--518c611aa0bc 253:3 0 1.9G 0 lvm
│ ├─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--db--28b00e6c--842d--4adf--9723--518c611aa0bc 253:4 0 55G 0 lvm
│ ├─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--wal--0441d557--fe20--4468--b338--c10a967cb464 253:7 0 1.9G 0 lvm
│ ├─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--db--0441d557--fe20--4468--b338--c10a967cb464 253:8 0 55G 0 lvm
│ ├─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--wal--590929bb--38de--4755--ac00--01f91518e455 253:11 0 1.9G 0 lvm
│ ├─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--db--590929bb--38de--4755--ac00--01f91518e455 253:12 0 55G 0 lvm
│ ├─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--wal--39c0b52e--46cd--4562--84b9--58d423e5495d 253:15 0 1.9G 0 lvm
│ ├─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--db--39c0b52e--46cd--4562--84b9--58d423e5495d 253:16 0 55G 0 lvm
│ ├─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--wal--41d66453--658f--4810--b0e1--48b9bf53018c 253:19 0 1.9G 0 lvm
│ ├─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--db--41d66453--658f--4810--b0e1--48b9bf53018c 253:20 0 55G 0 lvm
│ ├─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--wal--0ca74253--0c28--4444--a758--c6afe8a4af9d 253:23 0 1.9G 0 lvm
│ └─ceph--wal--2ac31521--535b--4d43--94be--20f4d96dbff1-osd--db--0ca74253--0c28--4444--a758--c6afe8a4af9d 253:24 0 55G 0 lvm
└─nvme1n1p2 259:2 0 2.1T 0 part
  ├─bcache0 252:0 0 5.5T 0 disk
  │ └─crypt-41d66453-658f-4810-b0e1-48b9bf53018c 253:17 0 5.5T 0 crypt
  │ └─ceph--41d66453--658f--4810--b0e1--48b9bf53018c-osd--block--41d66453--658f--4810--b0e1--48b9bf53018c 253:18 0 5.5T 0 lvm
  ├─bcache1 252:128 0 5.5T 0 disk
  │ └─crypt-0ca74253-0c28-4444-a758-c6afe8a4af9d 253:21 0 5.5T 0 crypt
  │ └─ceph--0ca74253--0c28--4444--a758--c6afe8a4af9d-osd--block--0ca74253--0c28--4444--a758--c6afe8a4af9d 253:22 0 5.5T 0 lvm
  └─bcache4 252:512 0 5.5T 0 disk
    └─crypt-39c0b52e-46cd-4562-84b9-58d423e5495d 253:13 0 5.5T 0 crypt
      └─ceph--39c0b52e--46cd--4562--84b9--58d423e5495d-osd--block--39c0b52e--46cd--4562--84b9--58d423e5495d 253:14 0 5.5T 0 lvm
nvme0n1 259:3 0 2.9T 0 disk
├─nvme0n1p1 259:4 0 676.1G 0 part
└─nvme0n1p2 259:5 0 2.3T 0 part
  ├─bcache2 252:256 0 5.5T 0 disk
  │ └─crypt-590929bb-38de-4755-ac00-01f91518e455 253:9 0 5.5T 0 crypt
  │ └─ceph--590929bb--38de--4755--ac00--01f91518e455-osd--block--590929bb--38de--4755--ac00--01f91518e455 253:10 0 5.5T 0 lvm
  ├─bcache3 252:384 0 5.5T 0 disk
  │ └─crypt-0441d557-fe20-4468-b338-c10a967cb464 253:5 0 5.5T 0 crypt
  │ └─ceph--0441d557--fe20--4468--b338--c10a967cb464-osd--block--0441d557--fe20--4468--b338--c10a967cb464 253:6 0 5.5T 0 lvm
  └─bcache5 252:640 0 5.5T 0 disk
    └─crypt-28b00e6c-842d-4adf-9723-518c611aa0bc 253:0 0 5.5T 0 crypt
      └─ceph--28b00e6c--842d--4adf--9723--518c611aa0bc-osd--block--28b00e6c--842d--4adf--9723--518c611aa0bc 253:1 0 5.5T 0 lvm

root@block-storage-2:/var/lib/juju/agents/unit-ceph-osd-1/charm# lvs
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  osd-block-0441d557-fe20-4468-b338-c10a967cb464 ceph-0441d557-fe20-4468-b338-c10a967cb464 -wi-ao---- <5.46t
  osd-block-0ca74253-0c28-4444-a758-c6afe8a4af9d ceph-0ca74253-0c28-4444-a758-c6afe8a4af9d -wi-ao---- <5.46t
  osd-block-28b00e6c-842d-4adf-9723-518c611aa0bc ceph-28b00e6c-842d-4adf-9723-518c611aa0bc -wi-ao---- <5.46t
  osd-block-39c0b52e-46cd-4562-84b9-58d423e5495d ceph-39c0b52e-46cd-4562-84b9-58d423e5495d -wi-ao---- <5.46t
  osd-block-41d66453-658f-4810-b0e1-48b9bf53018c ceph-41d66453-658f-4810-b0e1-48b9bf53018c -wi-ao---- <5.46t
  osd-block-590929bb-38de-4755-ac00-01f91518e455 ceph-590929bb-38de-4755-ac00-01f91518e455 -wi-ao---- <5.46t
  osd-db-0441d557-fe20-4468-b338-c10a967cb464 ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- <54.95g
  osd-db-0ca74253-0c28-4444-a758-c6afe8a4af9d ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- <54.95g
  osd-db-28b00e6c-842d-4adf-9723-518c611aa0bc ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- <54.95g
  osd-db-39c0b52e-46cd-4562-84b9-58d423e5495d ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- <54.95g
  osd-db-41d66453-658f-4810-b0e1-48b9bf53018c ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- <54.95g
  osd-db-590929bb-38de-4755-ac00-01f91518e455 ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- <54.95g
  osd-wal-0441d557-fe20-4468-b338-c10a967cb464 ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- 1.86g
  osd-wal-0ca74253-0c28-4444-a758-c6afe8a4af9d ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- 1.86g
  osd-wal-28b00e6c-842d-4adf-9723-518c611aa0bc ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- 1.86g
  osd-wal-39c0b52e-46cd-4562-84b9-58d423e5495d ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- 1.86g
  osd-wal-41d66453-658f-4810-b0e1-48b9bf53018c ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- 1.86g
  osd-wal-590929bb-38de-4755-ac00-01f91518e455 ceph-wal-2ac31521-535b-4d43-94be-20f4d96dbff1 -wi-ao---- 1.86g

[Alex Kavanagh (ajkavanagh)]

TRIAGE:

Interestingly (or not?) the code failed to find the partitions on nvme0 even though they exist; this might be a reason for the failure?

[Markus Kienast (elias1884)]

This might be related to this bug: https://bugs.launchpad.net/charm-ceph-osd/+bug/1844996

Are you having MON and OSD on the same hosts?

Then OSD specific config options are not respected, as the ceph.conf created for MON is being used on OSD creation.

[Dmitrii Shcherbakov (dmitriis)]

No, in this case ceph-mons are placed into LXD containers so they have separate file systems.