storage pool with persistent device names fails

[Jan Marquardt]

lvm2 tools do not store by-path values in their config.
That can drive scripts/tools (like the libvirt case here) mad.
Simplified example:

$ sudo /sbin/pvcreate /dev/disk/by-path/virtio-pci-0000:00:07.0
$ sudo /sbin/pvs --noheadings -o pv_name,vg_name
  /dev/vdc

It is NOT using /dev/disk/by-path/virtio-pci-0000:00:07.0 as instructed, but instead did the mapping to /dev/vdd and keeps that from now on.

I know that "inside" lvm all that is done via metadata and it won't matter.
But tools around it get confused.
Not sure yet if it is a real issue, or just "works as designed".

--- started as libvirt issue - keeping old desc below ---

I am trying to create a logical volume storage pool with persistent
device names for the block devices, because I can't/don't want to ensure
that /dev/sd* names are persistent through reboots.

virsh # pool-define-as web2067 logical - -
/dev/disk/by-path/pci-0000:18:00.0-scsi-0:2:2:0 web2067 /dev/web2067
virsh # pool-build web2067

These two steps work as expected and the pv and the vg get created, but
when I try to start the pool I get the following error:

virsh # pool-start web2067
error: Failed to start pool web2067
error: unsupported configuration: cannot find any matching source
devices for logical volume group 'web2067'

If I do the same steps just with /dev/sdX instead of
/dev/disk/by-path/..., the pool gets created correctly.

According to the libvirt mailing list /dev and /dev/disk/by-path are
supposed to work the same way.

[Christian Ehrhardt  (paelzer)]

Hi Jan,
to reproduce I started a KVM guest with two extra disks:

$ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test-storage-pools-1.img 30M
$ sudo qemu-img create -f qcow2 /var/lib/libvirt/images/test-storage-pools-2.img 30M
Add to the guest like:
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test-storage-pools-1.img'/>
      <backingStore/>
      <target dev='vdc' bus='virtio'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/test-storage-pools-2.img'/>
      <backingStore/>
      <target dev='vdd' bus='virtio'/>
    </disk>

This is fix on PCI slot 7 and 8.
So on by-path I can rely on
lrwxrwxrwx 1 root root 9 Nov 6 12:02 virtio-pci-0000:00:07.0 -> ../../vdc
lrwxrwxrwx 1 root root 9 Nov 6 12:02 virtio-pci-0000:00:08.0 -> ../../vdd

Since https://libvirt.org/storage.html#StorageBackendLogical doesn't need host and dir path I didn't define those.
I used:
$ virsh pool-define-as --print-xml testpool1 logical --source-dev virtio-pci-0000:00:07.0 --target /dev/testpoolVG
<pool type='logical'>
  <name>testpool1</name>
  <source>
    <device path='virtio-pci-0000:00:07.0'/>
  </source>
  <target>
    <path>/dev/testpoolVG</path>
  </target>
</pool>

$ sudo virsh pool-define-as testpool1 logical --source-dev virtio-pci-0000:00:07.0 --target /dev/testpoolVG
$ sudo virsh pool-define-as testpool2 logical --source-dev /dev/vdd --target /dev/testpoolVG2

Already on pool-build I see it missing the by path device
$ sudo virsh pool-build testpool1
error: Failed to build pool testpool1
error: Storage pool probe failed: Failed to create filesystem probe for device virtio-pci-0000:00:07.0
$ sudo virsh pool-build testpool2
Pool testpool2 built

The same is true if I pool-build one defined with the syntax you used:
$ sudo virsh pool-define-as testpool2 logical - - virtio-pci-0000:00:08.0 /dev/testpoolVG2

Related log message of the daemon:
 libvirtd[6311]: 6315: error : virStorageBackendBLKIDFindEmpty:3130 : Storage pool probe failed: Failed to create filesystem probe for device virtio-pci-0000:00:07.0

If we enable debug we see this before:
  debug: virStorageBackendBLKIDFindEmpty:3125 : Probe for existing filesystem/partition format LVM2_member on device virtio-pci-0000:00:07.0

In the case of the working /dev/vdd it uses a full path in this call:
  debug: virStorageBackendBLKIDFindEmpty:3125 : Probe for existing filesystem/partition format LVM2_member on device /dev/vdd

Now it is not the code of libvirt on pool-build that "looses" that information.
No if we look more carefully at the XML that was defined it is:
  <device path='virtio-pci-0000:00:07.0'/>
Silly me :-/ I forgot the full path in the definition
Redefine as:
$ sudo virsh pool-define-as testpool1 logical --source-dev /dev/disk/by-path/virtio-pci-0000:00:07.0 --target /dev/testpoolVG

Yeah that looks better:
$ sudo virsh pool-dumpxml testpool1
<pool type='logical'>
  <name>testpool1</name>
  <uuid>d85ec218-9b82-4f99-8b75-6b11840d06b5</uuid>
  <capacity unit='bytes'>0</capacity>
  <allocation unit='bytes'>0</allocation...

[Christian Ehrhardt  (paelzer)]

Error on starting the device:
  error : virStorageBackendLogicalMatchPoolSource:692 : unsupported configuration: cannot find any matching source devices for logical volume group 'testpool1'

That error from the log matches the error in the UI of virsh.

The reason is that even thou it was defined by-path the lvm toolchain has stored it as /dev/vdd and /dev/vdc

$ sudo /sbin/pvs --noheadings -o pv_name,vg_name
  /dev/vdc testpool1
  /dev/vdd testpool2

That is exactly the command it uses to build a list of existing VGs and then it looks for.
It finds the pool and then will check for the sources that are registered.

 666 /* If the pool has defined source device(s), then let's make sure
 667 * they match as well; otherwise, matching can only occur on the
 668 * pool's name.
 669 */
 670 if (!def->source.ndevice) {
 671 ret = true;
 672 goto cleanup;
 673 }
 674
 675 /* Let's make sure the pool's device(s) match what the pvs output has
 676 * for volume group devices.
 677 */
 678 for (i = 0; i < def->source.ndevice; i++) {
 679 for (j = 0; j < thisSource->ndevice; j++) {
 680 if (STREQ(def->source.devices[i].path,
 681 thisSource->devices[j].path))
 682 matchcount++;
 683 }
 684 }
 685
 686 /* If we didn't find any matches, then this pool has listed (a) source
 687 * device path(s) that don't/doesn't match what was created for the pool
 688 */
 689 if (matchcount == 0) {

Since this will not match as above output has stored it as /dev/vdd it cann't find it and will hit the issue described here.

Commands from the pool-build run:
 /sbin/pvcreate /dev/disk/by-path/virtio-pci-0000:00:07.0
 /sbin/vgcreate testpool1 /dev/disk/by-path/virtio-pci-0000:00:07.0

I cleared the vg/lv/pv configuration from my disks and did the same without any libvirt in play.
ubuntu@cosmic-storage-pools:~$ sudo /sbin/pvcreate /dev/disk/by-path/virtio-pci-0000:00:07.0
  Physical volume "/dev/disk/by-path/virtio-pci-0000:00:07.0"...

[Christian Ehrhardt  (paelzer)]

That is in fact a LVM limitation of the default config that is targetted at low multi-scanning and general compatibility with older setups.

You'll find in /etc/lvm/lvm.conf a disabled line like:
# filter = [ "a|.*/|" ]

That means that it will consider /dev/* but not more - in particular not /dev/disk/by-...

If you configure lvm to consider the paths you want (it will pick up the first it finds with metadata, so I made it ONLY by path)
  filter =[ "a|/dev/disk/by-path/.*|" ]

I wasn't sure what to reload, so I restarted all (might be too much)
$ sudo systemctl restart lvm2-lvmetad.service lvm2-monitor.service lvm2-lvmpolld.service

Now things work as you expected:
$ sudo /sbin/pvcreate /dev/disk/by-path/virtio-pci-0000:00:07.0
  Physical volume "/dev/disk/by-path/virtio-pci-0000:00:07.0" successfully created.
ubuntu@cosmic-storage-pools:~$ sudo pvdisplay
  "/dev/disk/by-path/pci-0000:00:07.0" is a new physical volume of "30.00 MiB"
  --- NEW Physical volume ---
  PV Name /dev/disk/by-path/pci-0000:00:07.0
  VG Name
  PV Size 30.00 MiB
  Allocatable NO
  PE Size 0
  Total PE 0
  Free PE 0
  Allocated PE 0
  PV UUID mwk7tq-fZ1C-zElW-u0Qy-KlqJ-PUK0-BVHOpz

That would fix things on the LVM side to be as libvirt expects.
Interestingly libvirt still can't start the pool now.

Haven't looked yet further, need to check what and how it parses in libvirt to get to the conclusion that it is not existing.
As now pvs looks like:
$ sudo /sbin/pvs --noheadings -o pv_name,vg_name
  /dev/disk/by-path/pci-0000:00:07.0 testpool1

[Christian Ehrhardt  (paelzer)]

lvm2 - config option intentionally starting safe, how to use it is outlined => invalid
libvirt - IMHO it should now work, need to debug more and wait for the mentioned upstream discussion => incomplete

[Christian Ehrhardt  (paelzer)]

This is again a case of please read all characters carefully :-)

definition from xml:
(gdb) p def->source.devices[i].path
$4 = 0x7f5204008e30 "/dev/disk/by-path/virtio-pci-0000:00:07.0"

compared against the array it built with the lvm tools
(gdb) p thisSource->devices[0]
$7 = {nfreeExtent = 0, freeExtents = 0x0, path = 0x7f51fc012b20 "/dev/disk/by-path/pci-0000:00:07.0", format = 0, part_separator = 0, geometry = {cylinders = 0, heads = 0, sectors = 0}}

That is different:
/dev/disk/by-path/virtio-pci-0000:00:07.0
/dev/disk/by-path/pci-0000:00:07.0

Lets further improve the lvm config we have.

[Christian Ehrhardt  (paelzer)]

I now used:
  filter =[ "a|/dev/disk/by-path/virtio-pci-*|" ]

Now it really is the same way on creation and query.
$ sudo /sbin/pvcreate /dev/disk/by-path/virtio-pci-0000:00:07.0
  Physical volume "/dev/disk/by-path/virtio-pci-0000:00:07.0" successfully created.
$ sudo pvdisplay
  "/dev/disk/by-path/virtio-pci-0000:00:07.0" is a new physical volume of "30.00 MiB"
  --- NEW Physical volume ---
  PV Name /dev/disk/by-path/virtio-pci-0000:00:07.0

And with that finally
$ sudo virsh pool-build testpool1
Pool testpool1 built

$ sudo virsh pool-start testpool1
Pool testpool1 started

I'd still be interested in the mail thread link.
But for your case, if you want to use by-path with LVM you should adapt the lvm.conf to use (only) these.

[Jan Marquardt (jm-artfiles)]

Christian, thank you for pointing me in the right direction. Adding

filter = [ "a|/dev/disk/by-path/pci-*|" ]

to /etc/lvm/lvm.conf solved this issue. (There is no need to restart any
service after changing this file. ;))

You can find the thread on the mailing list under this link:
https://www.redhat.com/archives/libvirt-users/2018-October/msg00059.html

I'll add a response there soon.

Best Regards

Jan

Am 06.11.18 um 15:48 schrieb  Christian Ehrhardt :
> I now used:
> filter =[ "a|/dev/disk/by-path/virtio-pci-*|" ]
>
> Now it really is the same way on creation and query.
> $ sudo /sbin/pvcreate /dev/disk/by-path/virtio-pci-0000:00:07.0
> Physical volume "/dev/disk/by-path/virtio-pci-0000:00:07.0" successfully created.
> $ sudo pvdisplay
> "/dev/disk/by-path/virtio-pci-0000:00:07.0" is a new physical volume of "30.00 MiB"
> --- NEW Physical volume ---
> PV Name /dev/disk/by-path/virtio-pci-0000:00:07.0
>
> And with that finally
> $ sudo virsh pool-build testpool1
> Pool testpool1 built
>
> $ sudo virsh pool-start testpool1
> Pool testpool1 started
>
> I'd still be interested in the mail thread link.
> But for your case, if you want to use by-path with LVM you should adapt the lvm.conf to use (only) these.
>
> ** Changed in: libvirt (Ubuntu)
> Status: Incomplete => Invalid
>

--
Artfiles New Media GmbH | Zirkusweg 1 | 20359 Hamburg
Tel: 040 - 32 02 72 90 | Fax: 040 - 32 02 72 95
E-Mail: <email address hidden> | Web: http://www.artfiles.de
Geschäftsführer: Harald Oltmanns | Tim Evers
Eingetragen im Handelsregister Hamburg - HRB 81478