[2.x] Consider using ID_SERIAL if ID_SERIAL_SHORT is not set for virtio storage devices

[Dmitrii Shcherbakov]

Created a domain with a number of different devices (virtio-blk, virtio-scsi, sata, NVMe) and added <serial/> tag to all of them for testing of lp:1735839. Querying udev, I can see that ID_SERIAL is set but MAAS does not use it to populate the "serial" field during commissioning.

The reason is that only ID_SERIAL_SHORT is collected by MAAS, not ID_SERIAL and it is not present for virtio-blk devices (even if it is provided via QEMU).

https://paste.ubuntu.com/p/pM7Tx8Hgbw/

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/sec1/libvirt-images/maas-vhost6-virtio0.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>disk2</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/sec1/libvirt-images/maas-vhost6-virtio1.qcow2'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <serial>disk3</serial>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </disk>

ubuntu@maas-vhost6:~$ udevadm info --query=property --name /dev/vda | grep ID_SERIAL
ID_SERIAL=disk2

ubuntu@maas-vhost6:~$ udevadm info --query=property --name /dev/vdb | grep ID_SERIAL
ID_SERIAL=disk3

Looking at the full output I can see that in the udev output there is no ID_SERIAL_SHORT for virtio devices:

udevadm info --query=property --name /dev/vda
DEVLINKS=/dev/disk/by-path/virtio-pci-0000:06:00.0 /dev/disk/by-dname/vda /dev/disk/by-path/pci-0000:06:00.0 /dev/disk/by-id/virtio-disk2
DEVNAME=/dev/vda
DEVPATH=/devices/pci0000:00/0000:00:02.5/0000:06:00.0/virtio5/block/vda
DEVTYPE=disk
ID_PATH=pci-0000:06:00.0
ID_PATH_TAG=pci-0000_06_00_0
ID_SERIAL=disk2
MAJOR=252
MINOR=0
SUBSYSTEM=block
TAGS=:systemd:
USEC_INITIALIZED=1789047

And ID_SERIAL_SHORT is something that MAAS collects in provisioning scripts for block devices:

https://github.com/maas/maas/blob/2.5.0-rc2/src/provisioningserver/refresh/node_info_scripts.py#L513-L521
    # Grab the device path, serial number, and sata connection.
    UDEV_MAPPINGS = {
        "DEVNAME": "PATH",
        "DEVPATH": "DEVPATH",
        "ID_SERIAL_SHORT": "SERIAL",

As a result the serial field is empty for virtio devices (noticed first by looking at the UI):

https://paste.ubuntu.com/p/RzMk7r5ZGH/

        {
            "firmware_version": null,
            "name": "vda",
            "filesystem": null,
            "id_path": "/dev/vda",
            "block_size": 2048,
            "system_id": "xeqrrw",
# ...
            "path": "/dev/disk/by-dname/vda",
# ...
            "serial": "",

[Dmitrii Shcherbakov (dmitriis)]

For a virtio-blk device without a serial specified in domain xml, ID_SERIAL is not auto-generated based on businfo:

udevadm info --query=property --name /dev/vda
DEVLINKS=/dev/disk/by-path/pci-0000:06:00.0 /dev/disk/by-path/virtio-pci-0000:06:00.0
DEVNAME=/dev/vda
DEVPATH=/devices/pci0000:00/0000:00:02.5/0000:06:00.0/virtio5/block/vda
DEVTYPE=disk
ID_PATH=pci-0000:06:00.0
ID_PATH_TAG=pci-0000_06_00_0
MAJOR=252
MINOR=0
SUBSYSTEM=block
TAGS=:systemd:
USEC_INITIALIZED=69719445

This is in contrast to virtio-scsi devices for which it is:

# no serial - businfo based ID_SERIAL and ID_SERIAL_SHORT are generated automatically
ubuntu@maas-vhost6:~$ udevadm info --query=property --name /dev/sda | grep ID_SERIAL
ID_SERIAL=0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0
ID_SERIAL_SHORT=drive-scsi0-0-0-0

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/sec1/libvirt-images/vhost6.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

# has serial - ID_SERIAL and ID_SERIAL_SHORT are used based on the serial provided to qemu via libvirt
ubuntu@maas-vhost6:~$ udevadm info --query=property --name /dev/sdb | grep ID_SERIAL
ID_SERIAL=0QEMU_QEMU_HARDDISK_disk0
ID_SERIAL_SHORT=disk0

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/sec1/libvirt-images/maas-vhost6.qcow2'/>
      <target dev='sdb' bus='scsi'/>
      <serial>disk0</serial>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>

[Andres Rodriguez (andreserl)]

The reason for this is that SERIAL is a made up name. It’s not the actual
serial of a device. The SERIAL_SHORT is the actual serial name.

I think MAAS already creates serial for the devices it creates, I would
have to check, but I don’t think we can rely on using SERIAL because it’s
jot the actual serial of the disk, it is again, just a made up name
assigned from the software perspective.

On Sat, Dec 8, 2018 at 10:35 AM Dmitrii Shcherbakov <
<email address hidden>> wrote:

> ** Description changed:
>
> Created a domain with a number of different devices (virtio-blk, virtio-
> scsi, sata, NVMe) and added <serial/> tag to all of them for testing of
> lp:1735839. Querying udev, I can see that ID_SERIAL is set but MAAS does
> not use it to populate the "serial" field during commissioning.
> +
> + The reason is that only ID_SERIAL_SHORT is collected by MAAS, not
> + ID_SERIAL.
>
> https://paste.ubuntu.com/p/pM7Tx8Hgbw/
>
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2'/>
> <source file='/mnt/sec1/libvirt-images/maas-vhost6-virtio0.qcow2'/>
> <backingStore/>
> <target dev='vda' bus='virtio'/>
> <serial>disk2</serial>
> <alias name='virtio-disk0'/>
> <address type='pci' domain='0x0000' bus='0x06' slot='0x00'
> function='0x0'/>
> </disk>
> <disk type='file' device='disk'>
> <driver name='qemu' type='qcow2'/>
> <source file='/mnt/sec1/libvirt-images/maas-vhost6-virtio1.qcow2'/>
> <backingStore/>
> <target dev='vdb' bus='virtio'/>
> <serial>disk3</serial>
> <alias name='virtio-disk1'/>
> <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
> function='0x0'/>
> </disk>
>
> ubuntu@maas-vhost6:~$ udevadm info --query=property --name /dev/vda |
> grep ID_SERIAL
> ID_SERIAL=disk2
>
> ubuntu@maas-vhost6:~$ udevadm info --query=property --name /dev/vdb |
> grep ID_SERIAL
> ID_SERIAL=disk3
>
> Looking at the full output I can see that in the udev output there is no
> ID_SERIAL_SHORT for virtio devices:
>
> udevadm info --query=property --name /dev/vda
> DEVLINKS=/dev/disk/by-path/virtio-pci-0000:06:00.0
> /dev/disk/by-dname/vda /dev/disk/by-path/pci-0000:06:00.0
> /dev/disk/by-id/virtio-disk2
> DEVNAME=/dev/vda
> DEVPATH=/devices/pci0000:00/0000:00:02.5/0000:06:00.0/virtio5/block/vda
> DEVTYPE=disk
> ID_PATH=pci-0000:06:00.0
> ID_PATH_TAG=pci-0000_06_00_0
> ID_SERIAL=disk2
> MAJOR=252
> MINOR=0
> SUBSYSTEM=block
> TAGS=:systemd:
> USEC_INITIALIZED=1789047
>
> And ID_SERIAL_SHORT is something that MAAS collects in provisioning
> scripts for block devices:
>
>
> https://github.com/maas/maas/blob/2.5.0-rc2/src/provisioningserver/refresh/node_info_scripts.py#L513-L521
> # Grab the device path, serial number, and sata connection.
> UDEV_MAPPINGS = {
> "DEVNAME": "PATH",
> "DEVPATH": "DEVPATH",
> "ID_SERIAL_SHORT": "SERIAL",
>
> As a result the serial field is empty for virtio devices (noticed first
> by looking at the UI):
>
> https://paste.ubuntu.com/p/RzMk7r5ZGH/
>
> {
> "f...

[Andres Rodriguez (andreserl)]

Also, please test disk creating virtual
Machines from MAAS as that is now our supported path for KVM in MAAS!

On Sat, Dec 8, 2018 at 11:03 AM Andres Rodriguez <email address hidden>
wrote:

> The reason for this is that SERIAL is a made up name. It’s not the actual
> serial of a device. The SERIAL_SHORT is the actual serial name.
>
> I think MAAS already creates serial for the devices it creates, I would
> have to check, but I don’t think we can rely on using SERIAL because it’s
> jot the actual serial of the disk, it is again, just a made up name
> assigned from the software perspective.
>
>
>
> On Sat, Dec 8, 2018 at 10:35 AM Dmitrii Shcherbakov <
> <email address hidden>> wrote:
>
>> ** Description changed:
>>
>> Created a domain with a number of different devices (virtio-blk, virtio-
>> scsi, sata, NVMe) and added <serial/> tag to all of them for testing of
>> lp:1735839. Querying udev, I can see that ID_SERIAL is set but MAAS does
>> not use it to populate the "serial" field during commissioning.
>> +
>> + The reason is that only ID_SERIAL_SHORT is collected by MAAS, not
>> + ID_SERIAL.
>>
>> https://paste.ubuntu.com/p/pM7Tx8Hgbw/
>>
>> <disk type='file' device='disk'>
>> <driver name='qemu' type='qcow2'/>
>> <source
>> file='/mnt/sec1/libvirt-images/maas-vhost6-virtio0.qcow2'/>
>> <backingStore/>
>> <target dev='vda' bus='virtio'/>
>> <serial>disk2</serial>
>> <alias name='virtio-disk0'/>
>> <address type='pci' domain='0x0000' bus='0x06' slot='0x00'
>> function='0x0'/>
>> </disk>
>> <disk type='file' device='disk'>
>> <driver name='qemu' type='qcow2'/>
>> <source
>> file='/mnt/sec1/libvirt-images/maas-vhost6-virtio1.qcow2'/>
>> <backingStore/>
>> <target dev='vdb' bus='virtio'/>
>> <serial>disk3</serial>
>> <alias name='virtio-disk1'/>
>> <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
>> function='0x0'/>
>> </disk>
>>
>> ubuntu@maas-vhost6:~$ udevadm info --query=property --name /dev/vda |
>> grep ID_SERIAL
>> ID_SERIAL=disk2
>>
>> ubuntu@maas-vhost6:~$ udevadm info --query=property --name /dev/vdb |
>> grep ID_SERIAL
>> ID_SERIAL=disk3
>>
>> Looking at the full output I can see that in the udev output there is no
>> ID_SERIAL_SHORT for virtio devices:
>>
>> udevadm info --query=property --name /dev/vda
>> DEVLINKS=/dev/disk/by-path/virtio-pci-0000:06:00.0
>> /dev/disk/by-dname/vda /dev/disk/by-path/pci-0000:06:00.0
>> /dev/disk/by-id/virtio-disk2
>> DEVNAME=/dev/vda
>> DEVPATH=/devices/pci0000:00/0000:00:02.5/0000:06:00.0/virtio5/block/vda
>> DEVTYPE=disk
>> ID_PATH=pci-0000:06:00.0
>> ID_PATH_TAG=pci-0000_06_00_0
>> ID_SERIAL=disk2
>> MAJOR=252
>> MINOR=0
>> SUBSYSTEM=block
>> TAGS=:systemd:
>> USEC_INITIALIZED=1789047
>>
>> And ID_SERIAL_SHORT is something that MAAS collects in provisioning
>> scripts for block devices:
>>
>>
>> https://github.com/maas/maas/blob/2.5.0-rc2/src/provisioningserver/refresh/node_info_scripts.py#L513-L521
>> # Grab the device path, serial number, and sata connection.
>> UDEV_MAPPINGS = {
>>...

[Andres Rodriguez (andreserl)]

Also, I think it is fair to say that MAAS requires / uses the
ID_SERIAL_SHORT and that’s the one you should be setting on your virtual
machine.

On Sat, Dec 8, 2018 at 11:04 AM Andres Rodriguez <email address hidden>
wrote:

> Also, please test disk creating virtual
> Machines from MAAS as that is now our supported path for KVM in MAAS!
>
> On Sat, Dec 8, 2018 at 11:03 AM Andres Rodriguez <email address hidden>
> wrote:
>
>> The reason for this is that SERIAL is a made up name. It’s not the actual
>> serial of a device. The SERIAL_SHORT is the actual serial name.
>>
>> I think MAAS already creates serial for the devices it creates, I would
>> have to check, but I don’t think we can rely on using SERIAL because it’s
>> jot the actual serial of the disk, it is again, just a made up name
>> assigned from the software perspective.
>>
>>
>>
>> On Sat, Dec 8, 2018 at 10:35 AM Dmitrii Shcherbakov <
>> <email address hidden>> wrote:
>>
>>> ** Description changed:
>>>
>>> Created a domain with a number of different devices (virtio-blk,
>>> virtio-
>>> scsi, sata, NVMe) and added <serial/> tag to all of them for testing of
>>> lp:1735839. Querying udev, I can see that ID_SERIAL is set but MAAS
>>> does
>>> not use it to populate the "serial" field during commissioning.
>>> +
>>> + The reason is that only ID_SERIAL_SHORT is collected by MAAS, not
>>> + ID_SERIAL.
>>>
>>> https://paste.ubuntu.com/p/pM7Tx8Hgbw/
>>>
>>> <disk type='file' device='disk'>
>>> <driver name='qemu' type='qcow2'/>
>>> <source
>>> file='/mnt/sec1/libvirt-images/maas-vhost6-virtio0.qcow2'/>
>>> <backingStore/>
>>> <target dev='vda' bus='virtio'/>
>>> <serial>disk2</serial>
>>> <alias name='virtio-disk0'/>
>>> <address type='pci' domain='0x0000' bus='0x06' slot='0x00'
>>> function='0x0'/>
>>> </disk>
>>> <disk type='file' device='disk'>
>>> <driver name='qemu' type='qcow2'/>
>>> <source
>>> file='/mnt/sec1/libvirt-images/maas-vhost6-virtio1.qcow2'/>
>>> <backingStore/>
>>> <target dev='vdb' bus='virtio'/>
>>> <serial>disk3</serial>
>>> <alias name='virtio-disk1'/>
>>> <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
>>> function='0x0'/>
>>> </disk>
>>>
>>> ubuntu@maas-vhost6:~$ udevadm info --query=property --name /dev/vda |
>>> grep ID_SERIAL
>>> ID_SERIAL=disk2
>>>
>>> ubuntu@maas-vhost6:~$ udevadm info --query=property --name /dev/vdb |
>>> grep ID_SERIAL
>>> ID_SERIAL=disk3
>>>
>>> Looking at the full output I can see that in the udev output there is
>>> no
>>> ID_SERIAL_SHORT for virtio devices:
>>>
>>> udevadm info --query=property --name /dev/vda
>>> DEVLINKS=/dev/disk/by-path/virtio-pci-0000:06:00.0
>>> /dev/disk/by-dname/vda /dev/disk/by-path/pci-0000:06:00.0
>>> /dev/disk/by-id/virtio-disk2
>>> DEVNAME=/dev/vda
>>> DEVPATH=/devices/pci0000:00/0000:00:02.5/0000:06:00.0/virtio5/block/vda
>>> DEVTYPE=disk
>>> ID_PATH=pci-0000:06:00.0
>>> ID_PATH_TAG=pci-0000_06_00_0
>>> ID_SERIAL=disk2
>>> MAJOR=252
>>> MINOR=0
>>> SUBSYSTEM=block
>>> TAGS=:systemd:
>>> USEC_INITIALIZED=17890...

[Dmitrii Shcherbakov (dmitriis)]

Andres,

It's the same with virsh pods because VMs are created with the serial field set to a generated UUID on the MAAS side and the device type used by MAAS for block devices is virtio-blk (just like I did by hand above):

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/maas-images/7bee76ba-b5ef-4113-8f33-27f26830fe84'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>7bee76ba-b5ef-4113-8f33-27f26830fe84</serial>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>

And this results in a serial being passed into qemu args (as expected):

pgrep -af qemu
6058 /usr/bin/qemu-system-x86_64 -name guest=testvm,debug-threads=on (...) -drive file=/var/lib/libvirt/maas-images/7bee76ba-b5ef-4113-8f33-27f26830fe84,format=raw,if=none,id=drive-virtio-disk0,serial=7bee76ba-b5ef-4113-8f33-27f26830fe84 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 (...)

# NOTE: the serial field is truncated by QEMU to VIRTIO_BLK_ID_BYTES=20 https://git.io/fp9ma
udevadm info --query=property --name /dev/vda
DEVLINKS=/dev/disk/by-id/virtio-7bee76ba-b5ef-4113-8 /dev/disk/by-path/pci-0000:00:07.0 /dev/disk/by-path/virtio-pci-0000:00:07.0 /dev/disk/by-dname/vda
DEVNAME=/dev/vda
DEVPATH=/devices/pci0000:00/0000:00:07.0/virtio4/block/vda
DEVTYPE=disk
ID_PART_TABLE_TYPE=dos
ID_PART_TABLE_UUID=44a59b19
ID_PATH=pci-0000:00:07.0
ID_PATH_TAG=pci-0000_00_07_0
ID_SERIAL=7bee76ba-b5ef-4113-8
MAJOR=252
MINOR=0
SUBSYSTEM=block
TAGS=:systemd:
USEC_INITIALIZED=2050503

ubuntu@testvm:~$ cat /sys/class/block/vda/serial
7bee76ba-b5ef-4113-8

Technically, virtio-blk is a PCI device (not a SATA or SCSI device) so its handling is different when it comes to udev.

https://github.com/systemd/systemd/commit/2d000ed9d85eb6dd8a535e5d5e8f7a5ef52bd794 (and this udev/systemd commit by Ryan seems to be quite straightforward - I could not find any reference to serial_short being exposed by virtio-blk).

In other words: the way I passed the serial above is the same as MAAS uses in pods. There is no other way for virtio-blk devices because wwn is not supported for virtio-blk:

virsh start testvm
error: Failed to start domain testvm
error: unsupported configuration: Only ide and scsi disk support wwn

If it was a scsi device (e.g. virtio scsi controller + a scsi device) the udev logic would be different and pages 83 and 80 would have been queried to retrieve serial and wwn:

https://github.com/systemd/systemd/blob/v237/src/udev/scsi_id/scsi_serial.c#L607-L619

[Andres Rodriguez (andreserl)]

In a physical system, the the 'ID_SERIAL' is auto-generated based on names. This is not the true serial of an actual physical device. It is auto-generated. I've seen other devices where ID_SERIAL only shows a name and not the actual device serial. For example, see below:

ubuntu@node04:~$ sudo udevadm info --query=property --name /dev/sda | grep SERIAL
ID_SERIAL=Samsung_SSD_850_EVO_M.2_120GB_S24ANXAH102622M
ID_SERIAL_SHORT=S24ANXAH102622M

As such, it seems to be that VM's using virtio devices with a 'serial' specified should *also* be setting ID_SERIAL_SHORT and not just ID_SERIAL. It seems to me where this should be fixed is in KVM and not work around in MAAS.

Setting this as opinion as I check with my team and talk to others about this, as again, I don't think MAAS should rely on ID_SERIAL for anything.

[Dmitrii Shcherbakov (dmitriis)]

I agree in general but that will take some time.

Another option is to use virtio-scsi controllers and SCSI block devices for VMs by default in MAAS instead of virtio-blk. Besides wwn support, using a virtio-scsi controller will only consume 1 virtual PCI lane (there are only 32 available in total per virtual PCI bus minus the ones used for default devices) and there will be no practical limit on SCSI storage devices per that controller.