nvidia driver limits to one single GPU per guest

[Sylvain Bauza]

If you want to provide a flavor with "resources:VGPU=2" (or more) and have compute nodes using nvidia cards (ie. having PCI devices that have a 16-bit vendor ID of "10de"), then QEMU throws an exception that is due to the nvidia driver not supporting more than 1 IOMMU group per guest.

libvirtError: internal error: qemu unexpectedly closed the monitor: 2018-03-22T13:14:39.272301Z qemu-kvm: -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/c949168d-d04d-4e74-925a-c38f3be11df5,bus=pci.0,addr=0x5: vfio warning: c949168d-d04d-4e74-925a-c38f3be11df5: Could not enable error recovery for the device
2018-03-22T13:14:39.273759Z qemu-kvm: -device vfio-pci,id=hostdev1,sysfsdev=/sys/bus/mdev/devices/f508c6d0-f859-4fa2-8976-94940e917709,bus=pci.0,addr=0x6: vfio error: f508c6d0-f859-4fa2-8976-94940e917709: error getting device from group 1: Operation not permitted
Verify all devices in group 1 are bound to vfio-<bus> or pci-stub and not already in use

Accordingly to that limitation, Nova should limit the maximum unit of possible resources per allocation depending on the PCI device vendor ID.

[Andrey Volkov (avolkov)]

Isn't that a thing could be solved with the max_unit property for Placement inventory?
https://developer.openstack.org/api-ref/placement/#update-resource-provider-inventories

[Sylvain Bauza (sylvain-bauza)]

@Andrey, no it needs to be depending on the vendor ID, since that problem is only related to the nvidia driver.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/562304

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/562304
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7529627488886f981a4c3d3d7994f92ef7956cee
Submitter: Zuul
Branch: master

commit 7529627488886f981a4c3d3d7994f92ef7956cee
Author: Sylvain Bauza <email address hidden>
Date: Tue Apr 17 15:46:05 2018 +0200

    libvirt: mdevs returning parent and vendor PCI info

    In order to provide inventories per physical GPUs, the libvirt driver needs to
    know the parent ID for each mediated device.
    Also, given we need to limit the max unit of vGPUs per guest to 1 if that's a
    nvidia GPU, we also need to pass the PCI vendor ID down to the driver.

    Related-Bug: #1758086
    Change-Id: Ib80af55ccf4e578e7f8b58531da1b1e95783c779

[Sylvain Bauza (sylvain-bauza)]

FWIW, tested with nvidia GRID10.2 release (R440.87) and the issue remains.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/906151