nvidia driver limits to one single GPU per guest

Bug #1758086 reported by Sylvain Bauza on 2018-03-22
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Low
Sylvain Bauza

Bug Description

If you want to provide a flavor with "resources:VGPU=2" (or more) and have compute nodes using nvidia cards (ie. having PCI devices that have a 16-bit vendor ID of "10de"), then QEMU throws an exception that is due to the nvidia driver not supporting more than 1 IOMMU group per guest.

libvirtError: internal error: qemu unexpectedly closed the monitor: 2018-03-22T13:14:39.272301Z qemu-kvm: -device vfio-pci,id=hostdev0,sysfsdev=/sys/bus/mdev/devices/c949168d-d04d-4e74-925a-c38f3be11df5,bus=pci.0,addr=0x5: vfio warning: c949168d-d04d-4e74-925a-c38f3be11df5: Could not enable error recovery for the device
2018-03-22T13:14:39.273759Z qemu-kvm: -device vfio-pci,id=hostdev1,sysfsdev=/sys/bus/mdev/devices/f508c6d0-f859-4fa2-8976-94940e917709,bus=pci.0,addr=0x6: vfio error: f508c6d0-f859-4fa2-8976-94940e917709: error getting device from group 1: Operation not permitted
Verify all devices in group 1 are bound to vfio-<bus> or pci-stub and not already in use

Accordingly to that limitation, Nova should limit the maximum unit of possible resources per allocation depending on the PCI device vendor ID.

Andrey Volkov (avolkov) wrote :

Isn't that a thing could be solved with the max_unit property for Placement inventory?
https://developer.openstack.org/api-ref/placement/#update-resource-provider-inventories

Sylvain Bauza (sylvain-bauza) wrote :

@Andrey, no it needs to be depending on the vendor ID, since that problem is only related to the nvidia driver.

Reviewed: https://review.openstack.org/562304
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7529627488886f981a4c3d3d7994f92ef7956cee
Submitter: Zuul
Branch: master

commit 7529627488886f981a4c3d3d7994f92ef7956cee
Author: Sylvain Bauza <email address hidden>
Date: Tue Apr 17 15:46:05 2018 +0200

    libvirt: mdevs returning parent and vendor PCI info

    In order to provide inventories per physical GPUs, the libvirt driver needs to
    know the parent ID for each mediated device.
    Also, given we need to limit the max unit of vGPUs per guest to 1 if that's a
    nvidia GPU, we also need to pass the PCI vendor ID down to the driver.

    Related-Bug: #1758086
    Change-Id: Ib80af55ccf4e578e7f8b58531da1b1e95783c779

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers