Charm doesn't initialise SRIOV gpu devices on nvidia-gpu versions >= 11.0 (was: charm does not create gpu virtual functions)

[Andy Wu]

Tested this on node with Nvidia Tesla A10 card with vGPU software: nvidia-vgpu-ubuntu-525_525.85.07_amd64.deb

channel : yoga/stable
OS: jammy

After attaching vGPU driver to nova-compute-nvidia-vgpu and reboot the node, the nova-compute-nvidia-vgpu unit is active with status : Unit is ready: NVIDIA GPU found; installed NVIDIA software: 525.85.07

Execute nvidia-smi on the node confirms driver is intalled successfully

However juju run-action --wait nova-compute-nvidia-vgpu/0 list-vgpu-types does not return anything

ubuntu@ps6-infra1:~$ juju run-action --wait nova-compute-nvidia-vgpu/5 list-vgpu-types
unit-nova-compute-nvidia-vgpu-5:
  UnitId: nova-compute-nvidia-vgpu/5
  id: "346"
  results:
    output: ""
  status: completed

Inside the node, gpu card bus info is 25:00.0
   ubuntu@ps6-rb2-n1:~$ lspci -nn | grep -i Nvidia
   25:00.0 3D controller [0302]: NVIDIA Corporation GA102GL [A10] [10de:2236] (rev a1)

But no virtual functions are created

  cd /sys/bus/pci/devices/0000\:25\:00.0/
  ls | grep virtfn

I need create virtual funciton manually
   /usr/lib/nvidia/sriov-manage -e 0000:25:00.0

after that I can see virtual functions
  ls | grep virtfn
   virtfn0
   virtfn1
   virtfn10
   virtfn11

Re-run list-vpu-types

ubuntu@ps6-infra1:~$ juju run-action --wait nova-compute-nvidia-vgpu/5 list-vgpu-types
unit-nova-compute-nvidia-vgpu-5:
  UnitId: nova-compute-nvidia-vgpu/5
  id: "348"
  results:
    output: |-
      nvidia-588, 0000:25:02.3, NVIDIA A10-1B, num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
      nvidia-589, 0000:25:02.3, NVIDIA A10-2B, num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
      nvidia-590, 0000:25:02.3, NVIDIA A10-1Q, num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
      nvidia-591, 0000:25:02.3, NVIDIA A10-2Q, num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12

[Alex Kavanagh (ajkavanagh)]

Hi Andy

Unfortunately, there's some more information we need please. Please could you have a read of https://docs.openstack.org/charm-guide/latest/community/software-bug.html- obviously, not everything will be relevant, but it would be good to get log files from all the relevant software, etc.

Thanks.

[Andy Wu (qch2012)]

cli output capture in https://pastebin.canonical.com/p/fBpkgkVc86/

also uploaded some logs , please let me know if you need anything else

[Alex Kavanagh (ajkavanagh)]

Andy, is it possible that the sriov-manage command is a new requirement for nvidia vgpu drivers; there's nothing in the current charm software that mentions it, and it only seems to appear in nvidia software from version 11.00: https://docs.nvidia.com/grid/index.html

[Andy Wu (qch2012)]

Possbily. Nvidia guide mentions there are two ways of creating vgpu, first one is to to use mdev for legancy gpu that does not support sriov. second one is to use sriov-manage for gpu that support sriov such as a GPU based on the NVIDIA Ampere architecture, like the A10 card I am testing

https://docs.nvidia.com/grid/15.0/grid-vgpu-user-guide/index.html#creating-vgpu-device-red-hat-el-kvm

[Alex Kavanagh (ajkavanagh)]

Based on https://docs.nvidia.com/grid/index.html and the driver you mentioned at the top (nvidia-vgpu-ubuntu-525_525.85.07_amd64.deb) you are running driver version 15.1.

Thus, I think I can conclude that the nova-compute-nvidia-vgpu charm doesn't fully setup that version of the software, hence your need to perform the manual step.

For future travellers:

* The charm ought to detect the version of the nvidia payload software and display a warning on the status line if it doesn't officially support that version: aka it hasn't actually been tested with it.
* To manually fix, use the sriov-manage command to initialise the driver completely.

[Alex Kavanagh (ajkavanagh)]

Triaged medium as there is a work-around.

[DUFOUR Olivier (odufourc)]

Subscribed ~Field-critical.

The workaround doesn't seem persistent across reboot of the server.
Meaning the command "sriov-manage" needs to be run at least once on each boot of the server to initialise the virtual GPUs.

[James Page (james-page)]

The original enablement was done with the 14 series release; so right now the supported versions of the vGPU driver would start with the prefix 510.y.z - the most recent being 510.108.03.

Would it be possible to test with this version please? Supporting a whole new major release of the driver (and from what we've seen some associated behavioural differences) is not something we should be looking todo as a bug fix under Field SLA.

[James Page (james-page)]

As for reference when the original enablement was done we only tested using mediated devices - I don't think the SR-IOV support in vGPU's was a feature last year.

[Billy Olsen (billy-olsen)]

The vGPU enablement support within the nova-compute-nvidia-gpu charm was designed around the idea of mediated devices in conjunction with the upstream Nova documentation and implementation identified here https://docs.openstack.org/nova/zed/admin/virtual-gpu.html.

The gpu card's SR-IOV capabilities were not included in this work and will need to be worked into the plan in the future. Note, the mediated device bits were patched in the HWE kernel in order to continue to support the mediated device story.

Additionally, you will find that the OpenStack Nova code is designed to use mediated devices for setting up devices again and attaching them to the right parent devices. Libvirt has the option of persistent mdev devices, but this is not leveraged by Nova in its current state.

The sriov formatted vgpu devices remain an untested feature at this point in time from the OpenStack perspective and guarantees are not provided. The time-sliced mediated devices options were the ones implemented.

-- Note --

The work-around provided in the description allows for the card to be observed appropriately and can potentially be used. However, as odufourc noted in comment #7, this is not persisted across reboots of the server. This can be worked-around by using Juju's cloud-init snippet for instances and running the command in the `per-boot` section.

You should be able to leverage this with the following snippet in the `per-boot` section:

for card in `lspci -nn | awk '/NVIDIA/ {print "0000:" $1}`; do
    echo "Enabling /usr/lib/nvidia/sriov-manage -e 0000:$card;
done