VGPU - I can only use one MIG profile in Nova
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Expired
|
Wishlist
|
Unassigned |
Bug Description
I have one Nvidia A100 card and can only use one MIG profile.
I divided the card into 4 different MIG profiles 2x A100-1-5C, 1x A100-2-10C, 1x A100-3-20C, As below.
+------
| MIG devices: |
+------
| GPU GI CI MIG | Memory-Usage | Vol| Shared |
| ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG|
| | | ECC| |
|======
| 0 2 0 0 | 10MiB / 20096MiB | 42 0 | 3 0 2 0 0 |
| | 0MiB / 32767MiB | | |
+------
| 0 3 0 1 | 6MiB / 9984MiB | 28 0 | 2 0 1 0 0 |
| | 0MiB / 16383MiB | | |
+------
| 0 9 0 2 | 4739MiB / 4864MiB | 14 0 | 1 0 0 0 0 |
| | 0MiB / 8191MiB | | |
+------
| 0 10 0 3 | 4739MiB / 4864MiB | 14 0 | 1 0 0 0 0 |
| | 0MiB / 8191MiB | | |
+------
My nova configuration: /etc/nova/nova.conf
[devices]
enabled_vgpu_types = nvidia-
[vgpu_nvidia-474]
device_addresses = 0000:61:
[vgpu_nvidia-475]
device_addresses = 0000:61:01.7
[vgpu_nvidia-476]
device_addresses = 0000:61:00.6
# openstack resource provider list
+------
| uuid | name | generation | root_provider_uuid | parent_
+------
| a0269b89-
| f2e5a4e0-
| a513c661-
| 9124d3a8-
| 5f443da2-
| 20da8814-
| 37014baa-
| a9e2f509-
| 202462de-
| 5e26d9e8-
| 315b5205-
| 208f395d-
| 7d5abf99-
| 315b2c35-
| c43be42e-
| 3cd4dbc7-
| 58fbbedb-
+------
Created Flavor:
openstack --os-placement-
openstack --os-placement-
openstack flavor create --private --description "vgpu-test" --ram $((8*1024)) --disk 0 --vcpus 8 vgpu-1 --project vgpu --property resources:VGPU=1 --property trait:CUSTOM_
openstack --os-placement-
openstack --os-placement-
openstack flavor create --private --description "vgpu-test" --ram $((8*1024)) --disk 0 --vcpus 8 vgpu-2 --project vgpu --property resources:VGPU=1 --property trait:CUSTOM_
openstack --os-placement-
openstack --os-placement-
openstack flavor create --private --description "vgpu-test" --ram $((8*1024)) --disk 0 --vcpus 8 vgpu-3 --project vgpu --property resources:VGPU=1 --property trait:CUSTOM_
openstack --os-placement-
openstack --os-placement-
openstack flavor create --private --description "vgpu-test" --ram $((8*1024)) --disk 0 --vcpus 8 vgpu-4 --project vgpu --property resources:VGPU=1 --property trait:CUSTOM_
gpu-a01:
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
'/sys/class/
nvidia-468 nvidia-469 nvidia-470 nvidia-471 nvidia-472 nvidia-473 nvidia-474 nvidia-475 nvidia-476 nvidia-477 nvidia-478 nvidia-706
Problem:
===========
I can create only two instances with A100-1-
If I edit the nova config and replace in /etc/nova/nova.conf
enabled_vgpu_types = nvidia-
on
enabled_vgpu_types = nvidia-
I will be able to use only one A100-2-
Packages:
============
Version: Ussuri
gpu-a01:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal
gpu-a01:~# dpkg -l | grep nova
ii nova-common 2:21.2.4-0ubuntu1 all OpenStack Compute - common files
ii nova-compute 2:21.2.4-0ubuntu1 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:21.2.4-0ubuntu1 all OpenStack Compute - compute node (KVM)
ii nova-compute-
ii python3-nova 2:21.2.4-0ubuntu1 all OpenStack Compute Python 3 libraries
ii python3-novaclient 2:17.0.0-0ubuntu1 all client library for OpenStack Compute API - 3.x
gpu-a01:~# uname -a
Linux compgpu-a01 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Nova does not impos any limits on the mig profiles or mdev types that can be used.
this is a hardware limitation of the nvida gpus not a nova limitation.
you are using A100 which does support using more then one type but only on specific types which they have documented in there product docs.
if you find that you cannot use the selced type then this is likely a result of using a unsuproted combination.
if they have documented that this should work you should file a bug with nvidia support.
you can find there documentiaon here
https:/ /docs.nvidia. com/datacenter/ tesla/mig- user-guide/ /docs.nvidia. com/datacenter/ tesla/mig- user-guide/ index.html# a100-profiles
https:/
i would suggest starting with one of the example toploies before trying your own to ensure it works.
https:/ /docs.nvidia. com/datacenter/ tesla/mig- user-guide/ index.html# create- gi
the nova community only provides enablement of the mdev based vgpu feature
we do not provide support for configuring the nvidia hardware and software.