GPU device override is not included in nova helm override by default

[Yang Liu]

Brief Description
-----------------
By default the two supported QAT pci-alias override is included in nova helm system overrides, which enables user to launch VM with QAT VF as is.

However, this override is not done for gpu devices, so user would have to first override the nova helm chart before launching GPU vm.
The GPU user override has to combine the existing user override, so it is fairly tedious to a regular user.

Severity
--------
Minor

Steps to Reproduce
------------------
- system helm-override-show stx-openstack nova openstack

| | pci: |
| | alias: |
| | type: multistring |
| | values: |
| | - '{"vendor_id": "8086", "product_id": "0435", "name": "qat-dh895xcc-pf"}' |
| | - '{"vendor_id": "8086", "product_id": "0443", "name": "qat-dh895xcc-vf"}' |
| | - '{"vendor_id": "8086", "product_id": "37c8", "name": "qat-c62x-pf"}' |
| | - '{"vendor_id": "8086", "product_id": "37c9", "name": "qat-c62x-vf"}' |
| | - '{"name": "gpu"}'

TC-name:

Expected Behavior
------------------
- gpu has similar overrides as qat with supported devices
Example:
'{"vendor_id": "102b", "product_id": "0522","device_type":"type-PF","name": "gpu"}'

Actual Behavior
----------------
- gpu override does not have any product_id/vendor_id in nova helm overrides

Reproducibility
---------------
Reproducible

System Configuration
--------------------
All

Branch/Pull Time/Commit
-----------------------
stx master as of 20200525 load

Last Pass
---------
Never

Timestamp/Logs
--------------
Easy to reproduce.

Test Activity
-------------
Regression Testing

[Ghada Khalil (gkhalil)]

stx.4.0 / medium priority - would be nice to fix to make it easier for users

[Jim Gauld (jgauld)]

The code that generates the static PCI aliases is here, it is a hard-coded dictionary based on constants.
./stx/openstack-armada-app/python-k8sapp-openstack/k8sapp_openstack/k8sapp_openstack/helm/nova.py

It would be very simple to add a list of well-defined aliases for very specific hardware (vendor_id, product_id, device_type) for each hardware we support.

For example, instead of having to apply this helm-override to specify matrox GPU, should define new constants in sysinv constants.py with for generic PCI alias PCI vendors/product/devices as we have for QAT.

cat << EOF > ./gpu_override.yaml
conf:
  nova:
    libvirt:
      cpu_mode: custom
      cpu_model: SandyBridge
    pci:
      alias:
        type: multistring
        values:
        - '{"vendor_id": "8086", "product_id": "0435", "name": "qat-dh895xcc-pf"}'
        - '{"vendor_id": "8086", "product_id": "0443", "name": "qat-dh895xcc-vf"}'
        - '{"vendor_id": "8086", "product_id": "37c8", "name": "qat-c62x-pf"}'
        - '{"vendor_id": "8086", "product_id": "37c9", "name": "qat-c62x-vf"}'
        - '{"name": "gpu"}'
        - '{"vendor_id": "102b", "product_id": "0522", "device_type": "type-PCI", "name": "matrox-g200e"}'
EOF

Although the names 'g200e' and T4 are unique, suggest we add specific GPU names for subset of known hardware, examples:
- matrox-g200e, with device_type type-PCI
- nvidia-t4-pf, with device_type type-PF

The PCI aliases are static, and are not a part of system device discovery.

[Jim Gauld (jgauld)]

For example, to support different Nvidia GPUs, there is different product_id and device type:
- '{"vendor_id": "10de", "product_id": "1b38", "device_type": "type-PCI", "name": "nvidia-tesla-p40"}'
- '{"vendor_id": "10de", "product_id": "1eb8", "device_type": "type-PF", "name": "nvidia-tesla-t4"}'

Just note that the "class_id" field is no longer supported in PCI alias.

To see what PCI attributes are actually matched against a given host:
Method 1: Instrument nova to dump out pci_stats.pools. (Jim Gauld has code for this).

Method 2: Query nova database table 'compute_nodes' field 'pci_stats' . Look at look at PciDevicePool, for each nova tracked pool. Should look like this :

"nova_object.name": "PciDevicePoolList",
"nova_object.data": {"objects": [{"nova_object.version": "1.1", "nova_object.changes": ["count", "numa_node", "vendor_id", "product_id", "tags"],
. .
"nova_object.name": "PciDevicePool", "nova_object.data":
 {"count": 1, "numa_node": 0, "vendor_id": "102b", "product_id": "0538", "tags": {"class_id": "030000", "dev_type": "type-PCI", "configured": "1"}},

[Jim Gauld (jgauld)]

Note that were are not currently getting any nova database data in 'collect', we are missing dumping mariadb.

To look at specific lab the pci_stats and other fields :
date; kubectl exec -it -n openstack mariadb-server-0 -- bash -c "mysql --password=\$MYSQL_DBADMIN_PASSWORD --user=root nova -e 'select host,numa_topology,pci_stats from compute_nodes;'"

Example :
date; kubectl exec -it -n openstack mariadb-server-0 -- bash -c "mysql --password=\$MYSQL_DBADMIN_PASSWORD --user=root nova -e 'select host,pci_stats from compute_nodes;'"
. .
| controller-0 | {"nova_object.version": "1.1", "nova_object.changes": ["objects"], "nova_object.name": "PciDevicePoolList", "nova_object.data": {"objects": [{"nova_object.version": "1.1", "nova_object.changes": ["count", "numa_node", "vendor_id", "product_id", "tags"], "nova_object.name": "PciDevicePool", "nova_object.data": {"count": 0, "numa_node": 0, "vendor_id": "102b", "product_id": "0522", "tags": {"dev_type": "type-PCI"}}, "nova_object.namespace": "nova"}, {"nova_object.version": "1.1", "nova_object.changes": ["count", "numa_node", "vendor_id", "product_id", "tags"], "nova_object.name": "PciDevicePool", "nova_object.data": {"count": 1, "numa_node": 1, "vendor_id": "8086", "product_id": "0435", "tags": {"dev_type": "type-PF"}}, "nova_object.namespace": "nova"}, {"nova_object.version": "1.1", "nova_object.changes": ["count", "numa_node", "vendor_id", "product_id", "tags"], "nova_object.name": "PciDevicePool", "nova_object.data": {"count": 32, "numa_node": 1, "vendor_id": "8086", "product_id": "0443", "tags": {"dev_type": "type-VF"}}, "nova_object.namespace": "nova"}]}, "nova_object.namespace": "nova"} |

The above is string json encoded nova object.

i.e., The Matrox g200e GPU
{"count": 0, "numa_node": 0, "vendor_id": "102b", "product_id": "0522", "tags": {"dev_type": "type-PCI"}}

i.e, the QAT vf
{"count": 32, "numa_node": 1, "vendor_id": "8086", "product_id": "0443", "tags": {"dev_type": "type-VF"}}

[sysadmin@controller-0 ~(keystone_admin)]$ system host-device-list controller-0
+------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+
| name | address | class id | vendor id | device id | class name | vendor name | device name | numa_node | enabled |
+------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------------+-----------+---------+
| pci_0000_08_00_0 | 0000:08:00.0 | 030000 | 102b | 0522 | VGA compatible controller | Matrox Electronics Systems Ltd. | MGA G200e [Pilot] ServerEngines (SEP1) | 0 | True |
| pci_0000_83_00_0 | 0000:83:00.0 | 0b4000 | 8086 | 0435 | Co-processor | Intel Corporation | DH895XCC Series QAT | 1 | True |
+------------------+--------------+----------+-----------+-----------+---------------------------+---------------------------------+----------------------------------...

[yong hu (yhu6)]

Suggest to continue addressing this LP in stx.4.0 maintenance releases.

[Ghada Khalil (gkhalil)]

As per 2020-07-30 release meeting, move to stx.5.0 as the goal of this is to make the GPU configuration easier. It's still functional in stx.4.0, but requires more steps.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/744245

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/744246

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jim Gauld (<email address hidden>) on branch: master
Review: https://review.opendev.org/744245
Reason: No changes required in constants.py ; new constants moved to openstack-armada-app repo. The existing constants are referenced in both sysinv and openstack-armada-app.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/744246
Committed: https://git.openstack.org/cgit/starlingx/openstack-armada-app/commit/?id=c86fae1b233a5b0595f218fe432f76b589dda2b5
Submitter: Zuul
Branch: master

commit c86fae1b233a5b0595f218fe432f76b589dda2b5
Author: Jim Gauld <email address hidden>
Date: Fri Jul 31 15:56:53 2020 -0400

    Add nova pci alias for GPUs Matrox G200E, NVidia M60, P40, T4

    This adds nova pci-alias definitions for these GPUs:
    - Matrox G200E (type-PCI), 'matrox-g200e'
    - NVidia Tesla M60 (type-PCI), 'nvidia-tesla-m60'
    - NVidia Tesla P40 (type-PCI), 'nvidia-tesla-p40'
    - NVidia Tesla T4 (type-PF), 'nvdia-tesla-t4-pf'

    The end user no longer needs to first override the nova helm chart
    to launch VMs with these GPUs.

    Previously the user needs to provide overrides like this:
    cat << EOF > ./gpu_override.yaml
    conf:
      nova:
        pci:
          alias:
            type: multistring
            values:
            - '{"vendor_id": "8086", "product_id": "0435", "name":
              "qat-dh895xcc-pf"}'
            - '{"vendor_id": "8086", "product_id": "0443", "name":
              "qat-dh895xcc-vf"}'
            - '{"vendor_id": "8086", "product_id": "37c8", "name":
              "qat-c62x-pf"}'
            - '{"vendor_id": "8086", "product_id": "37c9", "name":
              "qat-c62x-vf"}'
            - '{"name": "gpu"}'
            - '{"vendor_id": "102b", "product_id": "0522", "name":
              "matrox-g200e"}'
            - '{"vendor_id": "10de", "product_id": "13f2", "name":
              "nvidia-tesla-m60"}'
            - '{"vendor_id": "10de", "product_id": "1b38", "name":
              "nvidia-tesla-p40"}'
            - '{"vendor_id": "10de", "product_id": "1eb8",
              "device_type":
              "type-PF", "name": "nvidia-tesla-t4-pf"}'
    EOF

    system helm-override-update \
     --values ./gpu_override.yaml stx-openstack nova openstack --reuse-values
    system application-apply stx-openstack

    Closes-Bug: 1880997
    Signed-off-by: Jim Gauld <email address hidden>
    Change-Id: Iaa212351c13b9d279afff2d25dfeb1ffac0bb99d