Incorrect driver allocated to a container for netdevice vf over vfio interface

[Difu Hu]

Brief Description
-----------------
When having netdevice vf over vfio interface, incorrect SR-IOV device plugin capacity/allocation occurs, and incorrect driver being allocated to a container.

Severity
--------
Major

Steps to Reproduce
------------------
Lab installed with netdevice vf over vfio interface
create container using the datanetwork on netdevice vf

Expected Behavior
------------------
correct driver is used in container and interface come up in container
$ kubectl exec test-sriovdp-netdev-single-pod-0 -n default -- dir /sys/bus/pci/devices/0000:05:0d.3/net/
net1

Actual Behavior
----------------
incorrect driver is used in container and interface did not come up in container
$ kubectl exec test-sriovdp-netdev-single-pod-0 -n default -- dir /sys/bus/pci/devices/0000:05:0d.3/net/
dir: cannot access /sys/bus/pci/devices/0000\:81\:08.6/net/: No such file or directory
command terminated with exit code 2

Reproducibility
---------------
Permanent

System Configuration
--------------------
Lab-name: wp_112

Branch/Pull Time/Commit
-----------------------
2020-06-27_00-41-42

Last Pass
---------
2020-06-23_20-00-00

Timestamp/Logs
--------------
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-show controller-0 sriov0
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | sriov0 |
| iftype | ethernet |
| ports | [u'ens785f1'] |
| imac | 3c:fd:fe:ac:65:79 |
| imtu | 1500 |
| ifclass | pci-sriov |
| ptp_role | none |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | f7ee41ed-c8da-4e9a-b73c-9e5f60d1126b |
| ihost_uuid | 31968bbe-ccaf-4004-aa5f-335fe6655560 |
| vlan_id | None |
| uses | [] |
| used_by | [u'sriovnet'] |
| created_at | |
| updated_at | |
| sriov_numvfs | 32 |
| sriov_vf_driver | vfio |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-show controller-0 sriovnet
+-----------------+--------------------------------------+
| Property | Value |
+-----------------+--------------------------------------+
| ifname | sriovnet |
| iftype | vf |
| ports | [] |
| imac | 3c:fd:fe:ac:65:79 |
| imtu | 1500 |
| ifclass | pci-sriov |
| ptp_role | none |
| aemode | None |
| schedpolicy | None |
| txhashpolicy | None |
| uuid | da3737d1-87e6-481a-a6ba-5d2f9aa973fa |
| ihost_uuid | 31968bbe-ccaf-4004-aa5f-335fe6655560 |
| vlan_id | None |
| uses | [u'sriov0'] |
| used_by | [] |
| created_at | |
| updated_at | |
| sriov_numvfs | 16 |
| sriov_vf_driver | netdevice |
| accelerated | [True] |
+-----------------+--------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-datanetwork-list controller-0
+--------------+--------------------------------------+----------+------------------+
| hostname | uuid | ifname | datanetwork_name |
+--------------+--------------------------------------+----------+------------------+
| controller-0 | 51b4dd7a-b576-4b2b-9d3c-6ac698698134 | sriov0 | group0-data0 |
| controller-0 | e2ffe7a4-6fa3-450a-bac8-d9b2f1a0e344 | sriovnet | group0-data1 |
+--------------+--------------------------------------+----------+------------------+

# NetworkAttachmentDefinition yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_group0_data1
  name: test-sriovdp-netdev-connectivity-controller-0-group0-data1-vf1
spec:
  config: '{"\"ipam\"": {"\"assign_ipv6\"": "\"true\"", "\"type\"": "\"calico-ipam\"",
    "\"ipv6_pools\"": "[\"sriov-pool-group0-data1-vf1\"]", "\"subnet\"": "\"2626:1::/64\""},
    "\"type\"": "\"sriov\"", "\"cniVersion\"": "\"0.3.0\"", "\"kubernetes\"": {"\"kubeconfig\"":
    "\"/etc/cni/net.d/calico-kubeconfig\""}, "\"datastore_type\"": "\"kubernetes\""}'

# pod yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/ipv6pools: '"[\"default-ipv6-ippool\"]"'
    k8s.v1.cni.cncf.io/networks: test-sriovdp-netdev-connectivity-controller-0-group0-data1-vf1
  name: test-sriovdp-netdev-connectivity-0
spec:
  containers:
  - args:
    - while true; do sleep 300000; done;
    command:
    - /bin/bash
    - -c
    - --
    image: centos/tools
    imagePullPolicy: IfNotPresent
    name: test-sriovdp-netdev-connectivity-0
    resources:
      limits:
        intel.com/pci_sriov_net_group0_data1: 1
      requests:
        intel.com/pci_sriov_net_group0_data1: 1
    securityContext:
      capabilities:
        add:
        - IPC_LOCK
        - NET_ADMIN
        - NET_RAW
  nodeSelector:
    kubernetes.io/hostname: controller-0

$ kubectl exec test-sriovdp-netdev-connectivity-0 -n default -- printenv | grep PCIDEVICE | awk -FPCIDEVICE_INTEL_COM_PCI_SRIOV_NET_ '{print $2}'
GROUP0_DATA1=0000:05:0d.3

# kubectl exec test-sriovdp-netdev-connectivity-0 -n default -- dir /sys/bus/pci/devices/0000:05:0d.3/net/
dir: cannot access /sys/bus/pci/devices/0000\:05\:0d.3/net/: No such file or directory
command terminated with exit code 2

Test Activity
-------------
Regression Testing

Workaround
----------
Configure the parent SRIOV interface to use the netdevice VF driver and the child to use vfio

[Ghada Khalil (gkhalil)]

Low priority - can be avoided. See the workaround.

[Ghada Khalil (gkhalil)]

Raising the priority and marking for stx.5.0 as the workaround requires users to re-do their host config which is disruptive.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/759165

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759165
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=89799c44a0657aa795cff5739f4eaa1024b6fd2e
Submitter: Zuul
Branch: master

commit 89799c44a0657aa795cff5739f4eaa1024b6fd2e
Author: Steven Webster <email address hidden>
Date: Wed Oct 21 10:26:53 2020 -0400

    Use kernel module as port virtual function driver

    For network interfaces, the supported kernel module should be
    used as the virtual function (VF) driver for the underlying
    port. This allows a determination of the driver to use when
    one selects an interface VF driver of 'netdevice'

    This commit fixes an issue that can occur if the parent
    interface in an SR-IOV 'split-nic' scenario has a driver of
    vfio rather than netdevice. In that case, the port's VF
    driver was erroneously set to vfio-pci, which would be used
    instead of the supported kernel device for the set of netdevice
    VFs on the interface.

    Change-Id: Ie863e32d04f9636d52a65ce4d56c441f6b69e3a9
    Closes-Bug: 1885446
    Signed-off-by: Steven Webster <email address hidden>

[Difu Hu (difuhu)]

Verified on: 2020-06-27_18-35-20 + patch7