vSFO cant find SRIOV interface. VMs using SRIOV interfaces cannot connect to interfaces

[Alexander Rubtsov]

Using Intel XL710 interfaces for SRIOV. During the testing phase, following issue is discoverd.
VM which is just restarted due to any reason, the VM may not come up properly. This is usually seen when the VM has undergone several restarts.

Following errors are seen in the console log of the VM, when the VMs fail to come up -

/usr/bin/sort_ports.py:Couldn't create port: Couldn't find interface name of PCI device at 0000:00:05.0
/usr/bin/sort_ports.py:Couldn't create port: Couldn't find interface name of PCI device at 0000:00:07.0
/usr/bin/sort_ports.py:Couldn't create port: Couldn't find interface name of PCI device at 0000:00:09.0
/usr/bin/sort_ports.py:Couldn't create port: Couldn't find interface name of PCI device at 0000:00:0b.0
/usr/bin/sort_ports.py:Couldn't create port: Couldn't find interface name of PCI device at 0000:00:0d.0

from the investigation at VNF side - The system finds all the interfaces and matches to the expected set of interfaces defined in
the VM's meta-data / personality file.

In case of SRIOV vSFO the meta-data defines 8 "front panel port" using SRIOV, and two vFAB ports (total 10 SRIOV ports, 5 from each
NIC), plus it looks for some internal VxLAN networks.

It then matches up the information with the devices actually seen so it knows which is which.

When the problem is happening, the vSFO VM is unable to match the interface name to PCI device.

That suggests failure to install/connect the i40evf-dpdk driver, and going by PCI numbering seems to be for all 5 of the expected ports from one NIC.

--------

# ethtool -i eth9
driver: i40e
version: 2.4.6
firmware-version: 6.02 0x80003620 1.1747.0
bus-info: 0000:d8:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

[Alexander Rubtsov (arubtsov)]

sla1 for 9.0-updates

[Roman Lubianyi (rlubianyi)]

Hi Alexander,

I have tried to reproduce the issue on our environment but the VM rebooted many times without errors.
But during an examination of customers logs, some errors were observed. Let's try the next steps to resolve the issue:

1) Recently we have updated i40e drivers to the 2.7.29 version - https://review.fuel-infra.org/#/c/40697/ . Is issue reproduced with them?

2) Could you please check that next recommendation from XL710 configuration guide fulfilled:

The VF driver automatically loads in the host operating system as soon as the VFs are created by
the PF driver. The VF driver claims newly-created VFs, and these VF are not available for Virtual
Machine (VM) assignment. There are two methods to overcome this scenario:
a. Unload the VF driver from within host operating systems by executing the following command in
Linux terminal with superuser (root) permission.
#rmmod i40evf
b. Blacklist VF driver by adding blacklist i40evf to the /lib/modprobe.d/dist-blacklist file

3) I the dmesg logs of the compute instance the next error observed "Unable to send opcode 2 to PF, err I40E_ERR_QUEUE_EMPTY, aq_err OK".

Did this error persist after doing paragraphs 1 and 2?