Failure to get the correct UpLink Representor

[Mamduh]

[Impact]
An update to the mlx5_core driver [1] which will be made available to users of stable releases both through HWE kernels and DKMS packages provided by NVIDIA/Mellanox [2] makes some assumptions about sysfs layout made by OS-VIF and Libvirt apparent.

To allow users with this hardware to continue to enjoy their existing systems with the most recent drivers updates are required to os-vif and libvirt.

Without this update these systems will stop functioning when upgrading to the new mlx5_core driver.

[Test Plan]
Note: Hardware making use of the mlx5_core driver with support for HWOL is required to test these changes.

1. Deploy OpenStack on machines with HWOL enabled using kernel without [1]
2. Create an instance using an HWOL port
3. Confirm the instance can start and that it has connectivity
4. Upgrade to kernel with [1] and re-confirm

[Regression Potential]
For OS-VIF the changes are made to code paths used exclusively by consumers of this type of hardware and HWOL enabled. They are also made in a backward compatible way so that it works both with the old and new driver.

For Libvirt the change is made in such a way that it will behave as before when used to look up hardware that populates net/phys_port_id. When used with hardware that do not populate net/phys_port_id but use net_phys_port_name instead, which is typical for the hardware in question, the new behavior is used.

[Original Bug Description]
Due to new kernel patch here [1], the PF and VF representors are linked to their parent PCI device.

Old Structure:
The structure of VF's PCI Address/physfn/net contains only the PF of that VF

$ ls /sys/bus/pci/devices/<vf-pci-addre>/physfn/net/
enp2s0f0

$ ls -l /sys/class/net
...
lrwxrwxrwx 1 root root 0 Aug 17 11:11 enp2s0f0_0 -> ../../devices/virtual/net/enp2s0f0_0
lrwxrwxrwx 1 root root 0 Aug 17 11:11 enp2s0f0_1 -> ../../devices/virtual/net/enp2s0f0_1
lrwxrwxrwx 1 root root 0 Aug 17 11:11 enp2s0f0_2 -> ../../devices/virtual/net/enp2s0f0_2
lrwxrwxrwx 1 root root 0 Aug 17 11:11 enp2s0f0_3 -> ../../devices/virtual/net/enp2s0f0_3
...

New Structure:
The structure of VF's PCI Address/physfn/net contains the PF of that VF and the VF representors

$ ls /sys/bus/pci/devices/<vf-pci-addre>/physfn/net/
enp3s0f0 enp3s0f0_0 enp3s0f0_1 enp3s0f0_2 enp3s0f0_3

$ ls -l /sys/class/net
...
lrwxrwxrwx. 1 root root 0 Aug 17 08:43 enp3s0f0_0 -> ../../devices/pci0000:00/0000:00:02.0/0000:03:00.0/net/enp3s0f0_0
lrwxrwxrwx. 1 root root 0 Aug 17 08:43 enp3s0f0_1 -> ../../devices/pci0000:00/0000:00:02.0/0000:03:00.0/net/enp3s0f0_1
lrwxrwxrwx. 1 root root 0 Aug 17 08:43 enp3s0f0_2 -> ../../devices/pci0000:00/0000:00:02.0/0000:03:00.0/net/enp3s0f0_2
lrwxrwxrwx. 1 root root 0 Aug 17 08:43 enp3s0f0_3 -> ../../devices/pci0000:00/0000:00:02.0/0000:03:00.0/net/enp3s0f0_3
...

[1] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1
[2] https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/746836

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mamduh (<email address hidden>) on branch: master
Review: https://review.opendev.org/746836

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/746673
Committed: https://git.openstack.org/cgit/openstack/os-vif/commit/?id=76f7565b99e637d74878955a0033f35e9eb0e13f
Submitter: Zuul
Branch: master

commit 76f7565b99e637d74878955a0033f35e9eb0e13f
Author: Mamduh <email address hidden>
Date: Tue Aug 18 12:40:10 2020 +0300

    Refactor code of linux_net to more cleaner and increase performace

    The patch adds new functions '_get_phys_port_name' for reading physical
    port name of the SR-IOV port and '_get_phys_switch_id' for reading
    physical port switch ID of the SR-IOV port, in addition to refactoring
    'get_representor_port' to use the new functions and decrease calls for
    "_get_pf_func" and netdevs associated with the PF will now be processed
    in the loop, however it will not be matching 'phys_port_name' which
    ensures the correct behaviour.

    In addition to updating the unit test for linux_net and remove not
    needed mocks

    Related-Bug: #1892132
    Change-Id: I3fdbea4f48cb79ebfd03a4da21e2232ccafb7a76

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/756406

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/745895
Committed: https://git.openstack.org/cgit/openstack/os-vif/commit/?id=b37de19c58c877f5174d76d0a4ba5ab519f464e8
Submitter: Zuul
Branch: master

commit b37de19c58c877f5174d76d0a4ba5ab519f464e8
Author: Mamduh Alassi <email address hidden>
Date: Wed Aug 12 16:24:47 2020 +0300

    Fix - os-vif fails to get the correct UpLink Representor

    Till kernel 5.7 PF and VF representors are exposed as virtual device.
    They are not linked to its parent PCI device like how uplink
    representor is linked.

    Starting from kernel 5.8 due to new change [1] the PF and VF representors are
    linked to their parent PCI device, and so "get_ifname_by_pci_address" fails
    to get the correct UpLink Representor.

    This patch modifys the behviour of "get_ifname_by_pci_address" to
    check the physical port name of the netdev in
    vf_pci_addr_path/physfn/net to match the formart for the uplink "p\d+".

    [1] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1

    Closes-Bug: #1892132
    Change-Id: I49f6ae3f0e6bfbf555c8284bfd70371ce90da0c7

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/759255

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/756406
Committed: https://git.openstack.org/cgit/openstack/os-vif/commit/?id=167bb030f1143f37ce189673d831bd572f64d4ad
Submitter: Zuul
Branch: stable/victoria

commit 167bb030f1143f37ce189673d831bd572f64d4ad
Author: Mamduh <email address hidden>
Date: Tue Aug 18 12:40:10 2020 +0300

    Refactor code of linux_net to more cleaner and increase performace

    The patch adds new functions '_get_phys_port_name' for reading physical
    port name of the SR-IOV port and '_get_phys_switch_id' for reading
    physical port switch ID of the SR-IOV port, in addition to refactoring
    'get_representor_port' to use the new functions and decrease calls for
    "_get_pf_func" and netdevs associated with the PF will now be processed
    in the loop, however it will not be matching 'phys_port_name' which
    ensures the correct behaviour.

    In addition to updating the unit test for linux_net and remove not
    needed mocks

    Related-Bug: #1892132
    Change-Id: I3fdbea4f48cb79ebfd03a4da21e2232ccafb7a76
    (cherry picked from commit 76f7565b99e637d74878955a0033f35e9eb0e13f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759255
Committed: https://git.openstack.org/cgit/openstack/os-vif/commit/?id=a28aafa796378b8a36ccb51d2c0010a44f622d14
Submitter: Zuul
Branch: stable/victoria

commit a28aafa796378b8a36ccb51d2c0010a44f622d14
Author: Mamduh Alassi <email address hidden>
Date: Wed Aug 12 16:24:47 2020 +0300

    Fix - os-vif fails to get the correct UpLink Representor

    Till kernel 5.7 PF and VF representors are exposed as virtual device.
    They are not linked to its parent PCI device like how uplink
    representor is linked.

    Starting from kernel 5.8 due to new change [1] the PF and VF representors are
    linked to their parent PCI device, and so "get_ifname_by_pci_address" fails
    to get the correct UpLink Representor.

    This patch modifys the behviour of "get_ifname_by_pci_address" to
    check the physical port name of the netdev in
    vf_pci_addr_path/physfn/net to match the formart for the uplink "p\d+".

    [1] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1

    Closes-Bug: #1892132
    Change-Id: I49f6ae3f0e6bfbf555c8284bfd70371ce90da0c7
    (cherry picked from commit b37de19c58c877f5174d76d0a4ba5ab519f464e8)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-vif 2.3.0 release.

[Frode Nordahl (fnordahl)]

As noted in bug 1936341, with this os-vif change I still see the problem, and Nova is unable to plug representor ports. Does this fix depend on a new libvirt version with [0] or am I missing something else?

0: https://github.com/libvirt/libvirt/commit/5b1c525b1f3608156884aed0dc5e925306c1e260

[sean mooney (sean-k-mooney)]

the os-vif change simpley adds support for the new kernel layout after they broke the sys api by moving the devices.

it should have no depency on libvirt that i am aware of

[sean mooney (sean-k-mooney)]

i should point out that os-vif/nova does not use libvirt to find the parent device or plug the representor port. it looks like libvirt had a similar problem due to the kernel change but that is unrelated to this bug as far as i am aware.

[Frode Nordahl (fnordahl)]

It turns out I did not have the patch from https://review.opendev.org/c/openstack/os-vif/+/746673/, with that applied stuff comes farther but still does not work:

2021-07-15 14:09:44.821 1888819 ERROR nova.virt.libvirt.guest [req-5b690f1b-4a96-417b-952a-c10975cd4375 bedc4956b93f4d2da5637c739740bd4c 5ee76950b29a44b887faf10d6f084640 - a3d21d016da343458224473db4eeccc5 a3d21d016da343458224473db4eeccc5] Error launching a defined domain with XML: <domain type='kvm'>
  <name>instance-0000007f</name>
  <uuid>61442cbb-420b-4e00-9cf7-d332446c3997</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="21.2.0"/>
      <nova:name>node-laveran-64</nova:name>
      <nova:creationTime>2021-07-15 14:09:43</nova:creationTime>
      <nova:flavor name="m1.large">
        <nova:memory>512</nova:memory>
        <nova:disk>8</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>8</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="bedc4956b93f4d2da5637c739740bd4c">admin</nova:user>
        <nova:project uuid="5ee76950b29a44b887faf10d6f084640">admin</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="6979a978-6f28-4b4a-b291-b59ca7b904ad"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <shares>8192</shares>
  </cputune>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>OpenStack Foundation</entry>
      <entry name='product'>OpenStack Nova</entry>
      <entry name='version'>21.2.0</entry>
      <entry name='serial'>61442cbb-420b-4e00-9cf7-d332446c3997</entry>
      <entry name='uuid'>61442cbb-420b-4e00-9cf7-d332446c3997</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model' check='partial'>
    <topology sockets='8' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
      <source file='/var/lib/nova/instances/61442cbb-420b-4e00-9cf7-d332446c3997/disk'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='hostdev' managed='yes'>
      <mac address='fa:16:3e:57:89:96'/>
      <source>
        <address type='p...

[Frode Nordahl (fnordahl)]

Ok, closing the loop on this one, for the two patches to os-vif linked to this bug to work a libvirt with patches is required [0][1].

0: https://github.com/libvirt/libvirt/commit/5b1c525b1f3608156884aed0dc5e925306c1e260
1: https://github.com/libvirt/libvirt/commit/97ebb982453bc23759c5f180799d6f2207b81c80

[Christian Ehrhardt  (paelzer)]

Thanks for all the work Frode and all the involved openstack people.
Groovy is no more serviced via the SRU process, so I've disabled that. But I'll look at the libvirt branches for F/H now ...

[Brian Murray (brian-murray)]

Hello Mamduh, or anyone else affected,

Accepted libvirt into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/7.0.0-2ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Mamduh, or anyone else affected,

Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Frode Nordahl (fnordahl)]

Proposed libvirt package on Focal system with original unmodified kernel and driver:
$ uname -a
Linux node-laveran 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ cat /sys/class/net/enp129s0f0/device/driver/module/version
5.0-0

$ lspci -nnvv | grep Mellanox
03:00.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
        Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:0061]
03:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
        Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:0061]
03:00.2 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [15b3:1018]
        Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [15b3:0061]
...

# Note that in addition to libvirt from -proposed the system has a test
# package for the in-flight os-vif changes installed.
$ dpkg -l |grep libvirt
ii libvirt-clients 6.0.0-0ubuntu8.13 amd64 Programs for the libvirt library
ii libvirt-daemon 6.0.0-0ubuntu8.13 amd64 Virtualization daemon
ii libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.13 amd64 Virtualization daemon QEMU connection driver
ii libvirt-daemon-driver-storage-rbd 6.0.0-0ubuntu8.13 amd64 Virtualization daemon RBD storage driver
ii libvirt-daemon-system 6.0.0-0ubuntu8.13 amd64 Libvirt daemon configuration files
ii libvirt-daemon-system-systemd 6.0.0-0ubuntu8.13 amd64 Libvirt daemon configuration files (systemd)
ii libvirt0:amd64 6.0.0-0ubuntu8.13 amd64 library for interfacing with different virtualization systems

$ sudo grep -A6 hostdev /etc/libvirt/qemu/instance-00000001.xml
    <interface type='hostdev' managed='yes'>
      <mac address='fa:16:3e:1a:59:22'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x0b' function='0x4'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

$ openstack server list --long
...
| c575e200-74cf-45bd-801e-712d3405f460 | fnord-node-laveran-1 | ACTIVE | None | Running | network=10.42.2.217 | ubuntu | d9aa89af-9ad7-4770-83ba-194f03fec7dc | m1.large | 96afbb8a-697f-4de8-aa76-b8604bc01180 | nova | node-laveran.maas | |
...

$ ssh -i id_rsa fnord-node-laveran-1 lspci
...
00:03.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
...

<install OFED drivers, reboot and restart instances>

$ uname -a
Linux node-laveran 5.4.0-81-generic #91-Ubuntu SMP Thu Jul 15 19:09:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ cat /sys/class/net/enp129s0f0/device/driver/module/version
5.4-1.0.3

$ ssh -i id_rsa fnord-node-laveran-1 l...

[Łukasz Zemczak (sil2100)]

Could someone perform the same validation for hirsute?

[Frode Nordahl (fnordahl)]

Proposed libvirt package on Hirsute system with original unmodified kernel and d
river:
$ uname -a
Linux node-laveran 5.11.0-31-generic #33-Ubuntu SMP Wed Aug 11 13:19:04 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ sudo lshw|grep mlx5_core
                configuration: autonegotiation=on broadcast=yes driver=mlx5_core driverversion=5.11.0-31-generic firmware=16.31.1014 (MT_0000000183) latency=0 link=no multicast=yes
...

$ lspci -nnvv | grep Mellanox
03:00.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
        Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:0061]
03:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
        Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:0061]
03:00.2 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [15b3:1018]
        Subsystem: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] [15b3:0061]
...

$ dpkg -l | grep libvirt
ii libvirt-clients 7.0.0-2ubuntu2.1 amd64 Programs for the libvirt library
ii libvirt-daemon 7.0.0-2ubuntu2.1 amd64 Virtualization daemon
ii libvirt-daemon-config-network 7.0.0-2ubuntu2.1 all Libvirt daemon configuration files (default network)
ii libvirt-daemon-config-nwfilter 7.0.0-2ubuntu2.1 all Libvirt daemon configuration files (default network filters)
ii libvirt-daemon-driver-qemu 7.0.0-2ubuntu2.1 amd64 Virtualization daemon QEMU connection driver
ii libvirt-daemon-system 7.0.0-2ubuntu2.1 amd64 Libvirt daemon configuration files
ii libvirt-daemon-system-systemd 7.0.0-2ubuntu2.1 all Libvirt daemon configuration files (systemd)
ii libvirt0:amd64 7.0.0-2ubuntu2.1 amd64 library for interfacing with different virtualization systems

$ sudo grep -A6 hostdev /etc/libvirt/qemu/instance-00000002.xml
    <interface type='hostdev' managed='yes'>
      <mac address='fa:16:3e:3e:af:3d'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

$ openstack server list --long
| 9de43d8a-b63b-4747-9f6a-5ca20a501450 | fnord-node-laveran-1 | ACTIVE | None | Running | network=10.42.3.233 | auto-sync/ubuntu-focal-20.04-amd64-server-20210825-disk1.img | ee06a053-c350-474c-a03f-cf0afcb35591 | m1.large | 29873860-7b2e-49ad-a290-9a2d0600369d | nova | node-laveran.maas | |

$ ssh fnord-node-laveran-1 lspci
...
00:03.0 Etherne...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-vif/+/765419
Committed: https://opendev.org/openstack/os-vif/commit/3e1d898b752c327165aeca1fec0919d858a87ec4
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 3e1d898b752c327165aeca1fec0919d858a87ec4
Author: Mamduh <email address hidden>
Date: Tue Aug 18 12:40:10 2020 +0300

    Refactor code of linux_net to more cleaner and increase performace

    The patch adds new functions '_get_phys_port_name' for reading physical
    port name of the SR-IOV port and '_get_phys_switch_id' for reading
    physical port switch ID of the SR-IOV port, in addition to refactoring
    'get_representor_port' to use the new functions and decrease calls for
    "_get_pf_func" and netdevs associated with the PF will now be processed
    in the loop, however it will not be matching 'phys_port_name' which
    ensures the correct behaviour.

    In addition to updating the unit test for linux_net and remove not
    needed mocks

    Conflicts:
          vif_plug_ovs/linux_net.py
          vif_plug_ovs/tests/unit/test_linux_net.py

    Related-Bug: #1892132
    Change-Id: I3fdbea4f48cb79ebfd03a4da21e2232ccafb7a76
    (cherry picked from commit 167bb030f1143f37ce189673d831bd572f64d4ad)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-vif/+/765967
Committed: https://opendev.org/openstack/os-vif/commit/1226d46cc8bde9531821c68465a05292528142f0
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 1226d46cc8bde9531821c68465a05292528142f0
Author: Mamduh Alassi <email address hidden>
Date: Wed Aug 12 16:24:47 2020 +0300

    Fix - os-vif fails to get the correct UpLink Representor

    Till kernel 5.7 PF and VF representors are exposed as virtual device.
    They are not linked to its parent PCI device like how uplink
    representor is linked.

    Starting from kernel 5.8 due to new change [1] the PF and VF representors are
    linked to their parent PCI device, and so "get_ifname_by_pci_address" fails
    to get the correct UpLink Representor.

    This patch modifys the behviour of "get_ifname_by_pci_address" to
    check the physical port name of the netdev in
    vf_pci_addr_path/physfn/net to match the formart for the uplink "p\d+".

    [1] https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=123f0f53dd64b67e34142485fe866a8a581f12f1

    Closes-Bug: #1892132
    Change-Id: I49f6ae3f0e6bfbf555c8284bfd70371ce90da0c7
    (cherry picked from commit a28aafa796378b8a36ccb51d2c0010a44f622d14)

[Bartosz Bezak (bbezak)]

can we make a release os-vif for Victoria? 2.2.1? this bug is quite critical

[Corey Bryant (corey.bryant)]

A new version of python-os-vif has been uploaded to the focal unapproved queue with this fix:
https://launchpad.net/ubuntu/focal/+queue?queue_state=1&queue_text=python-os-vif

[Corey Bryant (corey.bryant)]

Also uploaded a new version with this fix to victoria-staging: https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/victoria-staging/+packages?field.name_filter=python-os-vif&field.status_filter=published&field.series_filter=

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 6.0.0-0ubuntu8.13

---------------
libvirt (6.0.0-0ubuntu8.13) focal; urgency=medium

  * Add support for switchdev NICs that link representor ports to parent PCI
    device. (LP: #1892132)
    - d/p/u/lp-1892132-Add-phys_port_name-support-on-virPCIGetNetName.patch
    - d/p/u/lp-1892132-add-virNetDevGetPhysPortName.patch

 -- Frode Nordahl <email address hidden> Fri, 16 Jul 2021 05:16:36 +0000

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for libvirt has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 7.0.0-2ubuntu2.1

---------------
libvirt (7.0.0-2ubuntu2.1) hirsute; urgency=medium

  * Add support for switchdev NICs that link representor ports to parent PCI
    device. (LP: #1892132)
    - d/p/u/lp-1892132-Add-phys_port_name-support-on-virPCIGetNetName.patch
    - d/p/u/lp-1892132-add-virNetDevGetPhysPortName.patch

 -- Frode Nordahl <email address hidden> Fri, 16 Jul 2021 05:16:36 +0000

[Brian Murray (brian-murray)]

Hello Mamduh, or anyone else affected,

Accepted python-os-vif into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-os-vif/2.0.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-vif 2.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-vif 2.2.1 release.

[Corey Bryant (corey.bryant)]

Hello Mamduh, or anyone else affected,

Accepted python-os-vif into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Mamduh (mmduh)]

Hello Corey,

Sorry for the late update, I was moved to another team several months ago, and I couldn't track the issue. Moshe Levi should be the contact point for this issue, I will update him internally on the topic

[Frode Nordahl (fnordahl)]

Mamduh,

Thank you for the original bits and for responding on the bug. The sudden flurry of activity here is a result of software distributions picking up the bits and providing them to the masses through Stable Release Updates.

[Frode Nordahl (fnordahl)]

os-vif focal/ussuri:

$ uname -a
Linux node-laveran 5.4.0-84-generic #94-Ubuntu SMP Thu Aug 26 20:27:37 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ dpkg -l | grep os-vif
ii python3-os-vif 2.0.0-0ubuntu2 all Integration library between network and compute - Python 3.x

$ sudo grep -A6 hostdev /etc/libvirt/qemu/instance-00000002.xml
    <interface type='hostdev' managed='yes'>
      <mac address='fa:16:3e:66:37:f9'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x03' function='0x7'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

$ sudo ovs-vsctl find interface name=eth29
_uuid : cce9cf12-28e2-4c4c-be49-6231eaac991c
admin_state : up
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : []
error : []
external_ids : {attached-mac="fa:16:3e:66:37:f9", iface-id="88dc4421-c068-4aea-995e-cf181966f6c8", iface-status=active, vm-uuid="6c0ea8c0-a7aa-4cf4-b879-af2523ea5ee1"}
ifindex : 104
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 0
link_speed : []
link_state : up
lldp : {}
mac : []
mac_in_use : "5e:03:fa:49:dc:37"
mtu : 8942
mtu_request : []
name : eth29
ofport : 3
ofport_request : []
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=65184, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_missed_errors=0, rx_over_err=0, rx_packets=446, tx_bytes=93555, tx_dropped=0, tx_errors=0, tx_packets=398}
status : {driver_name=mlx5e_rep, driver_version="5.4.0-84-generic", firmware_version="16.31.1014 (MT_0000000183)"}
type : ""

$ sudo ls -l /sys/class/net/eth29/device
ls: cannot access '/sys/class/net/eth29/device': No such file or directory

$ sudo ls -l /sys/class/net/enp129s0f0/device/net/
total 0
drwxr-xr-x 6 root root 0 Sep 9 13:16 enp129s0f0

$ ssh fnord-node-laveran-1 lspci
...
00:03.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]

$ sudo apt install --install-recommends linux-generic-hwe-20.04-edge-wip
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  linux-headers-5.13.0-14-generic linux-headers-generic-hwe-20.04-edge-wip
  linux-hwe-5.13-headers-5.13.0-14 linux-image-5.13.0-14-generic
  linux-image-generic-hwe-20.04-edge-wip linux-modules-5.13.0-14-generic
  linux-modules-extra-5.13.0-14-generic
Suggested packages:
  fdutils linux-doc | linux-hwe-5.13-source-5.13.0 linux-hwe-5.13-tools
The following NEW packages will be installed:
  linux-generic-hwe-20.04-edge-wip linux-headers-5.13.0-14-generic
  linux-headers-generic-hwe-20.04-edge-wip linux-hwe-5.13-h...

[Corey Bryant (corey.bryant)]

Hello Mamduh, or anyone else affected,

Accepted libvirt into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Mamduh, or anyone else affected,

Accepted python-os-vif into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Frode Nordahl (fnordahl)]

os-vif focal/victoria

$ uname -a
Linux node-laveran 5.4.0-84-generic #94-Ubuntu SMP Thu Aug 26 20:27:37 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ dpkg -l | grep os-vif
ii python3-os-vif 2.2.0-0ubuntu1~cloud1 all Integration library between network and compute - Python 3.x

$ sudo grep -A6 hostdev /etc/libvirt/qemu/instance-00000004.xml
    <interface type='hostdev' managed='yes'>
      <mac address='fa:16:3e:a2:74:a8'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x0b' function='0x5'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

$ sudo ovs-vsctl find interface name=eth59
_uuid : 27596589-93b4-419c-ac0d-77b848116334
admin_state : up
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : []
error : []
external_ids : {attached-mac="fa:16:3e:a2:74:a8", iface-id="bb8c0af4-6b0a-4ec0-87ee-2cb30eb76445", iface-status=active, vm-uuid="27787de4-e1f7-45b0-b32f-d49a81a0675f"}
ifindex : 135
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 0
link_speed : []
link_state : up
lldp : {}
mac : []
mac_in_use : "0e:e7:e4:a9:5c:d5"
mtu : 8942
mtu_request : []
name : eth59
ofport : 3
ofport_request : []
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=36121, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_missed_errors=0, rx_over_err=0, rx_packets=270, tx_bytes=49194, tx_dropped=0, tx_errors=0, tx_packets=175}
status : {driver_name=mlx5e_rep, driver_version="5.4.0-84-generic", firmware_version="16.31.1014 (MT_0000000183)"}
type : ""

$ sudo ls -l /sys/class/net/eth59/device
ls: cannot access '/sys/class/net/eth59/device': No such file or directory

$ sudo ls -l /sys/class/net/enp129s0f0/device/net/
total 0
drwxr-xr-x 6 root root 0 Sep 10 09:37 enp129s0f0

$ ssh fnord-node-laveran-1 lspci
00:03.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]

$ sudo apt install --install-recommends linux-generic-hwe-20.04-edge-wip
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  linux-headers-5.13.0-14-generic linux-headers-generic-hwe-20.04-edge-wip
  linux-hwe-5.13-headers-5.13.0-14 linux-image-5.13.0-14-generic
  linux-image-generic-hwe-20.04-edge-wip linux-modules-5.13.0-14-generic
  linux-modules-extra-5.13.0-14-generic
Suggested packages:
  fdutils linux-doc | linux-hwe-5.13-source-5.13.0 linux-hwe-5.13-tools
The following NEW packages will be installed:
  linux-generic-hwe-20.04-edge-wip linux-headers-5.13.0-14-generic
  linux-headers-generic-hwe-20.04-edge-wip linux-hwe-5.13-hea...

[Frode Nordahl (fnordahl)]

os-vif bionic/ussuri:

$ uname -a
Linux node-laveran 5.4.0-84-generic #94~18.04.1-Ubuntu SMP Thu Aug 26 23:17:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ dpkg -l | grep os-vif
ii python3-os-vif 2.0.0-0ubuntu2~cloud0 all Integration library between network and compute - Python 3.x

$ sudo grep -A6 hostdev /etc/libvirt/qemu/instance-00000019.xml
    <interface type='hostdev' managed='yes'>
      <mac address='fa:16:3e:a8:b0:d7'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x4'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

$ sudo ovs-vsctl find interface name=eth2
_uuid : c1b7651a-ca84-481a-bbb3-8531abddd7d4
admin_state : up
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : []
error : []
external_ids : {attached-mac="fa:16:3e:a8:b0:d7", iface-id="f58fbe72-698c-4aa7-b38a-427c68ea660d", iface-status=active, vm-uuid="850b1c28-1fd4-4c83-b3b4-cf1727253bb7"}
ifindex : 77
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 0
link_speed : []
link_state : up
lldp : {}
mac : []
mac_in_use : "0e:93:79:d4:cb:21"
mtu : 8942
mtu_request : []
name : eth2
ofport : 6
ofport_request : []
options : {}
other_config : {}
statistics : {collisions=0, rx_bytes=26846, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_missed_errors=0, rx_over_err=0, rx_packets=273, tx_bytes=15409, tx_dropped=0, tx_errors=0, tx_packets=101}
status : {driver_name=mlx5e_rep, driver_version="5.4.0-84-generic", firmware_version="16.31.1014 (MT_0000000183)"}
type : ""

$ sudo ls -l /sys/class/net/eth2/device
ls: cannot access '/sys/class/net/eth2/device': No such file or directory

$ sudo ls -l /sys/class/net/enp129s0f0/device/net/
total 0
drwxr-xr-x 6 root root 0 Sep 10 15:26 enp129s0f0

<INSTALL OFED DRIVERS AND REBOOT>

$ sudo ovs-vsctl find interface name=enp3s0f0_2
_uuid : 05cf3156-83fa-4c08-af91-45e510d32367
admin_state : up
bfd : {}
bfd_status : {}
cfm_fault : []
cfm_fault_status : []
cfm_flap_count : []
cfm_health : []
cfm_mpid : []
cfm_remote_mpids : []
cfm_remote_opstate : []
duplex : full
error : []
external_ids : {attached-mac="fa:16:3e:a8:b0:d7", iface-id="f58fbe72-698c-4aa7-b38a-427c68ea660d", iface-status=active, vm-uuid="850b1c28-1fd4-4c83-b3b4-cf1727253bb7"}
ifindex : 76
ingress_policing_burst: 0
ingress_policing_rate: 0
lacp_current : []
link_resets : 1
link_speed : 10000000000
link_state : up
lldp : {}
mac : []
mac_in...

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-os-vif has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-os-vif - 2.2.0-0ubuntu1~cloud1
---------------

 python-os-vif (2.2.0-0ubuntu1~cloud1) focal-victoria; urgency=medium
 .
   [ Frode Nordahl ]
   * Add support for switchdev NICs that link representor ports to parent PCI
     device. (LP: #1892132)
     - d/p/Refactor-code-of-linux_net-to-more-cleaner-and-increase-performace.patch
     - d/p/Fix-os-vif-fails-to-get-the-correct-UpLink-Representor.patch
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/victoria branch.
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-os-vif - 2.0.0-0ubuntu2

---------------
python-os-vif (2.0.0-0ubuntu2) focal; urgency=medium

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/ussuri branch.

  [ Chris MacNaughton ]
  * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.

  [ Frode Nordahl ]
  * Add support for switchdev NICs that link representor ports to parent PCI
    device. (LP: #1892132)
    - d/p/Refactor-code-of-linux_net-to-more-cleaner-and-increase-performace.patch
    - d/p/Fix-os-vif-fails-to-get-the-correct-UpLink-Representor.patch

 -- Frode Nordahl <email address hidden> Fri, 16 Jul 2021 09:43:06 +0000

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for python-os-vif has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package python-os-vif - 2.0.0-0ubuntu2~cloud0
---------------

 python-os-vif (2.0.0-0ubuntu2~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 python-os-vif (2.0.0-0ubuntu2) focal; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/ussuri branch.
 .
   [ Chris MacNaughton ]
   * d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
 .
   [ Frode Nordahl ]
   * Add support for switchdev NICs that link representor ports to parent PCI
     device. (LP: #1892132)
     - d/p/Refactor-code-of-linux_net-to-more-cleaner-and-increase-performace.patch
     - d/p/Fix-os-vif-fails-to-get-the-correct-UpLink-Representor.patch

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/os-vif/+/765983
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/queens
Review: https://review.opendev.org/c/openstack/os-vif/+/765941
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/os-vif/+/765977
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/rocky
Review: https://review.opendev.org/c/openstack/os-vif/+/765962
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/os-vif/+/765914
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/stein
Review: https://review.opendev.org/c/openstack/os-vif/+/765974
Reason: This branch transitioned to End of Life for this project, open patches needs to be closed to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/os-vif/+/765912
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/os-vif/+/765970
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.