Comment 14 for bug 1999181

Revision history for this message
Enoch Leung (leun0036) wrote :

Seems like X540 is fine, can create VFs in DOWN state, not like Broadcom; just that X540 is missing quite a bit of interesting features like VEPA and VEB that Broadcom offers...

dmesg | grep ixgbe
-----------------------
[ 3.673668] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver
[ 3.673671] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[ 3.681029] ixgbe 0000:01:00.0: enabling device (0000 -> 0002)
[ 4.026720] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 4.148607] ixgbe 0000:01:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:1c.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 4.183467] ixgbe 0000:01:00.0: MAC: 3, PHY: 0, PBA No: 000000-000
[ 4.183471] ixgbe 0000:01:00.0: **:**:**:**:**:**
[ 4.389907] ixgbe 0000:01:00.0: Intel(R) 10 Gigabit Network Connection
[ 4.392301] ixgbe 0000:01:00.1: enabling device (0000 -> 0002)
[ 4.685988] ixgbe 0000:01:00.1: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 4.702556] ixgbe 0000:01:00.0 eth2: SR-IOV enabled with 4 VFs
[ 4.774971] ixgbe 0000:01:00.1: 16.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x4 link at 0000:00:1c.0 (capable of 32.000 Gb/s with 5.0 GT/s PCIe x8 link)
[ 4.798867] ixgbe 0000:01:00.1: MAC: 3, PHY: 0, PBA No: 000000-000
[ 4.798871] ixgbe 0000:01:00.1: **:**:**:**:**:**
[ 5.022018] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 5.041904] ixgbe 0000:01:00.1: Intel(R) 10 Gigabit Network Connection
[ 5.068702] ixgbe 0000:01:00.1 x540_p1: renamed from eth1
[ 5.190445] ixgbe 0000:01:00.0 x540_p0: renamed from eth2
[ 5.196997] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver
[ 5.196999] ixgbevf: Copyright (c) 2009 - 2018 Intel Corporation.
[ 5.220879] ixgbevf 0000:01:10.0: enabling device (0000 -> 0002)
[ 5.232365] ixgbevf 0000:01:10.0: PF still in reset state. Is the PF interface up?
[ 5.232369] ixgbevf 0000:01:10.0: Assigning random MAC address
[ 5.235195] ixgbevf 0000:01:10.0: **:**:**:**:**:**
[ 5.235199] ixgbevf 0000:01:10.0: MAC: 2
[ 5.235200] ixgbevf 0000:01:10.0: Intel(R) X540 Virtual Function

put PF to DOWN state
============================
ip link set dev x540_p0 down
5: x540_p0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 6c:92:bf:5d:de:6c brd ff:ff:ff:ff:ff:ff

can create VF in DOWN state
============================
cat /sys/class/net/x540_p0/device/sriov_numvfs = 0
cat /sys/class/net/x540_p0/device/sriov_totalvfs = 63
echo 4 > /sys/class/net/x540_p0/device/sriov_numvfs
5: x540_p0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether **:**:**:**:**:** brd ff:ff:ff:ff:ff:ff
    vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off
    vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off, query_rss off

well, even with X540 I still have issue with libvirt not starting a SR-IOV pool as expected, which is why I am filing this bug in the first place, sigh; so I filed a bug with libvirt (somehow I see no one in the notification?) = https://bugs.launchpad.net/libvirt/+bug/2031078

I'm not sure if it is related to netplan or not, but even with X540, the same message "PF still in reset state. Is the PF interface up?" appears; would this an indicator of failing to autostart a SR-IOV network? I also tried blacklisted ixgbevf from loading, so this "PF still in reset state. Is the PF interface up?" message will not show, but it will be the same that VF can be created after reboot, but SR-IOV network will not autostart. A dependency on PF readiness to create SR-IOV network? but that could be libvirt's issue I guess.