kunpeng920

[pcie-82599]82599 clear vf got error msg "failed to kill vid 0081/0 for device enp130s16"

Series upstream-kernel
Bug #1847460

Bug #1847460 reported by Andrew Cloke on 2019-10-09

This bug affects 1 person

	Status	Importance	Assigned to
kunpeng920	Invalid	Undecided	Unassigned
Ubuntu-18.04	Invalid	Undecided	Unassigned
Ubuntu-18.04-hwe	Invalid	Undecided	Unassigned
Ubuntu-19.04	Won't Fix	Undecided	Unassigned
Ubuntu-19.10	Won't Fix	Undecided	Unassigned
Upstream-kernel	Invalid	Undecided	Unassigned

Bug Description

[Bug Description]

82599 clear vf got error msg "failed to kill vid 0081/0 for device enp130s16"

[Steps to Reproduce]
1 $ echo 1 > sriov_numvfs
2 $ ifconfig eth10 192.168.1.79 #eth10 is vf(make sure vf is link up)
3 $ echo 0 > sriov_numvfs

[Actual Results]
full reproduce log:
[root@localhost ~]# cd /sys/bus/pci/devices/0000:82:00.0
[root@localhost 0000:82:00.0]# echo 1 > sriov_numvfs
[ 209.863226] ixgbe 0000:82:00.0 enp130s0f0: SR-IOV enabled with 1 VFs
[ 209.869778] ixgbe 0000:82:00.0: removed PHC on enp130s0f0
[ 209.964028] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Down
[ 210.010467] ixgbe 0000:82:00.0: registered PHC device on enp130s0f0

[ 210.252951] pci 0000:82:10.0: can't set Max Payload Size to 512; if necessary, use "pci=pcie_bus_safe" and report a bug
[root@localhost 0000:82:00.0]#
[ 210.271246] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[ 210.273958] ixgbe 0000:82:00.0 enp130s0f0: detected SFP+: 5
[ 210.286066] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[ 210.292327] iommu: Adding device 0000:82:10.0 to group 48, default domain type 3
[root@localhost [ 210.300307] ixgbevf 0000:82:10.0: enabling device (0000 -> 0002)
0000:82:00.0]# [ 210.307499] ixgbe 0000:82:00.0 enp130s0f0: VF Reset msg received from vf 0
[ 210.315830] ixgbe 0000:82:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[ 210.335565] ixgbevf 0000:82:10.0: MAC address not assigned by administrator.
[ 210.342587] ixgbevf 0000:82:10.0: Assigning random MAC address
[ 210.349277] ixgbevf 0000:82:10.0: ba:92:c8:d9:4d:e2
[ 210.354140] ixgbevf 0000:82:10.0: MAC: 1
[ 210.358045] ixgbevf 0000:82:10.0: Intel(R) 82599 Virtual Function
[ 210.369572] ixgbevf 0000:82:10.0 enp130s16: renamed from eth0
[ 210.426072] IPv6: ADDRCONF(NETDEV_UP): enp130s16: link is not ready
[ 210.448470] IPv6: ADDRCONF(NETDEV_UP): enp130s16: link is not ready
[ 210.454722] 8021q: adding VLAN 0 to HW filter on device enp130s16
[ 210.773132] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 210.781278] ixgbe 0000:82:00.0 enp130s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 210.789771] ixgbevf 0000:82:10.0: NIC Link is Up 10 Gbps
[ 210.795089] IPv6: ADDRCONF(NETDEV_CHANGE): enp130s16: link becomes ready

[root@localhost 0000:82:00.0]#
[root@localhost 0000:82:00.0]#
[root@localhost 0000:82:00.0]#
[root@localhost 0000:82:00.0]#
[root@localhost 0000:82:00.0]# ethtool -i enp130s16
driver: ixgbevf
version: 4.1.0-k
firmware-version:
expansion-rom-version:
bus-info: 0000:82:10.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
[root@localhost 0000:82:00.0]#
[root@localhost 0000:82:00.0]# ip addr add 192.168.2.116/24 dev enp130s16
[root@localhost 0000:82:00.0]#
[root@localhost 0000:82:00.0]#
[root@localhost 0000:82:00.0]#
[root@localhost 0000:82:00.0]# ethtool enp130s16
Settings for enp130s16:
     Supported ports: [ ]
     Supported link modes: 10000baseT/Full
     Supported pause frame use: No
     Supports auto-negotiation: No
     Supported FEC modes: Not reported
     Advertised link modes: Not reported
     Advertised pause frame use: No
     Advertised auto-negotiation: No
     Advertised FEC modes: Not reported
     Speed: 10000Mb/s
     Duplex: Full
     Port: Other
     PHYAD: 0
     Transceiver: internal
     Auto-negotiation: off
     Current message level: 0x00000007 (7)
                    drv probe link
     Link detected: yes
[root@localhost 0000:82:00.0]#
[root@localhost 0000:82:00.0]# dmesg
[ 209.863226] ixgbe 0000:82:00.0 enp130s0f0: SR-IOV enabled with 1 VFs
[ 209.869778] ixgbe 0000:82:00.0: removed PHC on enp130s0f0
[ 209.964028] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Down
[ 209.978348] ixgbe 0000:82:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 210.010467] ixgbe 0000:82:00.0: registered PHC device on enp130s0f0
[ 210.252892] pci 0000:82:10.0: [8086:10ed] type 00 class 0x020000
[ 210.252951] pci 0000:82:10.0: can't set Max Payload Size to 512; if necessary, use "pci=pcie_bus_safe" and report a bug
[ 210.271246] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[ 210.273958] ixgbe 0000:82:00.0 enp130s0f0: detected SFP+: 5
[ 210.286066] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[ 210.292327] iommu: Adding device 0000:82:10.0 to group 48, default domain type 3
[ 210.300307] ixgbevf 0000:82:10.0: enabling device (0000 -> 0002)
[ 210.307499] ixgbe 0000:82:00.0 enp130s0f0: VF Reset msg received from vf 0
[ 210.315830] ixgbe 0000:82:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[ 210.335565] ixgbevf 0000:82:10.0: MAC address not assigned by administrator.
[ 210.342587] ixgbevf 0000:82:10.0: Assigning random MAC address
[ 210.349277] ixgbevf 0000:82:10.0: ba:92:c8:d9:4d:e2
[ 210.354140] ixgbevf 0000:82:10.0: MAC: 1
[ 210.358045] ixgbevf 0000:82:10.0: Intel(R) 82599 Virtual Function
[ 210.369572] ixgbevf 0000:82:10.0 enp130s16: renamed from eth0
[ 210.426072] IPv6: ADDRCONF(NETDEV_UP): enp130s16: link is not ready
[ 210.448470] IPv6: ADDRCONF(NETDEV_UP): enp130s16: link is not ready
[ 210.454722] 8021q: adding VLAN 0 to HW filter on device enp130s16
[ 210.773132] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 210.781278] ixgbe 0000:82:00.0 enp130s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 210.789771] ixgbevf 0000:82:10.0: NIC Link is Up 10 Gbps
[ 210.795089] IPv6: ADDRCONF(NETDEV_CHANGE): enp130s16: link becomes ready
[root@localhost 0000:82:00.0]#
[root@localhost 0000:82:00.0]#
[root@localhost 0000:82:00.0]# echo 0 > sriov_numvfs
[ 262.666102] failed to kill vid 0081/0 for device enp130s16
[ 262.783042] iommu: Removing device 0000:82:10.0 from group 48
[ 263.932947] ixgbe 0000:82:00.0: removed PHC on enp130s0f0
[ 264.023999] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Down
[ 264.026256] ixgbe 0000:82:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[ 264.287280] ixgbe 0000:82:00.0: registered PHC device on enp130s0f0

[Expected Results]
no print
[ 262.666102] failed to kill vid 0081/0 for device enp130s16

[Reproducibility]
100%

[Additional information]
(Firmware version, kernel version, affected hardware, etc. if required):

[Resolution]

Tags:

Revision history for this message

Andrew Cloke (andrew-cloke) wrote on 2019-10-09:

Also reported at https://bugs.centos.org/view.php?id=16342

Revision history for this message

dann frazier (dannf) wrote on 2019-10-11:

I can't reproduce w/ eoan (kernel 5.3.0-13.14):
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 1 | sudo tee sriov_numvfs
[339836.821452] vfio-pci 0000:81:00.0: Driver doesn't support SRIOV configuration via sysfs

Revision history for this message

dann frazier (dannf) wrote on 2019-10-11:

Ignore comment #2 - I see that the message was from vfio-pci, not from ixgbe.

Changed in kunpeng920:
status:	New → Incomplete

Revision history for this message

dann frazier (dannf) wrote on 2019-10-11:

I can not reproduce with 5.3.0-17.18 (see below). What kernel version did you reproduce on?

ubuntu@segers:~$ cat /proc/version
Linux version 5.3.0-17-generic (buildd@bos02-arm64-059) (gcc version 9.2.1 20190909 (Ubuntu 9.2.1-8ubuntu1)) #18-Ubuntu SMP Fri Oct 4 03:05:05 UTC 2019
ubuntu@segers:~$ cd /sys/bus/pci/devices/0000\:81\:00.0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ dmesg -w &
[1] 2985
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 1 | sudo tee sriov_numvfs
1
[ 469.842996] ixgbe 0000:81:00.0 enp129s0f0: SR-IOV enabled with 1 VFs
[ 469.843085] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[ 469.958325] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 469.982192] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
[ 470.152024] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[ 470.194507] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[ 470.194801] ixgbevf 0000:81:10.0: Adding to iommu group 34
[ 470.195291] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[ 470.195388] ixgbe 0000:81:00.0 enp129s0f0: VF Reset msg received from vf 0
[ 470.206381] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[ 470.206383] ixgbevf 0000:81:10.0: Assigning random MAC address
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [ 470.207399] ixgbevf 0000:81:10.0: b6:db:3d:65:78:c5
[ 470.207401] ixgbevf 0000:81:10.0: MAC: 1
[ 470.207402] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[ 470.209377] ixgbevf 0000:81:10.0 enp129s0f0v0: renamed from eth0
[ 470.298533] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ sudo ip addr add 192.168.2.116/24 dev enp129s0f0v0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 0 | sudo tee sriov_numvfs
0
[ 482.195737] pci 0000:81:10.0: Removing from iommu group 34
[ 483.315757] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[ 483.385025] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[ 483.677052] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [ 483.855744] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[ 483.995818] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$

I can not reproduce with 5.3.0-17.18 (see below). What kernel version did you reproduce on?

ubuntu@segers:~$ cat /proc/version
Linux version 5.3.0-17-generic (buildd@bos02-arm64-059) (gcc version 9.2.1 20190909 (Ubuntu 9.2.1-8ubuntu1)) #18-Ubuntu SMP Fri Oct 4 03:05:05 UTC 2019
ubuntu@segers:~$ cd /sys/bus/pci/devices/0000\:81\:00.0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ dmesg -w &
[1] 2985
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 1 | sudo tee sriov_numvfs 
1
[  469.842996] ixgbe 0000:81:00.0 enp129s0f0: SR-IOV enabled with 1 VFs
[  469.843085] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[  469.958325] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[  469.982192] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
[  470.152024] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[  470.194507] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  470.194801] ixgbevf 0000:81:10.0: Adding to iommu group 34
[  470.195291] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  470.195388] ixgbe 0000:81:00.0 enp129s0f0: VF Reset msg received from vf 0
[  470.206381] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  470.206383] ixgbevf 0000:81:10.0: Assigning random MAC address
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [  470.207399] ixgbevf 0000:81:10.0: b6:db:3d:65:78:c5
[  470.207401] ixgbevf 0000:81:10.0: MAC: 1
[  470.207402] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  470.209377] ixgbevf 0000:81:10.0 enp129s0f0v0: renamed from eth0
[  470.298533] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ sudo ip addr add 192.168.2.116/24 dev enp129s0f0v0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 0 | sudo tee sriov_numvfs 
0
[  482.195737] pci 0000:81:10.0: Removing from iommu group 34
[  483.315757] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[  483.385025] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[  483.677052] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [  483.855744] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[  483.995818] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$

Revision history for this message

dann frazier (dannf) wrote on 2019-10-13:

Also not reproducible w/ the current 5.0 kernel:

ubuntu@segers:~$ uname -a
Linux segers 5.0.0-31-generic #33~18.04.1-Ubuntu SMP Tue Oct 1 10:19:33 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@segers:~$ cat /proc/version
Linux version 5.0.0-31-generic (buildd@bos02-arm64-011) (gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1)) #33~18.04.1-Ubuntu SMP Tue Oct 1 10:19:33 UTC 2019
ubuntu@segers:~$ sudo dmesg -c > /dev/null
ubuntu@segers:~$ dmesg -w &
[1] 3035
ubuntu@segers:~$ cd /sys/bus/pci/devices/0000\:81\:00.0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 1 | sudo tee sriov_numvfs
1
[ 5606.483299] ixgbe 0000:81:00.0 enp129s0f0: SR-IOV enabled with 1 VFs
[ 5606.483387] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[ 5606.598362] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 5606.624755] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
[ 5606.796683] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [ 5606.838703] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[ 5606.845813] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[ 5606.845815] ixgbevf: Copyright (c) 2009 - 2018 Intel Corporation.
[ 5606.845960] iommu: Adding device 0000:81:10.0 to group 34
[ 5606.846483] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[ 5606.846649] ixgbe 0000:81:00.0 enp129s0f0: VF Reset msg received from vf 0
[ 5606.857641] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[ 5606.857642] ixgbevf 0000:81:10.0: Assigning random MAC address
[ 5606.858768] ixgbevf 0000:81:10.0: ca:29:68:dc:ae:86
[ 5606.858770] ixgbevf 0000:81:10.0: MAC: 1
[ 5606.858771] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[ 5606.861196] ixgbevf 0000:81:10.0 enp129s16: renamed from eth0
[ 5606.942707] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ sudo ip addr add 192.168.2.116/24 dev enp129s16
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 0 | sudo tee sriov_numvfs
0
[ 5660.414513] iommu: Removing device 0000:81:10.0 from group 34
[ 5661.542398] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[ 5661.611872] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[ 5661.954012] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [ 5662.130337] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[ 5662.374400] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[64860.763923] hns3 0000:7d:00.0 enp125s0f0: link up
[64860.763943] IPv6: ADDRCONF(NETDEV_CHANGE): enp125s0f0: link becomes ready
[64908.891639] hns3 0000:7d:00.0 enp125s0f0: link down

Also not reproducible w/ the current 5.0 kernel:

ubuntu@segers:~$ uname -a
Linux segers 5.0.0-31-generic #33~18.04.1-Ubuntu SMP Tue Oct 1 10:19:33 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@segers:~$ cat /proc/version
Linux version 5.0.0-31-generic (buildd@bos02-arm64-011) (gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1)) #33~18.04.1-Ubuntu SMP Tue Oct 1 10:19:33 UTC 2019
ubuntu@segers:~$ sudo dmesg -c > /dev/null
ubuntu@segers:~$ dmesg -w &
[1] 3035
ubuntu@segers:~$ cd /sys/bus/pci/devices/0000\:81\:00.0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 1 | sudo tee sriov_numvfs 
1
[ 5606.483299] ixgbe 0000:81:00.0 enp129s0f0: SR-IOV enabled with 1 VFs
[ 5606.483387] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[ 5606.598362] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 5606.624755] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
[ 5606.796683] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [ 5606.838703] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[ 5606.845813] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[ 5606.845815] ixgbevf: Copyright (c) 2009 - 2018 Intel Corporation.
[ 5606.845960] iommu: Adding device 0000:81:10.0 to group 34
[ 5606.846483] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[ 5606.846649] ixgbe 0000:81:00.0 enp129s0f0: VF Reset msg received from vf 0
[ 5606.857641] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[ 5606.857642] ixgbevf 0000:81:10.0: Assigning random MAC address
[ 5606.858768] ixgbevf 0000:81:10.0: ca:29:68:dc:ae:86
[ 5606.858770] ixgbevf 0000:81:10.0: MAC: 1
[ 5606.858771] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[ 5606.861196] ixgbevf 0000:81:10.0 enp129s16: renamed from eth0
[ 5606.942707] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ sudo ip addr add 192.168.2.116/24 dev enp129s16
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 0 | sudo tee sriov_numvfs 
0
[ 5660.414513] iommu: Removing device 0000:81:10.0 from group 34
[ 5661.542398] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[ 5661.611872] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[ 5661.954012] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [ 5662.130337] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[ 5662.374400] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[64860.763923] hns3 0000:7d:00.0 enp125s0f0: link up
[64860.763943] IPv6: ADDRCONF(NETDEV_CHANGE): enp125s0f0: link becomes ready
[64908.891639] hns3 0000:7d:00.0 enp125s0f0: link down

Revision history for this message

dann frazier (dannf) wrote on 2019-10-13:

Download full text (5.0 KiB)

Also cannot reproduce w/ latest 4.15.0:

ubuntu@segers:~$ sudo dmesg -c > /dev/null
ubuntu@segers:~$ cat /proc/version
Linux version 4.15.0-65-generic (buildd@bos02-arm64-005) (gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1)) #74-Ubuntu SMP Tue Sep 17 17:06:57 UTC 2019
ubuntu@segers:~$ dmesg -w &
[1] 2589
ubuntu@segers:~$ cd /sys/bus/pci/devices/0000\:81\:00.0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 1 | sudo tee sriov_numvfs
1
[ 187.831396] ixgbe 0000:81:00.0 enp129s0f0: SR-IOV enabled with 1 VFs
[ 187.831499] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[ 187.945690] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 187.967264] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
[ 188.138013] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [ 188.180642] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[ 188.186470] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[ 188.186472] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[ 188.186625] iommu: Adding device 0000:81:10.0 to group 34
[ 188.187137] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[ 188.187860] ixgbe 0000:81:00.0 enp129s0f0: VF Reset msg received from vf 0
[ 188.188291] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[ 188.198852] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[ 188.198854] ixgbevf 0000:81:10.0: Assigning random MAC address
[ 188.199817] ixgbevf 0000:81:10.0: 42:0a:4d:f5:23:a7
[ 188.199819] ixgbevf 0000:81:10.0: MAC: 1
[ 188.199820] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[ 188.202735] ixgbevf 0000:81:10.0 enp129s16: renamed from eth0
[ 188.284641] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ sudo ip addr add 192.168.2.116/24 dev enp129s16
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 0 | sudo tee sriov_numvfs
0
[ 211.716800] iommu: Removing device 0000:81:10.0 from group 34
[ 212.844697] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[ 212.929165] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[ 213.202899] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [ 213.373900] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[ 213.520693] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

Also cannot reproduce w/ latest 4.15.0:

ubuntu@segers:~$ sudo dmesg -c > /dev/null
ubuntu@segers:~$ cat /proc/version
Linux version 4.15.0-65-generic (buildd@bos02-arm64-005) (gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1)) #74-Ubuntu SMP Tue Sep 17 17:06:57 UTC 2019
ubuntu@segers:~$ dmesg -w &
[1] 2589
ubuntu@segers:~$ cd /sys/bus/pci/devices/0000\:81\:00.0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 1 | sudo tee sriov_numvfs 
1
[  187.831396] ixgbe 0000:81:00.0 enp129s0f0: SR-IOV enabled with 1 VFs
[  187.831499] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[  187.945690] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[  187.967264] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
[  188.138013] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [  188.180642] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  188.186470] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[  188.186472] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[  188.186625] iommu: Adding device 0000:81:10.0 to group 34
[  188.187137] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  188.187860] ixgbe 0000:81:00.0 enp129s0f0: VF Reset msg received from vf 0
[  188.188291] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[  188.198852] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  188.198854] ixgbevf 0000:81:10.0: Assigning random MAC address
[  188.199817] ixgbevf 0000:81:10.0: 42:0a:4d:f5:23:a7
[  188.199819] ixgbevf 0000:81:10.0: MAC: 1
[  188.199820] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  188.202735] ixgbevf 0000:81:10.0 enp129s16: renamed from eth0
[  188.284641] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ 
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ sudo ip addr add 192.168.2.116/24 dev enp129s16
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 0 | sudo tee sriov_numvfs 
0
[  211.716800] iommu: Removing device 0000:81:10.0 from group 34
[  212.844697] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[  212.929165] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[  213.202899] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [  213.373900] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[  213.520693] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 1 | sudo tee sriov_numvfs 
1
[  237.761147] ixgbe 0000:81:00.0 enp129s0f0: SR-IOV enabled with 1 VFs
[  237.761231] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[  237.876844] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[  237.899934] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
[  238.069911] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[  238.112636] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  238.112999] iommu: Adding device 0000:81:10.0 to group 34
[  238.113471] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  238.114120] ixgbe 0000:81:00.0 enp129s0f0: VF Reset msg received from vf 0
[  238.114552] ixgbe 0000:81:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [  238.125128] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  238.125130] ixgbevf 0000:81:10.0: Assigning random MAC address
[  238.126083] ixgbevf 0000:81:10.0: 6a:57:cf:1c:53:41
[  238.126085] ixgbevf 0000:81:10.0: MAC: 1
[  238.126086] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  238.129235] ixgbevf 0000:81:10.0 enp129s16: renamed from eth0
[  238.216639] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ sudo ip addr add 192.168.2.116/24 dev enp129s16
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ sudo ip link set dev enp129s16 up
[  253.791936] ixgbevf 0000:81:10.0: NIC Link is Up 10 Gbps
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [  253.793511] IPv6: ADDRCONF(NETDEV_UP): enp129s16: link is not ready
[  253.793528] IPv6: ADDRCONF(NETDEV_CHANGE): enp129s16: link becomes ready

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 0 | sudo tee sriov_numvfs 
0
[  259.572800] iommu: Removing device 0000:81:10.0 from group 34
[  260.716687] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[  260.802013] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[  261.089237] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [  261.261936] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[  261.408678] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$

Revision history for this message

Fred Kimmy (kongzizaixian) wrote on 2019-10-14:

I double check this reproduce step, I find in this past it will use "modprobe 8021q" to load this 8021q.ko module. Can you try it?

note: you should ensure that this vf link is success.

Revision history for this message

dann frazier (dannf) wrote on 2019-10-15:

@Fred: Please confirm the kernel version you are using.

You can see below that I've loaded 8021q and the VF reports a link up event:

ubuntu@segers:~$ cat /proc/version
Linux version 5.0.0-31-generic (buildd@bos02-arm64-011) (gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1)) #33~18.04.1-Ubuntu SMP Tue Oct 1 10:19:33 UTC 2019
ubuntu@segers:~$ sudo modprobe 8021q
ubuntu@segers:~$ sudo dmesg -c > /dev/null
ubuntu@segers:~$ dmesg -w &
[1] 3409
ubuntu@segers:~$ cd /sys/bus/pci/devices/0000\:81\:00.0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 1 | sudo tee sriov_numvfs
1
[ 796.517318] ixgbe 0000:81:00.0 enp129s0f0: SR-IOV enabled with 1 VFs
[ 796.517409] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[ 796.631941] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 796.658152] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
[ 796.829375] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [ 796.871843] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[ 796.880268] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[ 796.880270] ixgbevf: Copyright (c) 2009 - 2018 Intel Corporation.
[ 796.880462] iommu: Adding device 0000:81:10.0 to group 34
[ 796.881017] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[ 796.881184] ixgbe 0000:81:00.0 enp129s0f0: VF Reset msg received from vf 0
[ 796.892175] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[ 796.892177] ixgbevf 0000:81:10.0: Assigning random MAC address
[ 796.893210] ixgbevf 0000:81:10.0: 72:e9:09:1f:9f:50
[ 796.893212] ixgbevf 0000:81:10.0: MAC: 1
[ 796.893213] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[ 796.895518] ixgbevf 0000:81:10.0 enp129s16: renamed from eth0
[ 796.975848] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ sudo ip addr add 192.168.2.116/24 dev enp129s16
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 0 | sudo tee sriov_numvfs
0
[ 820.875839] iommu: Removing device 0000:81:10.0 from group 34
[ 822.007733] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[ 822.077184] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[ 822.403546] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [ 822.576974] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[ 822.723731] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$

@Fred: Please confirm the kernel version you are using.

You can see below that I've loaded 8021q and the VF reports a link up event:

ubuntu@segers:~$ cat /proc/version
Linux version 5.0.0-31-generic (buildd@bos02-arm64-011) (gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1)) #33~18.04.1-Ubuntu SMP Tue Oct 1 10:19:33 UTC 2019
ubuntu@segers:~$ sudo modprobe 8021q
ubuntu@segers:~$ sudo dmesg -c > /dev/null
ubuntu@segers:~$ dmesg -w &
[1] 3409
ubuntu@segers:~$ cd /sys/bus/pci/devices/0000\:81\:00.0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 1 | sudo tee sriov_numvfs 
1
[  796.517318] ixgbe 0000:81:00.0 enp129s0f0: SR-IOV enabled with 1 VFs
[  796.517409] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[  796.631941] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[  796.658152] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
[  796.829375] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [  796.871843] pci 0000:81:10.0: [8086:10ed] type 00 class 0x020000
[  796.880268] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[  796.880270] ixgbevf: Copyright (c) 2009 - 2018 Intel Corporation.
[  796.880462] iommu: Adding device 0000:81:10.0 to group 34
[  796.881017] ixgbevf 0000:81:10.0: enabling device (0000 -> 0002)
[  796.881184] ixgbe 0000:81:00.0 enp129s0f0: VF Reset msg received from vf 0
[  796.892175] ixgbevf 0000:81:10.0: MAC address not assigned by administrator.
[  796.892177] ixgbevf 0000:81:10.0: Assigning random MAC address
[  796.893210] ixgbevf 0000:81:10.0: 72:e9:09:1f:9f:50
[  796.893212] ixgbevf 0000:81:10.0: MAC: 1
[  796.893213] ixgbevf 0000:81:10.0: Intel(R) 82599 Virtual Function
[  796.895518] ixgbevf 0000:81:10.0 enp129s16: renamed from eth0
[  796.975848] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ sudo ip addr add 192.168.2.116/24 dev enp129s16
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ echo 0 | sudo tee sriov_numvfs 
0
[  820.875839] iommu: Removing device 0000:81:10.0 from group 34
[  822.007733] ixgbe 0000:81:00.0: removed PHC on enp129s0f0
[  822.077184] ixgbe 0000:81:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[  822.403546] ixgbe 0000:81:00.0: registered PHC device on enp129s0f0
ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$ [  822.576974] ixgbe 0000:81:00.0 enp129s0f0: detected SFP+: 5
[  822.723731] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

ubuntu@segers:/sys/bus/pci/devices/0000:81:00.0$

Revision history for this message

Fred Kimmy (kongzizaixian) wrote on 2019-10-21:

ok, I will repeatly check it.

Revision history for this message

Ike Panhc (ikepanhc) wrote on 2020-02-23:

#10

19.04 is EOLed. There is no more SRU cycle for it.

Revision history for this message

Taihsiang Ho (tai271828) wrote on 2020-03-24:

#11

Fred will check again according to the discussion today.

Revision history for this message

Fred Kimmy (kongzizaixian) wrote on 2020-03-25:

#12

Download full text (9.8 KiB)

uname -a
Linux localhost 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:41:03 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
root@localhost:/sys/bus/pci/devices/0000:82:00.0#
root@localhost:/sys/bus/pci/devices/0000:82:00.0# ethtool -i enp130s16
driver: ixgbevf
version: 4.1.0-k
firmware-version:
expansion-rom-version:
bus-info: 0000:82:10.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
root@localhost:/sys/bus/pci/devices/0000:82:00.0# ifconfig enp130s16 192.168.2.60
0
[ 2127.818575] ixgbevf 0000:82:10.0: NIC Link is Up 10 Gbps
[ 2127.820181] IPv6: ADDRCONF(NETDEV_UP): enp130s16: link is not ready
[ 2127.830127] IPv6: ADDRCONF(NETDEV_CHANGE): enp130s16: link becomes ready
root@localhost:/sys/bus/pci/devices/0000:82:00.0# dmesg | grep VLAN
root@localhost:/sys/bus/pci/devices/0000:82:00.0#
root@localhost:/sys/bus/pci/devices/0000:82:00.0# echo 0 > sriov_numvfs
[ 2189.635335] iommu: Removing device 0000:82:10.0 from group 48
[ 2190.783237] ixgbe 0000:82:00.0: removed PHC on enp130s0f0
[ 2190.872295] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Down
[ 2190.876305] ixgbe 0000:82:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[ 2191.149771] ixgbe 0000:82:00.0: registered PHC device on enp130s0f0
root@localhost:/sys/bus/pci/devices/0000:82:00.0#
root@localhost:/sys/bus/pci/devices/0000:82:00.0# [ 2191.328131] ixgbe 0000:82:00.0 enp130s0f0: detected SFP+: 5
[ 2191.530229] ixgbe 0000:82:00.0 enp130s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 2191.571239] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
root@localhost:/sys/bus/pci/devices/0000:82:00.0#
root@localhost:/sys/bus/pci/devices/0000:82:00.0# echo 0 > sriov_numvfsdmesg | grep -i VLANVLANifconfig enp130s16 192.168.2.600ethtool -i enp130s16

uname -a
Linux localhost 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:41:03 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# ethtool -i enp130s16
driver: ixgbevf
version: 4.1.0-k
firmware-version: 
expansion-rom-version: 
bus-info: 0000:82:10.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
root@localhost:/sys/bus/pci/devices/0000:82:00.0# ifconfig enp130s16 192.168.2.60
0
[ 2127.818575] ixgbevf 0000:82:10.0: NIC Link is Up 10 Gbps
[ 2127.820181] IPv6: ADDRCONF(NETDEV_UP): enp130s16: link is not ready
[ 2127.830127] IPv6: ADDRCONF(NETDEV_CHANGE): enp130s16: link becomes ready 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# dmesg | grep VLAN
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# echo 0 > sriov_numvfs
[ 2189.635335] iommu: Removing device 0000:82:10.0 from group 48
[ 2190.783237] ixgbe 0000:82:00.0: removed PHC on enp130s0f0
[ 2190.872295] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Down
[ 2190.876305] ixgbe 0000:82:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[ 2191.149771] ixgbe 0000:82:00.0: registered PHC device on enp130s0f0
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# [ 2191.328131] ixgbe 0000:82:00.0 enp130s0f0: detected SFP+: 5
[ 2191.530229] ixgbe 0000:82:00.0 enp130s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 2191.571239] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# echo 0 > sriov_numvfsdmesg | grep -i VLANVLANifconfig enp130s16 192.168.2.600ethtool -i enp130s16

[K[A[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[9Pifconfig -a[3Puname[C[C[Cecho 1 > sriov_numvfscd  /sys/bus/pci/devices/0000:8
82:00.0[A[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[C[3Pat /proc/sys/kernel/printk
echo "7 4 1 7" > /proc/sys/kernnel/printk dmesg 
-C 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# echo 1 > sriov_numvfs
[ 2279.031230] ixgbe 0000:82:00.0 enp130s0f0: SR-IOV enabled with 1 VFs
[ 2279.037659] ixgbe 0000:82:00.0: removed PHC on enp130s0f0
[ 2279.128322] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Down
[ 2279.174546] ixgbe 0000:82:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[ 2279.207126] ixgbe 0000:82:00.0: registered PHC device on enp130s0f0
[ 2279.384169] ixgbe 0000:82:00.0 enp130s0f0: detected SFP+: 5
[ 2279.427245] pci 0000:82:10.0: can't set Max Payload Size to 512; if necessary, use "pci=pcie_bus_safe" and report a bug
[ 2279.438362] iommu: Adding device 0000:82:10.0 to group 48
[ 2279.444356] ixgbevf 0000:82:10.0: enabling device (0000 -> 0002)
[ 2279.452373] ixgbe 0000:82:00.0 enp130s0f0: VF Reset msg received from vf 0
[ 2279.459673] ixgbe 0000:82:00.0: VF 0 has no MAC address assigned, you may have to assign one manually
[ 2279.479494] ixgbevf 0000:82:10.0: MAC address not assigned by administrator.
[ 2279.486517] ixgbevf 0000:82:10.0: Assigning random MAC address
[ 2279.493544] ixgbevf 0000:82:10.0: ca:64:35:84:b2:e7
[ 2279.498439] ixgbevf 0000:82:10.0: MAC: 1
[ 2279.498991] ixgbevf 0000:82:10.0 enp130s16: renamed from eth0
[ 2279.508082] ixgbevf 0000:82:10.0: Intel(R) 82599 Virtual Function
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# [ 2279.606707] ixgbe 0000:82:00.0 enp130s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 2280.499245] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
root@localhost:/sys/bus/pci/devices/0000:82:00.0# ifconfig enp130s16 192.168.2.60
0
[ 2347.250581] ixgbevf 0000:82:10.0: NIC Link is Up 10 Gbps
[ 2347.252196] IPv6: ADDRCONF(NETDEV_UP): enp130s16: link is not ready
[ 2347.262140] IPv6: ADDRCONF(NETDEV_CHANGE): enp130s16: link becomes ready
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0#    
root@localhost:/sys/bus/pci/devices/0000:82:00.0# cd /
root@localhost:/# lsmod | grep vlan
root@localhost:/# lsmod | grep vlan98021
root@localhost:/lib# 
root@localhost:/lib# 
root@localhost:/lib# find -name *.ko | grep vlan_core
./modules/4.15.0-29-generic/kernel/net/sched/act_vlan.ko
./modules/4.15.0-29-generic/kernel/net/bridge/netfilter/ebt_vlan.ko
./modules/4.15.0-29-generic/kernel/drivers/net/ipvlan/ipvtap.ko
./modules/4.15.0-29-generic/kernel/drivers/net/ipvlan/ipvlan.ko
./modules/4.15.0-29-generic/kernel/drivers/net/macvlan.ko
root@localhost:~# modprobe 8021q
[ 9425.244330] 8021q: 802.1Q VLAN Support v1.8
[ 9425.248519] 8021q: adding VLAN 0 to HW filter on device enp129s0f0
[ 9425.254695] 8021q: adding VLAN 0 to HW filter on device enp130s0f0
[ 9425.260869] 8021q: adding VLAN 0 to HW filter on device enp130s16 
root@localhost:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:18:2d:00:00:59 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:18:2d:01:00:59 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:18:2d:02:00:59 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:18:2d:03:00:59 brd ff:ff:ff:ff:ff:ff
6: enp3s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c6:cc:ec:d7:58 brd ff:ff:ff:ff:ff:ff
7: enp3s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c6:cc:ec:d7:59 brd ff:ff:ff:ff:ff:ff
8: enp129s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 68:a8:28:2e:c9:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.40/24 brd 192.168.2.255 scope global enp129s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::6aa8:28ff:fe2e:c910/64 scope link 
       valid_lft forever preferred_lft forever
9: enp3s0f2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c6:cc:ec:d7:5a brd ff:ff:ff:ff:ff:ff
10: enp3s0f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 0c:c6:cc:ec:d7:5b brd ff:ff:ff:ff:ff:ff
11: enp1s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 50:6b:4b:c3:62:10 brd ff:ff:ff:ff:ff:ff
12: enp1s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 50:6b:4b:c3:62:11 brd ff:ff:ff:ff:ff:ff
13: enp129s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 68:a8:28:2e:c9:11 brd ff:ff:ff:ff:ff:ff
14: enp130s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether c8:1f:be:26:71:de brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.50/24 brd 192.168.2.255 scope global enp130s0f0
       valid_lft forever preferred_lft forever
    inet6 fe80::ca1f:beff:fe26:71de/64 scope link 
       valid_lft forever preferred_lft forever
15: enp130s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether c8:1f:be:26:71:df brd ff:ff:ff:ff:ff:ff
17: enp130s16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 76:35:aa:37:af:05 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.60/24 brd 192.168.2.255 scope global enp130s16
       valid_lft forever preferred_lft forever
    inet6 fe80::7435:aaff:fe37:af05/64 scope link 
       valid_lft forever preferred_lft forever
root@localhost:~# cd  /sys/bus/pci/devices/0000:82:00.0
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# cat sriov_numvfs 
1
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# echo 0 > sriov_numvfs

[ 9513.555672] failed to kill vid 0081/0 for device enp130s16
[ 9513.583367] iommu: Removing device 0000:82:10.0 from group 48
[ 9514.719232] ixgbe 0000:82:00.0: removed PHC on enp130s0f0
[ 9514.808292] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Down
[ 9514.812064] ixgbe 0000:82:00.0: Multiqueue Enabled: Rx Queue count = 63, Tx Queue count = 63 XDP Queue count = 0
[ 9515.109867] ixgbe 0000:82:00.0: registered PHC device on enp130s0f0
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# [ 9515.288076] ixgbe 0000:82:00.0 enp130s0f0: detected SFP+: 5
[ 9515.536483] ixgbe 0000:82:00.0 enp130s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[ 9516.147241] ixgbe 0000:81:00.0 enp129s0f0: NIC Link is Up 10 Gbps, Flow Control: RX/TX

root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# uname -a
Linux localhost 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:41:03 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0# 
root@localhost:/sys/bus/pci/devices/0000:82:00.0#

Ike Panhc (ikepanhc) on 2020-07-01

tags:

added: ikeradar

Revision history for this message

Ike Panhc (ikepanhc) wrote on 2020-07-14:

#13

I try to reproduce on 4.15.0-111.112 (for both KVM host/guest) but failed

KVM host setup:
$ echo 1 | sudo tee /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs
and lspci shows 82559 devices are
$ lspci | grep 82599
87:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
87:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
87:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

KVM guest setup:
$ sudo virsh edit test #on host edit guest config and add VF interface
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:23:7b:d6'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x87' slot='0x10' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>

Power on guest:
$ sudo virsh start test # on host

Verify VF interface work-able
$ sudo dhclient enp4s0 # on guest
cmp: EOF on /tmp/tmp.e2o216Un50 which is empty
$ ifconfig enp4s0 # on guest
enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.228.68.63 netmask 255.255.255.0 broadcast 10.228.68.255

Shutdown VF on host
$ echo 0 | sudo tee /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs # on host

Check dmesg on host
$ dmesg | grep ixgbevf
[37430.312863] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
[37430.312868] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[37430.314584] ixgbevf 0000:87:10.0: enabling device (0000 -> 0002)
[37430.409375] ixgbevf 0000:87:10.0: MAC address not assigned by administrator.
[37430.409383] ixgbevf 0000:87:10.0: Assigning random MAC address
[37430.458478] ixgbevf 0000:87:10.0: f2:35:fd:fe:39:a3
[37430.458490] ixgbevf 0000:87:10.0: MAC: 1
[37430.458497] ixgbevf 0000:87:10.0: Intel(R) 82599 Virtual Function
[37430.494272] ixgbevf 0000:87:10.0 enp135s16: renamed from eth0

Check dmesg on guest
$ dmesg | grep pcie004 | tail -4
[ 610.086723] pciehp 0000:00:01.3:pcie004: Slot(0-3): Attention button pressed
[ 610.087195] pciehp 0000:00:01.3:pcie004: Slot(0-3): Powering off due to button press
[ 617.290496] pciehp 0000:00:01.3:pcie004: Slot(0-3): Card not present
[ 617.290604] pciehp 0000:00:01.3:pcie004: Slot(0-3): Already disabled

and enp4s0 is gone in guest.

I will use 4.15.0-29.31 to check again, but looks we don't have this issue on bionic GA -update kernel.

I try to reproduce on 4.15.0-111.112 (for both KVM host/guest) but failed

KVM host setup:
 $ echo 1 | sudo tee /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs
and lspci shows 82559 devices are
 $ lspci | grep 82599
 87:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
 87:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
 87:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)

KVM guest setup:
 $ sudo virsh edit test #on host edit guest config and add VF interface
    <interface type='hostdev' managed='yes'>
      <mac address='52:54:00:23:7b:d6'/>
      <driver name='vfio'/>
      <source>
        <address type='pci' domain='0x0000' bus='0x87' slot='0x10' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </interface>

Power on guest:
 $ sudo virsh start test # on host

Verify VF interface work-able
 $ sudo dhclient enp4s0 # on guest
cmp: EOF on /tmp/tmp.e2o216Un50 which is empty
 $ ifconfig enp4s0 # on guest
 enp4s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.228.68.63  netmask 255.255.255.0  broadcast 10.228.68.255

Shutdown VF on host
 $ echo 0 | sudo tee /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs # on host

Check dmesg on host
 $ dmesg | grep ixgbevf
 [37430.312863] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k
 [37430.312868] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
 [37430.314584] ixgbevf 0000:87:10.0: enabling device (0000 -> 0002)
 [37430.409375] ixgbevf 0000:87:10.0: MAC address not assigned by administrator.
 [37430.409383] ixgbevf 0000:87:10.0: Assigning random MAC address
 [37430.458478] ixgbevf 0000:87:10.0: f2:35:fd:fe:39:a3
 [37430.458490] ixgbevf 0000:87:10.0: MAC: 1
 [37430.458497] ixgbevf 0000:87:10.0: Intel(R) 82599 Virtual Function
 [37430.494272] ixgbevf 0000:87:10.0 enp135s16: renamed from eth0

Check dmesg on guest
 $ dmesg | grep pcie004 | tail -4
 [  610.086723] pciehp 0000:00:01.3:pcie004: Slot(0-3): Attention button pressed
 [  610.087195] pciehp 0000:00:01.3:pcie004: Slot(0-3): Powering off due to button press
 [  617.290496] pciehp 0000:00:01.3:pcie004: Slot(0-3): Card not present
 [  617.290604] pciehp 0000:00:01.3:pcie004: Slot(0-3): Already disabled

and enp4s0 is gone in guest.

I will use 4.15.0-29.31 to check again, but looks we don't have this issue on bionic GA -update kernel.

Revision history for this message

Ike Panhc (ikepanhc) wrote on 2020-07-14:

#14

Can not reproduce with same steps as #13 on 4.15.0-29.31

Revision history for this message

Ike Panhc (ikepanhc) wrote on 2020-07-15:

#15

Revisit bug description and find out I might misunderstood the problem. I shall set IP for VF on host not setup KVM guest.

Re-test on 4.15.0-111.112 but still can not reproduce.

ubuntu@kreiken:~$ echo 1 | sudo tee /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs
1
ubuntu@kreiken:~$ lspci | grep 82599
87:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
87:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
87:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
ubuntu@kreiken:~$ sudo ifconfig enp135s16 192.168.100.1
ubuntu@kreiken:~$ sudo ethtool enp135s16
Settings for enp135s16:
Supported ports: [ ]
Supported link modes: 10000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: Other
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
ubuntu@kreiken:~$ echo 0 | sudo tee /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs
0
ubuntu@kreiken:~$ uname -a
Linux kreiken 4.15.0-111-generic #112-Ubuntu SMP Thu Jul 9 20:33:50 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
ubuntu@kreiken:~$ dmesg | grep 'failed to'

Revision history for this message

Ike Panhc (ikepanhc) wrote on 2020-07-15:

#16

Can not reproduce on 4.15.0-29.31 either.

@Xinwei, is there anything I missed? My step-by-step is

1) echo 1 | sudo tee /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs
2) sudo ifconfig enp135s16 192.168.100.1
3) sudo ethtool enp135s16 # Make sure VF has link
4) echo 0 | sudo tee /sys/bus/pci/devices/0000:87:00.0/sriov_numvfs
5) dmesg | grep 'failed to'

Revision history for this message

Ike Panhc (ikepanhc) wrote on 2020-07-20:

#17

https://lists.ubuntu.com/archives/ubuntu-announce/2020-July/000258.html

Eoan has been EOLed.

Revision history for this message

Ike Panhc (ikepanhc) wrote on 2020-08-05:

#18

Hi Xinwei,

Could you please check steps in #16 and see if there is anything missing?

If we can not reproduce this issue anymore, I think we can close this bug.

Revision history for this message

Fred Kimmy (kongzizaixian) wrote on 2020-08-27:

#19

Can you close this bug, if you reproduce it and i WILL repened it.

Revision history for this message

Ike Panhc (ikepanhc) wrote on 2020-08-27:

#20

Okay. Since we all can not reproduce it. Close this bug for now and we can re-open if we see this issue again.

Changed in kunpeng920:
status:	Incomplete → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

auto-bugs.centos.org #16342 Edit

Bug watches keep track of this bug in other bug trackers.