VLAN creation fails with VF multi queueuing
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Expired
|
Undecided
|
Unassigned |
Bug Description
Description
------------
It can be read in SR-IOV companion guide that 82599 chipsets :
* The 82599 has 128 Transmit and 128 Receive queues. They are generally referred to/thought of as queue pairs (1 Transmit and 1 Receive queue). This gives the 82599 128 queue pairs.
* The 82599 has up to 64 pools, with each pool having two queue pairs in it. That is two Transmit and Receive queue assigned to each VF.
* The 82599 allows for a variable configuration for the number of pools. In SR-IOV mode, there can be 16, 32 or 64 pools.
The last line suggest using combined queues of 2, 4 and 8.
The only way, for now we found, is using tc on the host especially as the ethtool command is not supported (ethtool -L eth10 combined 8 for example).
And, with following command "tc qdisc add dev eth10 root mqprio num_tc 8", we will be able to use up to 8 combined queues on the host PF and guests which are using VFs.
Yet, with 8 combined queues with tc commnad, VLAN creation was not permitted.
Following driver versions have been used on host and guest to enable this multiqueue option:
* 4.3.15 ixgbe in the host
* 3.1.2 ixgbevf in the guest
To reproduce
-------------
On the host side
----------------
root@ubuntu1404:~# rmmod ixgbe
root@ubuntu1404:~# modprobe ixgbe
[ 4817.027637] ixgbe 0000:05:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
[ 4817.038630] ixgbe 0000:05:00.0: Reload the driver after installing a supported module.
[ 4817.418360] ixgbe 0000:41:00.1: failed to load because an unsupported SFP+ or QSFP module type was detected.
[ 4817.429354] ixgbe 0000:41:00.1: Reload the driver after installing a supported module.
[ 4817.838086] ixgbe 0000:43:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
[ 4817.849091] ixgbe 0000:43:00.0: Reload the driver after installing a supported module.
[ 4817.889189] ixgbe 0000:43:00.1: failed to load because an unsupported SFP+ or QSFP module type was detected.
[ 4817.900178] ixgbe 0000:43:00.1: Reload the driver after installing a supported module.
[ 4817.940318] ixgbe 0000:44:00.0: failed to load because an unsupported SFP+ or QSFP module type was detected.
[ 4817.951357] ixgbe 0000:44:00.0: Reload the driver after installing a supported module.
root@ubuntu1404:~# modinfo ixgbe
filename: /lib/modules/
version: 4.3.15
license: GPL
description: Intel(R) 10 Gigabit PCI Express Network Driver
author: Intel Corporation, <email address hidden>
srcversion: 7AED484083B2C5B
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
alias: pci:v00008086d0
depends: ptp,dca,vxlan
vermagic: 3.13.0-91-generic SMP mod_unload modversions
parm: InterruptType:
parm: IntMode:Change Interrupt Mode (0=Legacy, 1=MSI, 2=MSI-X), default 2 (array of int)
parm: MQ:Disable or enable Multiple Queues, default 1 (array of int)
parm: DCA:Disable or enable Direct Cache Access, 0=disabled, 1=descriptor only, 2=descriptor and data (array of int)
parm: RSS:Number of Receive-Side Scaling Descriptor Queues, default 0=number of cpus (array of int)
parm: VMDQ:Number of Virtual Machine Device Queues: 0/1 = disable, 2-16 enable (default=8) (array of int)
parm: max_vfs:Number of Virtual Functions: 0 = disable (default), 1-63 = enable this many VFs (array of int)
parm: VEPA:VEPA Bridge Mode: 0 = VEB (default), 1 = VEPA (array of int)
parm: InterruptThrott
parm: LLIPort:Low Latency Interrupt TCP Port (0-65535) (array of int)
parm: LLIPush:Low Latency Interrupt on TCP Push flag (0,1) (array of int)
parm: LLISize:Low Latency Interrupt on Packet Size (0-1500) (array of int)
parm: LLIEType:Low Latency Interrupt Ethernet Protocol Type (array of int)
parm: LLIVLANP:Low Latency Interrupt on VLAN priority threshold (array of int)
parm: FdirPballoc:Flow Director packet buffer allocation level:
1 = 8k hash filters or 2k perfect filters
2 = 16k hash filters or 4k perfect filters
3 = 32k hash filters or 8k perfect filters (array of int)
parm: AtrSampleRate:
parm: FCoE:Disable or enable FCoE Offload, default 1 (array of int)
parm: MDD:Malicious Driver Detection: (0,1), default 1 = on (array of int)
parm: LRO:Large Receive Offload (0,1), default 0 = off (array of int)
parm: allow_unsupport
parm: dmac_watchdog:DMA coalescing watchdog in microseconds (0,41-10000), default 0 = off (array of int)
parm: vxlan_rx:VXLAN receive checksum offload (0,1), default 1 = Enable (array of int)
root@ubuntu1404:~# modprobe vfio-pci
00.0/sriov_numvfs
sriov_numvfssys
sriov_numvfs404:~# echo "1" > /sys/devices/
root@ubuntu1404:~# resize
COLUMNS=211;
LINES=53;
export COLUMNS LINES;
root@ubuntu1404:~# ip link set dev eth1 vf 0 spoofchk off
et dev eth1 vf 0 mac de:ad:be:ef:00:00
ip link set dev eth9 vf 0 mac de:ad:be:ef:00:01
root@ubuntu1404:~# ip link set dev eth9 vf 0 spoofchk off
root@ubuntu1404:~# ip link set dev eth1 vf 0 mac de:ad:be:ef:00:00
root@ubuntu1404:~# ip link set dev eth9 vf 0 mac de:ad:be:ef:00:01
root@ubuntu1404:~#
root@ubuntu1404:~# rmmod ixgbevf
root@ubuntu1404:~# modprobe ixgbevf
root@ubuntu1404:~# ip link set eth1 up
root@ubuntu1404:~# ip link set eth9 up
root@ubuntu1404:~#
root@ubuntu1404:~# bridge link set dev eth1 hwmode veb
root@ubuntu1404:~# bridge link set dev eth9 hwmode veb
root@ubuntu1404:~#
root@ubuntu1404:~# echo "8086 10ed" > /sys/bus/
root@ubuntu1404:~# echo "0000:01:10.0" > /sys/bus/
root@ubuntu1404:~# echo "0000:01:10.1" > /sys/bus/
root@ubuntu1404:~# echo "0000:01:10.0" > /sys/bus/
root@ubuntu1404:~# echo "0000:01:10.1" > /sys/bus/
root@ubuntu1404:~#
root@ubuntu1404:~#
root@ubuntu1404:~# ethtool -l eth1
Channel parameters for eth1:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 1
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
root@ubuntu1404:~# ethtool -l eth9
Channel parameters for eth9:
Pre-set maximums:
RX: 0
TX: 0
Other: 1
Combined: 1
Current hardware settings:
RX: 0
TX: 0
Other: 1
Combined: 4
root@ubuntu1404:~# tc qdisc add dev eth1 root mqprio num_tc 8
root@ubuntu1404:~# tc qdisc add dev eth9 root mqprio num_tc 8
root@ubuntu1404:~#
root@ubuntu1404:~# taskset -c 0,2,26,
> -serial telnet:
> -monitor telnet:
> -device virtio-
> -netdev user,id=
> -hda centos-
> -device vfio-pci,
[1] 2851
root@ubuntu1404:~# [ 4917.696698] kvm [2851]: vcpu0 unhandled rdmsr: 0x1c9
[ 4917.702297] kvm [2851]: vcpu0 unhandled rdmsr: 0x1a6
[ 4917.707877] kvm [2851]: vcpu0 unhandled rdmsr: 0x1a7
[ 4917.713456] kvm [2851]: vcpu0 unhandled rdmsr: 0x3f6
[ 4918.731234] kvm [2851]: vcpu2 unhandled rdmsr: 0x606
[ 4922.596487] kvm [2851]: vcpu12 unhandled rdmsr: 0xd90
On the VM side
---------------
CentOS Linux 7 (Core)
Kernel 3.10.0-
centos7 login: root
Password:
Last login: Tue Sep 6 17:33:59 on ttyS0
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
3: ens5: <BROADCAST,
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
4: mgmt0: <BROADCAST,
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0
valid_lft 86256sec preferred_lft 86256sec
inet6 fe80::dcad:
valid_lft forever preferred_lft forever
[root@centos7 ~]# ip li set ens4 up
[ 162.228284] ixgbevf 0000:00:04.0: NIC Link is Up 10 Gbps
[root@centos7 ~]# ip li set ens5 up
[ 166.828243] ixgbevf 0000:00:05.0: NIC Link is Up 10 Gbps
[ 166.830297] IPv6: ADDRCONF(
[ 166.831105] IPv6: ADDRCONF(
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:
valid_lft forever preferred_lft forever
3: ens5: <BROADCAST,
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:
valid_lft forever preferred_lft forever
4: mgmt0: <BROADCAST,
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0
valid_lft 86238sec preferred_lft 86238sec
inet6 fe80::dcad:
valid_lft forever preferred_lft forever
[root@centos7 ~]# ip link add link ens4 name vlan1 type vlan id 1
[ 198.884719] 8021q: 802.1Q VLAN Support v1.8
[ 198.885423] 8021q: adding VLAN 0 to HW filter on device ens4
[ 198.886916] 8021q: adding VLAN 0 to HW filter on device ens5
[ 198.891302] 8021q: adding VLAN 0 to HW filter on device mgmt0
RTNETLINK answers: Permission denied
[root@centos7 ~]# ip link add link ens5 name vlan2 type vlan id 2
RTNETLINK answers: Permission denied
[root@centos7 ~]# ip a
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:
valid_lft forever preferred_lft forever
3: ens5: <BROADCAST,
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:
valid_lft forever preferred_lft forever
4: mgmt0: <BROADCAST,
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic mgmt0
valid_lft 86188sec preferred_lft 86188sec
inet6 fe80::dcad:
valid_lft forever preferred_lft forever
[root@centos7 ~]# tail -10 /var/log/messages
Sep 6 18:56:13 centos7 systemd: Started Session 1 of user root.
Sep 6 18:56:13 centos7 systemd: Starting Session 1 of user root.
Sep 6 18:56:39 centos7 kernel: ixgbevf 0000:00:04.0: NIC Link is Up 10 Gbps
Sep 6 18:56:43 centos7 kernel: ixgbevf 0000:00:05.0: NIC Link is Up 10 Gbps
Sep 6 18:56:43 centos7 kernel: IPv6: ADDRCONF(
Sep 6 18:56:43 centos7 kernel: IPv6: ADDRCONF(
Sep 6 18:57:15 centos7 kernel: 8021q: 802.1Q VLAN Support v1.8
Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device ens4
Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device ens5
Sep 6 18:57:15 centos7 kernel: 8021q: adding VLAN 0 to HW filter on device mgmt0
[root@centos7 ~]#
Even with an ubuntu-14.04 VM, issue can be reproduced:
root@ubuntu1404:~# modinfo ixgbevf
filename: /lib/modules/
version: 2.16.1
license: GPL
description: Intel(R) 10 Gigabit Virtual Function Network Driver
author: Intel Corporation, <email address hidden>
srcversion: BD04DA0615170C3
alias: pci:v00008086d0
alias: pci:v00008086d0
depends:
vermagic: 3.13.0-92-generic SMP mod_unload modversions
parm: InterruptThrott
root@ubuntu1404:~# uname -a
Linux ubuntu1404 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu1404:~# lsb_release -da
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
root@ubuntu1404:~# ip a
1: lo: <LOOPBACK,
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: mgmt0: <BROADCAST,
link/ether de:ad:de:01:02:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global mgmt0
valid_lft forever preferred_lft forever
inet6 fe80::dcad:
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,
link/ether de:ad:be:ef:00:00 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,
link/ether de:ad:be:ef:00:01 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:
valid_lft forever preferred_lft forever
root@ubuntu1404:~# ip link add link eth1 name vlan1 type vlan id 1
RTNETLINK answers: Permission denied
root@ubuntu1404:~# ip link add link eth2 name vlan2 type vlan id 2
RTNETLINK answers: Permission denied
root@ubuntu1404:~#
Thus, the VLAN creation is not supported with VF queueuing enabled (8 combined queues which is the maximum).
Yet, with default queues (without typing tc command), VLANs were properly created.
affects: | dkms (Ubuntu) → linux (Ubuntu) |
This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:
apport-collect 1621474
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.