SR-IOV interface goes down when testing network performance
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
Hello,
We are testing network performance with SR-IOV on ubuntu and security onion in Rhev Hypervisor. We are using a SuperMicro box running T-rex traffic generator on CentOS which is connected to a gigamon to push traffic to the Ubuntu/Security Onion VMs.
The ubuntu 16.04 vm is running on a RHVH host node and is running SR-IOV. We have ensured that SR-IOV is enabled in the BIOS. We are finding that packets stop coming into the interface after a few seconds. This is because the interface is going down after a few seconds of receiving traffic. We are running promiscuous mode and it is required for our use case of measuring network traffic.
As a troubleshooting measure, 5.3.0-rc3 was built using the 3.2.3 iavf SR-IOV driver. Note that the i40evf driver was renamed to iavf, which is why the name discrepancy exists. After updating the Kernel we stopped seeing this issue.
Non-working default security onion kernel information
root@ubuntu:~# uname -r
3.10.0-
root@ubuntu:~# ethtool -i ens7
driver: iavf
version: 3.0.1-k
firmware-version: N/A
expansion-
bus-info: 0000:00:07.0
supports-
supports-test: no
supports-
supports-
supports-
Working modified security onion kernel information
root@ubuntu:~# uname -r
5.3.0-rc3-custom
root@ubuntu:~# ethtool -i ens7
driver: i40evf
version: 3.2.3-k
firmware-version: N/A
expansion-
bus-info: 0000:00:07.0
supports-
supports-test: no
supports-
supports-
supports-
We were able to perform testing using the 5.3.0 kernel, but long term a better solution is needed as this would require supporting a modified Linux kernel with updates, testing, etc.
From the hypervisor in /var/log/messages:
Aug 27 09:05:00 localhost libvirtd: 2019-08-27 13:05:00.522+0000: 10748: error : qemuDomainAgent
Aug 27 09:05:07 localhost goferd: [ERROR][worker-0] gofer.messaging
Aug 27 09:05:07 localhost goferd: [INFO][worker-0] gofer.messaging
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
From the VM, journalctl -xe gives:
-- The start-up result is done.
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: PF reset warning received
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: Scheduling reset task
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:08 ubuntu systemd-
-- Subject: Session 16 has been terminated
-- Defined-By: systemd
-- Support: http://
-- Documentation: http://
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Aug 28 17:06 seq
crw-rw---- 1 root audio 116, 33 Aug 28 17:06 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 16.04
HibernationDevice: RESUME=
InstallationDate: Installed on 2019-08-21 (6 days ago)
InstallationMedia: Ubuntu-Server 16.04.6 LTS "Xenial Xerus" - Release amd64 (20190226)
IwConfig: Error: [Errno 2] No such file or directory
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Red Hat RHEV Hypervisor
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=linux
PATH=(custom, no user)
XDG_RUNTIME_
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 qxldrmfb
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware 1.157.21
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial
Uname: Linux 4.15.0-45-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.11.0-2.el7
dmi.chassis.type: 1
dmi.chassis.vendor: Red Hat
dmi.chassis.
dmi.modalias: dmi:bvnSeaBIOS:
dmi.product.family: Red Hat Enterprise Linux
dmi.product.name: RHEV Hypervisor
dmi.product.
dmi.sys.vendor: Red Hat
affects: | ubuntu → kernel-package (Ubuntu) |
description: | updated |
affects: | kernel-package (Ubuntu) → linux (Ubuntu) |
Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https:/ /wiki.ubuntu. com/Bugs/ FindRightPackag e. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.
To change the source package that this bug is filed about visit https:/ /bugs.launchpad .net/ubuntu/ +bug/1841669/ +editstatus and add the package name in the text box next to the word Package.
[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]