Hello,
We are testing network performance with SR-IOV on ubuntu and security onion in Rhev Hypervisor. We are using a SuperMicro box running T-rex traffic generator on CentOS which is connected to a gigamon to push traffic to the Ubuntu/Security Onion VMs.
The ubuntu 16.04 vm is running on a RHVH host node and is running SR-IOV. We have ensured that SR-IOV is enabled in the BIOS. We are finding that packets stop coming into the interface after a few seconds. This is because the interface is going down after a few seconds of receiving traffic. We are running promiscuous mode and it is required for our use case of measuring network traffic.
As a troubleshooting measure, 5.3.0-rc3 was built using the 3.2.3 iavf SR-IOV driver. Note that the i40evf driver was renamed to iavf, which is why the name discrepancy exists. After updating the Kernel we stopped seeing this issue.
Non-working default security onion kernel information
root@ubuntu:~# uname -r
3.10.0-957.21.3.el7.x86_64
root@ubuntu:~# ethtool -i ens7
driver: iavf
version: 3.0.1-k
firmware-version: N/A
expansion-rom-version:
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
Working modified security onion kernel information
root@ubuntu:~# uname -r
5.3.0-rc3-custom
root@ubuntu:~# ethtool -i ens7
driver: i40evf
version: 3.2.3-k
firmware-version: N/A
expansion-rom-version:
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
We were able to perform testing using the 5.3.0 kernel, but long term a better solution is needed as this would require supporting a modified Linux kernel with updates, testing, etc.
From the hypervisor in /var/log/messages:
Aug 27 09:05:00 localhost libvirtd: 2019-08-27 13:05:00.522+0000: 10748: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Aug 27 09:05:07 localhost goferd: [ERROR][worker-0] gofer.messaging.adapter.conn ect:33 - connect: proton+amqps://subscription.rhsm.redhat.com:5647, failed: Conn ection amqps://subscription.rhsm.redhat.com:5647disconnected: Condition('amqp:r esource-limit-exceeded', 'local-idle-timeout expired')
Aug 27 09:05:07 localhost goferd: [INFO][worker-0] gofer.messaging.adapter.conne ct:35 - retry in 106 seconds
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
From the VM, journalctl -xe gives:
-- The start-up result is done.
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: PF reset warning received
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: Scheduling reset task
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:08 ubuntu systemd-logind[1251]: Removed session 16.
-- Subject: Session 16 has been terminated
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Aug 28 17:06 seq
crw-rw---- 1 root audio 116, 33 Aug 28 17:06 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 16.04
HibernationDevice: RESUME=/dev/mapper/ubuntu--vg-swap_1
InstallationDate: Installed on 2019-08-21 (6 days ago)
InstallationMedia: Ubuntu-Server 16.04.6 LTS "Xenial Xerus" - Release amd64 (20190226)
IwConfig: Error: [Errno 2] No such file or directory
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Red Hat RHEV Hypervisor
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=linux
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 qxldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.15.0-45-generic root=/dev/mapper/hostname--vg-root ro quiet splash isolcpus=2-27 vt.handoff=7
ProcVersionSignature: Ubuntu 4.15.0-45.48~16.04.1-generic 4.15.18
RelatedPackageVersions:
linux-restricted-modules-4.15.0-45-generic N/A
linux-backports-modules-4.15.0-45-generic N/A
linux-firmware 1.157.21
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial
Uname: Linux 4.15.0-45-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.11.0-2.el7
dmi.chassis.type: 1
dmi.chassis.vendor: Red Hat
dmi.chassis.version: RHEL 7.6.0 PC (i440FX + PIIX, 1996)
dmi.modalias: dmi:bvnSeaBIOS:bvr1.11.0-2.el7:bd04/01/2014:svnRedHat:pnRHEVHypervisor:pvr7.6-1.el7ev:cvnRedHat:ct1:cvrRHEL7.6.0PC(i440FX+PIIX,1996):
dmi.product.family: Red Hat Enterprise Linux
dmi.product.name: RHEV Hypervisor
dmi.product.version: 7.6-1.el7ev
dmi.sys.vendor: Red Hat
Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https:/
To change the source package that this bug is filed about visit https:/
[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]