| 2019-08-27 20:56:44 |
Tristan Sjolie |
bug |
|
|
added bug |
| 2019-08-28 00:26:46 |
Ubuntu Foundations Team Bug Bot |
tags |
|
bot-comment |
|
| 2019-08-28 12:43:07 |
Tristan Sjolie |
affects |
ubuntu |
kernel-package (Ubuntu) |
|
| 2019-08-28 12:47:08 |
Tristan Sjolie |
description |
Hello,
We are testing network performance with SR-IOV and bridging on ubuntu and security onion in Rhev Hypervisor which is connected to a SuperMicro box running T-rex traffic generator on CentOS which is connected to a gigmon.
The ubuntu 16.04 vm is running on a RHVH host node and is running SR-IOV. We have ensured that SR_IOV is enabled in the BIOS. We are finding that on security onion that the packets are dropping after a few seconds. This is because the interface is going down after a certain point. We found that packets are getting dropped as soon as the interface shows as down in ip -s link. We are running promiscuous mode and it is required for our use case of measuring network traffic. Once this process was completed, Kernel 5.3.0-rc3 was built using the 3.2.3 iavf SR-IOV driver. Note that the i40evf driver was renamed to iavf, which is why the name discrepancy exists. After updating the Kernel we stopped seeing this issue.
Non-working default security onion kernel information
root@ubuntu:~# uname -r
3.10.0-957.21.3.el7.x86_64
root@ubuntu:~# ethtool -i ens7
driver: iavf
version: 3.0.1-k
firmware-version: N/A
expansion-rom-version:
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
Working modified security onion kernel information
root@ubuntu:~# uname -r
5.3.0-rc3-custom
root@ubuntu:~# ethtool -i ens7
driver: i40evf
version: 3.2.3-k
firmware-version: N/A
expansion-rom-version:
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
We were able to perform testing using the 5.3.0 kernel, but long term a better solution is needed as this would require supporting a modified Linux kernel with updates, testing, etc.
From the hypervisor in /var/log/messages:
Aug 27 09:05:00 localhost libvirtd: 2019-08-27 13:05:00.522+0000: 10748: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Aug 27 09:05:07 localhost goferd: [ERROR][worker-0] gofer.messaging.adapter.conn ect:33 - connect: proton+amqps://subscription.rhsm.redhat.com:5647, failed: Conn ection amqps://subscription.rhsm.redhat.com:5647disconnected: Condition('amqp:r esource-limit-exceeded', 'local-idle-timeout expired')
Aug 27 09:05:07 localhost goferd: [INFO][worker-0] gofer.messaging.adapter.conne ct:35 - retry in 106 seconds
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
From the VM, journalctl -xe gives:
-- The start-up result is done.
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: PF reset warning received
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: Scheduling reset task
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:08 ubuntu systemd-logind[1251]: Removed session 16.
-- Subject: Session 16 has been terminated
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat |
Hello,
We are testing network performance with SR-IOV on ubuntu and security onion in Rhev Hypervisor. We are using a SuperMicro box running T-rex traffic generator on CentOS which is connected to a gigamon to push traffic to the Ubuntu/Security Onion VMs.
The ubuntu 16.04 vm is running on a RHVH host node and is running SR-IOV. We have ensured that SR-IOV is enabled in the BIOS. We are finding that packets stop coming into the interface after a few seconds. This is because the interface is going down after a few seconds of receiving traffic. We are running promiscuous mode and it is required for our use case of measuring network traffic.
As a troubleshooting measure, 5.3.0-rc3 was built using the 3.2.3 iavf SR-IOV driver. Note that the i40evf driver was renamed to iavf, which is why the name discrepancy exists. After updating the Kernel we stopped seeing this issue.
Non-working default security onion kernel information
root@ubuntu:~# uname -r
3.10.0-957.21.3.el7.x86_64
root@ubuntu:~# ethtool -i ens7
driver: iavf
version: 3.0.1-k
firmware-version: N/A
expansion-rom-version:
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
Working modified security onion kernel information
root@ubuntu:~# uname -r
5.3.0-rc3-custom
root@ubuntu:~# ethtool -i ens7
driver: i40evf
version: 3.2.3-k
firmware-version: N/A
expansion-rom-version:
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
We were able to perform testing using the 5.3.0 kernel, but long term a better solution is needed as this would require supporting a modified Linux kernel with updates, testing, etc.
From the hypervisor in /var/log/messages:
Aug 27 09:05:00 localhost libvirtd: 2019-08-27 13:05:00.522+0000: 10748: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Aug 27 09:05:07 localhost goferd: [ERROR][worker-0] gofer.messaging.adapter.conn ect:33 - connect: proton+amqps://subscription.rhsm.redhat.com:5647, failed: Conn ection amqps://subscription.rhsm.redhat.com:5647disconnected: Condition('amqp:r esource-limit-exceeded', 'local-idle-timeout expired')
Aug 27 09:05:07 localhost goferd: [INFO][worker-0] gofer.messaging.adapter.conne ct:35 - retry in 106 seconds
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
From the VM, journalctl -xe gives:
-- The start-up result is done.
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: PF reset warning received
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: Scheduling reset task
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:08 ubuntu systemd-logind[1251]: Removed session 16.
-- Subject: Session 16 has been terminated
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat |
|
| 2019-08-28 16:04:37 |
Brian Murray |
affects |
kernel-package (Ubuntu) |
linux (Ubuntu) |
|
| 2019-08-28 16:08:44 |
Ubuntu Kernel Bot |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2019-08-28 17:36:52 |
Tristan Sjolie |
tags |
bot-comment |
apport-collected bot-comment xenial |
|
| 2019-08-28 17:36:53 |
Tristan Sjolie |
description |
Hello,
We are testing network performance with SR-IOV on ubuntu and security onion in Rhev Hypervisor. We are using a SuperMicro box running T-rex traffic generator on CentOS which is connected to a gigamon to push traffic to the Ubuntu/Security Onion VMs.
The ubuntu 16.04 vm is running on a RHVH host node and is running SR-IOV. We have ensured that SR-IOV is enabled in the BIOS. We are finding that packets stop coming into the interface after a few seconds. This is because the interface is going down after a few seconds of receiving traffic. We are running promiscuous mode and it is required for our use case of measuring network traffic.
As a troubleshooting measure, 5.3.0-rc3 was built using the 3.2.3 iavf SR-IOV driver. Note that the i40evf driver was renamed to iavf, which is why the name discrepancy exists. After updating the Kernel we stopped seeing this issue.
Non-working default security onion kernel information
root@ubuntu:~# uname -r
3.10.0-957.21.3.el7.x86_64
root@ubuntu:~# ethtool -i ens7
driver: iavf
version: 3.0.1-k
firmware-version: N/A
expansion-rom-version:
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
Working modified security onion kernel information
root@ubuntu:~# uname -r
5.3.0-rc3-custom
root@ubuntu:~# ethtool -i ens7
driver: i40evf
version: 3.2.3-k
firmware-version: N/A
expansion-rom-version:
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
We were able to perform testing using the 5.3.0 kernel, but long term a better solution is needed as this would require supporting a modified Linux kernel with updates, testing, etc.
From the hypervisor in /var/log/messages:
Aug 27 09:05:00 localhost libvirtd: 2019-08-27 13:05:00.522+0000: 10748: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Aug 27 09:05:07 localhost goferd: [ERROR][worker-0] gofer.messaging.adapter.conn ect:33 - connect: proton+amqps://subscription.rhsm.redhat.com:5647, failed: Conn ection amqps://subscription.rhsm.redhat.com:5647disconnected: Condition('amqp:r esource-limit-exceeded', 'local-idle-timeout expired')
Aug 27 09:05:07 localhost goferd: [INFO][worker-0] gofer.messaging.adapter.conne ct:35 - retry in 106 seconds
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
From the VM, journalctl -xe gives:
-- The start-up result is done.
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: PF reset warning received
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: Scheduling reset task
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:08 ubuntu systemd-logind[1251]: Removed session 16.
-- Subject: Session 16 has been terminated
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat |
Hello,
We are testing network performance with SR-IOV on ubuntu and security onion in Rhev Hypervisor. We are using a SuperMicro box running T-rex traffic generator on CentOS which is connected to a gigamon to push traffic to the Ubuntu/Security Onion VMs.
The ubuntu 16.04 vm is running on a RHVH host node and is running SR-IOV. We have ensured that SR-IOV is enabled in the BIOS. We are finding that packets stop coming into the interface after a few seconds. This is because the interface is going down after a few seconds of receiving traffic. We are running promiscuous mode and it is required for our use case of measuring network traffic.
As a troubleshooting measure, 5.3.0-rc3 was built using the 3.2.3 iavf SR-IOV driver. Note that the i40evf driver was renamed to iavf, which is why the name discrepancy exists. After updating the Kernel we stopped seeing this issue.
Non-working default security onion kernel information
root@ubuntu:~# uname -r
3.10.0-957.21.3.el7.x86_64
root@ubuntu:~# ethtool -i ens7
driver: iavf
version: 3.0.1-k
firmware-version: N/A
expansion-rom-version:
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
Working modified security onion kernel information
root@ubuntu:~# uname -r
5.3.0-rc3-custom
root@ubuntu:~# ethtool -i ens7
driver: i40evf
version: 3.2.3-k
firmware-version: N/A
expansion-rom-version:
bus-info: 0000:00:07.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
We were able to perform testing using the 5.3.0 kernel, but long term a better solution is needed as this would require supporting a modified Linux kernel with updates, testing, etc.
From the hypervisor in /var/log/messages:
Aug 27 09:05:00 localhost libvirtd: 2019-08-27 13:05:00.522+0000: 10748: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Aug 27 09:05:07 localhost goferd: [ERROR][worker-0] gofer.messaging.adapter.conn ect:33 - connect: proton+amqps://subscription.rhsm.redhat.com:5647, failed: Conn ection amqps://subscription.rhsm.redhat.com:5647disconnected: Condition('amqp:r esource-limit-exceeded', 'local-idle-timeout expired')
Aug 27 09:05:07 localhost goferd: [INFO][worker-0] gofer.messaging.adapter.conne ct:35 - retry in 106 seconds
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
Aug 27 09:05:22 localhost kernel: i40e 0000:b5:00.3: Unable to send the message to VF 0 aq_err 12
From the VM, journalctl -xe gives:
-- The start-up result is done.
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: PF reset warning received
Aug 27 14:35:02 ubuntu kernel: i40evf 0000:00:04.0: Scheduling reset task
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:03 ubuntu kernel: i40evf 0000:00:04.0: Hardware reset detected
Aug 27 14:35:08 ubuntu systemd-logind[1251]: Removed session 16.
-- Subject: Session 16 has been terminated
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Aug 28 17:06 seq
crw-rw---- 1 root audio 116, 33 Aug 28 17:06 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.18
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 16.04
HibernationDevice: RESUME=/dev/mapper/ubuntu--vg-swap_1
InstallationDate: Installed on 2019-08-21 (6 days ago)
InstallationMedia: Ubuntu-Server 16.04.6 LTS "Xenial Xerus" - Release amd64 (20190226)
IwConfig: Error: [Errno 2] No such file or directory
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Red Hat RHEV Hypervisor
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=linux
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB: 0 qxldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.15.0-45-generic root=/dev/mapper/hostname--vg-root ro quiet splash isolcpus=2-27 vt.handoff=7
ProcVersionSignature: Ubuntu 4.15.0-45.48~16.04.1-generic 4.15.18
RelatedPackageVersions:
linux-restricted-modules-4.15.0-45-generic N/A
linux-backports-modules-4.15.0-45-generic N/A
linux-firmware 1.157.21
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial
Uname: Linux 4.15.0-45-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.11.0-2.el7
dmi.chassis.type: 1
dmi.chassis.vendor: Red Hat
dmi.chassis.version: RHEL 7.6.0 PC (i440FX + PIIX, 1996)
dmi.modalias: dmi:bvnSeaBIOS:bvr1.11.0-2.el7:bd04/01/2014:svnRedHat:pnRHEVHypervisor:pvr7.6-1.el7ev:cvnRedHat:ct1:cvrRHEL7.6.0PC(i440FX+PIIX,1996):
dmi.product.family: Red Hat Enterprise Linux
dmi.product.name: RHEV Hypervisor
dmi.product.version: 7.6-1.el7ev
dmi.sys.vendor: Red Hat |
|
| 2019-08-28 17:36:53 |
Tristan Sjolie |
attachment added |
|
CRDA.txt https://bugs.launchpad.net/bugs/1841669/+attachment/5285444/+files/CRDA.txt |
|
| 2019-08-28 17:36:55 |
Tristan Sjolie |
attachment added |
|
CurrentDmesg.txt https://bugs.launchpad.net/bugs/1841669/+attachment/5285445/+files/CurrentDmesg.txt |
|
| 2019-08-28 17:36:56 |
Tristan Sjolie |
attachment added |
|
Lspci.txt https://bugs.launchpad.net/bugs/1841669/+attachment/5285446/+files/Lspci.txt |
|
| 2019-08-28 17:36:58 |
Tristan Sjolie |
attachment added |
|
ProcCpuinfo.txt https://bugs.launchpad.net/bugs/1841669/+attachment/5285447/+files/ProcCpuinfo.txt |
|
| 2019-08-28 17:36:59 |
Tristan Sjolie |
attachment added |
|
ProcCpuinfoMinimal.txt https://bugs.launchpad.net/bugs/1841669/+attachment/5285448/+files/ProcCpuinfoMinimal.txt |
|
| 2019-08-28 17:37:01 |
Tristan Sjolie |
attachment added |
|
ProcInterrupts.txt https://bugs.launchpad.net/bugs/1841669/+attachment/5285449/+files/ProcInterrupts.txt |
|
| 2019-08-28 17:37:03 |
Tristan Sjolie |
attachment added |
|
ProcModules.txt https://bugs.launchpad.net/bugs/1841669/+attachment/5285450/+files/ProcModules.txt |
|
| 2019-08-28 17:37:05 |
Tristan Sjolie |
attachment added |
|
UdevDb.txt https://bugs.launchpad.net/bugs/1841669/+attachment/5285451/+files/UdevDb.txt |
|
| 2019-08-28 17:37:08 |
Tristan Sjolie |
attachment added |
|
WifiSyslog.txt https://bugs.launchpad.net/bugs/1841669/+attachment/5285452/+files/WifiSyslog.txt |
|
| 2019-08-28 17:37:50 |
Tristan Sjolie |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
