| 2019-06-26 11:11:57 |
Przemyslaw Hausman |
bug |
|
|
added bug |
| 2019-06-26 11:19:47 |
Przemyslaw Hausman |
bug |
|
|
added subscriber Canonical Field Critical |
| 2019-06-26 11:22:57 |
Dean Henrichsmeyer |
removed subscriber Canonical Field Critical |
|
|
|
| 2019-06-26 11:30:05 |
Ubuntu Kernel Bot |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2019-06-26 11:30:07 |
Ubuntu Kernel Bot |
tags |
|
bionic |
|
| 2019-06-26 11:33:46 |
Dominique Poulain |
bug |
|
|
added subscriber Dominique Poulain |
| 2019-06-26 11:43:24 |
Przemyslaw Hausman |
tags |
bionic |
apport-collected bionic uec-images |
|
| 2019-06-26 11:43:24 |
Przemyslaw Hausman |
description |
We are losing port channel aggregation on reboot.
After the reboot, /var/log/syslog contains the entries:
[ 250.790758] bond2: An illegal loopback occurred on adapter (enp24s0f1np1)
Check the configuration to verify that all adapters are connected to 802.3ad compliant switch ports
[ 282.029426] bond2: An illegal loopback occurred on adapter (enp24s0f1np1)
Check the configuration to verify that all adapters are connected to 802.3ad compliant switch ports
Aggregator IDs of the slave interfaces are different:
ubuntu@node-6:~$ cat /proc/net/bonding/bond2
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
Slave Interface: enp24s0f1np1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: b0:26:28:48:9f:51
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
Slave Interface: enp24s0f0np0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: b0:26:28:48:9f:50
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
The mismatch in "Aggregator ID" on the port is a symptom of the issue. If we do 'ip link set dev bond2 down' and 'ip link set dev bond2 up', the port with the mismatched ID appears to renegotiate with the port-channel and becomes aggregated.
The other way to workaround this issue is to put bond ports down and bring up port enp24s0f0np0 first and port enp24s0f1np1 second.
When I change the order of bringing the ports up (first enp24s0f1np1, and second enp24s0f0np0), the issue is still there.
When the issue occurs, a port on the switch, corresponding to interface enp24s0f0np0 is in Suspended state. After applying the workaround the port is no longer in Suspended state and Aggregator IDs in /proc/net/bonding/bond2 are equal.
I installed 5.0.0 kernel, the issue is still there.
Operating System:
Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-52-generic x86_64)
ubuntu@node-6:~$ uname -a
Linux node-6 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@node-6:~$ sudo lspci -vnvn
https://pastebin.ubuntu.com/p/Dy2CKDbySC/
Hardware: Dell PowerEdge R740xd
BIOS version: 2.1.7
sosreport: https://drive.google.com/open?id=1-eN7cZJIeu-AQBEU7Gw8a_AJTuq0AOZO
ubuntu@node-6:~$ lspci | grep Ethernet | grep 10G
https://pastebin.ubuntu.com/p/sqCx79vZWM/
ubuntu@node-6:~$ lspci -n | grep 18:00
18:00.0 0200: 14e4:16d8 (rev 01)
18:00.1 0200: 14e4:16d8 (rev 01)
ubuntu@node-6:~$ modinfo bnx2x
https://pastebin.ubuntu.com/p/pkmzsFjK8M/
ubuntu@node-6:~$ ip -o l
https://pastebin.ubuntu.com/p/QpW7TjnT2v/
ubuntu@node-6:~$ ip -o a
https://pastebin.ubuntu.com/p/MczKtrnmDR/
ubuntu@node-6:~$ cat /etc/netplan/98-juju.yaml
https://pastebin.ubuntu.com/p/9cZpPc7C6P/
ubuntu@node-6:~$ sudo lshw -c network
https://pastebin.ubuntu.com/p/gmfgZptzDT/ |
We are losing port channel aggregation on reboot.
After the reboot, /var/log/syslog contains the entries:
[ 250.790758] bond2: An illegal loopback occurred on adapter (enp24s0f1np1)
Check the configuration to verify that all adapters are connected to 802.3ad compliant switch ports
[ 282.029426] bond2: An illegal loopback occurred on adapter (enp24s0f1np1)
Check the configuration to verify that all adapters are connected to 802.3ad compliant switch ports
Aggregator IDs of the slave interfaces are different:
ubuntu@node-6:~$ cat /proc/net/bonding/bond2
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
Slave Interface: enp24s0f1np1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: b0:26:28:48:9f:51
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
Slave Interface: enp24s0f0np0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: b0:26:28:48:9f:50
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: churned
Partner Churn State: churned
Actor Churned Count: 1
Partner Churned Count: 1
The mismatch in "Aggregator ID" on the port is a symptom of the issue. If we do 'ip link set dev bond2 down' and 'ip link set dev bond2 up', the port with the mismatched ID appears to renegotiate with the port-channel and becomes aggregated.
The other way to workaround this issue is to put bond ports down and bring up port enp24s0f0np0 first and port enp24s0f1np1 second.
When I change the order of bringing the ports up (first enp24s0f1np1, and second enp24s0f0np0), the issue is still there.
When the issue occurs, a port on the switch, corresponding to interface enp24s0f0np0 is in Suspended state. After applying the workaround the port is no longer in Suspended state and Aggregator IDs in /proc/net/bonding/bond2 are equal.
I installed 5.0.0 kernel, the issue is still there.
Operating System:
Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-52-generic x86_64)
ubuntu@node-6:~$ uname -a
Linux node-6 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@node-6:~$ sudo lspci -vnvn
https://pastebin.ubuntu.com/p/Dy2CKDbySC/
Hardware: Dell PowerEdge R740xd
BIOS version: 2.1.7
sosreport: https://drive.google.com/open?id=1-eN7cZJIeu-AQBEU7Gw8a_AJTuq0AOZO
ubuntu@node-6:~$ lspci | grep Ethernet | grep 10G
https://pastebin.ubuntu.com/p/sqCx79vZWM/
ubuntu@node-6:~$ lspci -n | grep 18:00
18:00.0 0200: 14e4:16d8 (rev 01)
18:00.1 0200: 14e4:16d8 (rev 01)
ubuntu@node-6:~$ modinfo bnx2x
https://pastebin.ubuntu.com/p/pkmzsFjK8M/
ubuntu@node-6:~$ ip -o l
https://pastebin.ubuntu.com/p/QpW7TjnT2v/
ubuntu@node-6:~$ ip -o a
https://pastebin.ubuntu.com/p/MczKtrnmDR/
ubuntu@node-6:~$ cat /etc/netplan/98-juju.yaml
https://pastebin.ubuntu.com/p/9cZpPc7C6P/
ubuntu@node-6:~$ sudo lshw -c network
https://pastebin.ubuntu.com/p/gmfgZptzDT/
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Jun 26 10:21 seq
crw-rw---- 1 root audio 116, 33 Jun 26 10:21 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
DistroRelease: Ubuntu 18.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 1604:10c0 Tascam
Bus 001 Device 003: ID 1604:10c0 Tascam
Bus 001 Device 002: ID 1604:10c0 Tascam
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Dell Inc. PowerEdge R740xd
Package: linux (not installed)
PciMultimedia:
ProcFB: 0 mgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-52-generic root=UUID=9b1eae60-0941-4638-a22a-98a719104259 ro console=tty0 console=ttyS0,115200 console=ttyS1,115200 raid=noautodetect intel_iommu=on iommu=pt pti=off
ProcVersionSignature: Ubuntu 4.15.0-52.56-generic 4.15.18
RelatedPackageVersions:
linux-restricted-modules-4.15.0-52-generic N/A
linux-backports-modules-4.15.0-52-generic N/A
linux-firmware 1.173.6
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic uec-images
Uname: Linux 4.15.0-52-generic x86_64
UnreportableReason: This report is about a package that is not installed.
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy libvirt netdev plugdev sudo video
_MarkForUpload: False
dmi.bios.date: 04/03/2019
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 2.1.7
dmi.board.name: 0JMK61
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr2.1.7:bd04/03/2019:svnDellInc.:pnPowerEdgeR740xd:pvr:rvnDellInc.:rn0JMK61:rvrA00:cvnDellInc.:ct23:cvr:
dmi.product.family: PowerEdge
dmi.product.name: PowerEdge R740xd
dmi.sys.vendor: Dell Inc. |
|
| 2019-06-26 11:43:25 |
Przemyslaw Hausman |
attachment added |
|
CRDA.txt https://bugs.launchpad.net/bugs/1834322/+attachment/5273475/+files/CRDA.txt |
|
| 2019-06-26 11:43:26 |
Przemyslaw Hausman |
attachment added |
|
CurrentDmesg.txt https://bugs.launchpad.net/bugs/1834322/+attachment/5273476/+files/CurrentDmesg.txt |
|
| 2019-06-26 11:43:27 |
Przemyslaw Hausman |
attachment added |
|
Lspci.txt https://bugs.launchpad.net/bugs/1834322/+attachment/5273477/+files/Lspci.txt |
|
| 2019-06-26 11:43:28 |
Przemyslaw Hausman |
attachment added |
|
ProcCpuinfoMinimal.txt https://bugs.launchpad.net/bugs/1834322/+attachment/5273478/+files/ProcCpuinfoMinimal.txt |
|
| 2019-06-26 11:43:29 |
Przemyslaw Hausman |
attachment added |
|
ProcEnviron.txt https://bugs.launchpad.net/bugs/1834322/+attachment/5273479/+files/ProcEnviron.txt |
|
| 2019-06-26 11:43:30 |
Przemyslaw Hausman |
attachment added |
|
ProcInterrupts.txt https://bugs.launchpad.net/bugs/1834322/+attachment/5273480/+files/ProcInterrupts.txt |
|
| 2019-06-26 11:43:31 |
Przemyslaw Hausman |
attachment added |
|
ProcModules.txt https://bugs.launchpad.net/bugs/1834322/+attachment/5273481/+files/ProcModules.txt |
|
| 2019-06-26 11:43:32 |
Przemyslaw Hausman |
attachment added |
|
UdevDb.txt https://bugs.launchpad.net/bugs/1834322/+attachment/5273482/+files/UdevDb.txt |
|
| 2019-06-26 11:43:33 |
Przemyslaw Hausman |
attachment added |
|
WifiSyslog.txt https://bugs.launchpad.net/bugs/1834322/+attachment/5273483/+files/WifiSyslog.txt |
|
| 2019-06-26 14:24:54 |
Jay Vosburgh |
bug |
|
|
added subscriber Jay Vosburgh |
| 2019-07-24 18:52:28 |
Brad Figg |
tags |
apport-collected bionic uec-images |
apport-collected bionic ubuntu-certified uec-images |
|
| 2019-09-23 04:19:19 |
Launchpad Janitor |
linux (Ubuntu): status |
Incomplete |
Expired |
|
| 2019-11-21 18:53:52 |
Nivedita Singhvi |
linux (Ubuntu): status |
Expired |
In Progress |
|
| 2019-11-21 18:54:36 |
Nivedita Singhvi |
tags |
apport-collected bionic ubuntu-certified uec-images |
apport-collected bionic sts ubuntu-certified uec-images |
|
| 2019-11-21 18:55:25 |
Nivedita Singhvi |
nominated for series |
|
Ubuntu Disco |
|
| 2019-11-21 18:55:25 |
Nivedita Singhvi |
bug task added |
|
linux (Ubuntu Disco) |
|
| 2019-11-21 18:55:25 |
Nivedita Singhvi |
nominated for series |
|
Ubuntu Bionic |
|
| 2019-11-21 18:55:25 |
Nivedita Singhvi |
bug task added |
|
linux (Ubuntu Bionic) |
|
| 2019-11-21 18:55:25 |
Nivedita Singhvi |
nominated for series |
|
Ubuntu Focal |
|
| 2019-11-21 18:55:25 |
Nivedita Singhvi |
bug task added |
|
linux (Ubuntu Focal) |
|
| 2019-11-21 18:55:25 |
Nivedita Singhvi |
nominated for series |
|
Ubuntu Eoan |
|
| 2019-11-21 18:55:25 |
Nivedita Singhvi |
bug task added |
|
linux (Ubuntu Eoan) |
|
| 2019-12-30 15:28:35 |
Nivedita Singhvi |
linux (Ubuntu Focal): importance |
Undecided |
High |
|
| 2019-12-30 15:28:39 |
Nivedita Singhvi |
linux (Ubuntu Eoan): importance |
Undecided |
High |
|
| 2019-12-30 15:28:43 |
Nivedita Singhvi |
linux (Ubuntu Disco): importance |
Undecided |
High |
|
| 2019-12-30 15:28:47 |
Nivedita Singhvi |
linux (Ubuntu Bionic): importance |
Undecided |
High |
|
| 2019-12-30 15:28:57 |
Nivedita Singhvi |
linux (Ubuntu Bionic): status |
New |
Fix Committed |
|
| 2019-12-30 15:29:04 |
Nivedita Singhvi |
linux (Ubuntu Disco): status |
New |
Fix Committed |
|
| 2019-12-30 15:29:09 |
Nivedita Singhvi |
linux (Ubuntu Eoan): status |
New |
Fix Committed |
|
| 2020-05-28 14:47:10 |
Nivedita Singhvi |
bug |
|
|
added subscriber Nivedita Singhvi |
| 2020-05-28 14:48:38 |
Nivedita Singhvi |
linux (Ubuntu Focal): status |
In Progress |
Fix Released |
|
| 2020-05-28 14:52:16 |
Nivedita Singhvi |
linux (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2020-05-28 14:52:21 |
Nivedita Singhvi |
linux (Ubuntu Disco): status |
Fix Committed |
Fix Released |
|
| 2020-05-28 14:52:26 |
Nivedita Singhvi |
linux (Ubuntu Eoan): status |
Fix Committed |
Fix Released |
|
| 2020-07-14 15:06:14 |
Jay Vosburgh |
linux (Ubuntu): status |
In Progress |
Fix Released |
|
