bond interface showing dropped packets

[Albert Chin]

# ifconfig bond0
bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea
          inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
          RX packets:17 errors:0 dropped:16 overruns:0 frame:0
          TX packets:25 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2044 (2.0 KB) TX bytes:2912 (2.9 KB)

# ifconfig eth2
eth2 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea
          UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
          RX packets:36 errors:0 dropped:0 overruns:0 frame:0
          TX packets:50 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4464 (4.4 KB) TX bytes:5958 (5.9 KB)
          Memory:b2420000-b2440000

# ifconfig eth3
root@trunks:/home/china# ifconfig eth3
eth3 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea
          UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
          RX packets:52 errors:0 dropped:1 overruns:0 frame:0
          TX packets:54 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:6012 (6.0 KB) TX bytes:6696 (6.6 KB)
          Memory:b2400000-b2420000

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 0

802.3ad info
LACP rate: fast
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 17
        Partner Key: 24
        Partner Mac Address: 00:04:96:18:54:d5

Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:1b:21:b7:21:ea
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:1b:21:b7:21:eb
Aggregator ID: 1
Slave queue ID: 0

...

# ethtool -k bond0
Offload parameters for bond0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off

# ethtool -i bond0
driver: bonding
version: 3.7.1
firmware-version: 2
bus-info:
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
---
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.21.
Architecture: i386
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Audigy2'/'SB Audigy 2 ZS [SB0350] (rev.4, serial:0x20021102) at 0xb800, irq 21'
   Mixer name : 'SigmaTel STAC9750,51'
   Components : 'AC97a:83847650'
   Controls : 211
   Simple ctrls : 46
DistroRelease: Ubuntu 10.04
HibernationDevice: RESUME=UUID=72a255fb-d7ef-43e8-b41d-a64dfa2e0d93
InstallationMedia: Ubuntu 10.04.2 LTS "Lucid Lynx" - Release i386 (20110211.3)
NonfreeKernelModules: nvidia
Package: linux (not installed)
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.32-28-generic root=UUID=86b85e0a-4cfc-452e-8944-add406605f60 ro quiet splash
ProcEnviron:
 LANG=en_US.utf8
 SHELL=/bin/zsh
ProcVersionSignature: Ubuntu 2.6.32-28.55-generic 2.6.32.27+drm33.12
Regression: No
RelatedPackageVersions: linux-firmware 1.34.7
Reproducible: Yes
RfKill:

Tags: lucid networking needs-upstream-testing
Uname: Linux 2.6.32-28-generic i686
UserGroups: adm admin cdrom dialout libvirtd lpadmin plugdev sambashare
dmi.bios.date: 02/07/2005
dmi.bios.vendor: Intel Corp.
dmi.bios.version: WP87510A.86B.0059.P18.0502071117
dmi.board.name: S875WP1
dmi.board.vendor: Intel Corporation
dmi.board.version: AAC28366-305
dmi.chassis.type: 2
dmi.modalias: dmi:bvnIntelCorp.:bvrWP87510A.86B.0059.P18.0502071117:bd02/07/2005:svn:pn:pvr:rvnIntelCorporation:rnS875WP1:rvrAAC28366-305:cvn:ct2:cvr:
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 2011-11-17 06:29 seq
 crw-rw---- 1 root audio 116, 33 2011-11-17 06:29 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 1.23-0ubuntu3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
DistroRelease: Ubuntu 11.10
HibernationDevice: RESUME=UUID=372c7fdc-eb4c-42e0-8cf9-de4b6434a6eb
InstallationMedia: Ubuntu-Server 11.10 "Oneiric Ocelot" - Release amd64 (20111011)
MachineType: Intel Corporation S5520UR
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/usr/bin/zsh
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.0.0-12-server root=/dev/mapper/hostname-lv_root ro crashkernel=384M-2G:64M,2G-:128M console=tty0 console=ttyS0,9600n8
ProcVersionSignature: Ubuntu 3.0.0-12.20-server 3.0.4
RelatedPackageVersions:
 linux-restricted-modules-3.0.0-12-server N/A
 linux-backports-modules-3.0.0-12-server N/A
 linux-firmware 1.60
RfKill: Error: [Errno 2] No such file or directory
Tags: oneiric
Uname: Linux 3.0.0-12-server x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm admin cdrom dialout libvirtd lpadmin plugdev sambashare
dmi.bios.date: 08/23/2011
dmi.bios.vendor: Intel Corp.
dmi.bios.version: S5500.86B.01.00.0059.082320111421
dmi.board.asset.tag: ....................
dmi.board.name: S5520UR
dmi.board.vendor: Intel Corporation
dmi.board.version: E22554-752
dmi.chassis.asset.tag: ....................
dmi.chassis.type: 23
dmi.chassis.vendor: ...............................
dmi.chassis.version: ..................
dmi.modalias: dmi:bvnIntelCorp.:bvrS5500.86B.01.00.0059.082320111421:bd08/23/2011:svnIntelCorporation:pnS5520UR:pvr....................:rvnIntelCorporation:rnS5520UR:rvrE22554-752:cvn...............................:ct23:cvr..................:
dmi.product.name: S5520UR
dmi.product.version: ....................
dmi.sys.vendor: Intel Corporation

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 890475

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Albert Chin (bugs-ubuntu-vendor)]

Tried linux-image-3.0.0-13-server and get the same result. Using an older kernel, linux-image-2.6.39-3-server, shows no dropped packets. On the same hardware, with the interfaces configured the same way, a RHEL6 2.6.32-131.17.1.el6.x86_64 kernel shows no dropped packets.

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

linux-image-3.1.0-2-server gives the same result.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . If possible, please test the latest v3.2-rcN kernel (Not a kernel in the daily directory). Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag(Only that one tag, please leave the others). This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed by the mainline kernel, please add the following tag 'kernel-fixed-upstream-KERNEL-VERSION'. For example, if kernel version 3.2-rc1 fixed and issue, the tag would be: 'kernel-fixed-upstream-v3.2-rc1'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Thanks in advance.

[Albert Chin (bugs-ubuntu-vendor)]

Ok, downloading linux-image-3.2.0-030200rc2-generic_3.2.0-030200rc2.201111151435_amd64.deb. Note I have tried booting from linux-image-3.0.9-030009-generic_3.0.9-030009.201111111635_amd64.deb and linux-headers-3.1.1-030101-generic_3.1.1-030101.201111111651_amd64.deb to do further testing. Both had errors during the boot sequence so I'm not really optimistic the 3.2-rc2 will boot but I'll try.

[Albert Chin (bugs-ubuntu-vendor)]

Ok, 3.2-rc2 isn't booting. I see lots of these errors:
[ 29.078074] scsi: killing requests for dead queue
[ 29.078076] scsi: killing requests for dead queue
[ 29.078193] scsi: killing requests for dead queue
[ 29.078195] scsi: killing requests for dead queue

Seems to be http://www.spinics.net/lists/linux-scsi/msg55348.html

The boot process finally dies with the following:

[ 88.715404] Copyright (c) 2007-2011 Intel Corporation.
[ 88.715781] sd 4:2:1:0: [sdb] Attached SCSI disk
[ 88.715786] sd 4:2:0:0: [sda] Attached SCSI disk
[ 88.805205] INFO: rcu_sched detected stalls on CPUs/tasks: { 8} (detected by 3, t=15002 jiffies)
[ 88.805210] Pid: 0, comm: kworker/0:1 Not tainted 3.2.0-030200rc2-generic #201111151435
[ 88.805211] Call Trace:
[ 88.805213] <IRQ> [<ffffffff810df45f>] print_other_cpu_stall+0x15f/0x1b0
[ 88.805221] [<ffffffff8109a220>] ? tick_nohz_handler+0xf0/0xf0
[ 88.805223] [<ffffffff810df528>] __rcu_pending+0x78/0x290
[ 88.805225] [<ffffffff810df7ba>] rcu_check_callbacks+0x7a/0x1c0
[ 88.805229] [<ffffffff81074db8>] update_process_times+0x48/0x90
[ 88.805230] [<ffffffff8109a286>] tick_sched_timer+0x66/0xc0
[ 88.805234] [<ffffffff8108cb54>] __run_hrtimer+0x84/0x1e0
[ 88.805237] [<ffffffff8101a529>] ? read_tsc+0x9/0x20
[ 88.805239] [<ffffffff8108cf46>] hrtimer_interrupt+0xd6/0x210
[ 88.805242] [<ffffffff81619809>] smp_apic_timer_interrupt+0x69/0x99
[ 88.805245] [<ffffffff816176de>] apic_timer_interrupt+0x6e/0x80
[ 88.805246] <EOI> [<ffffffff81371d70>] ? intel_idle+0xe0/0x150
[ 88.805249] [<ffffffff81371d56>] ? intel_idle+0xc6/0x150
[ 88.805252] [<ffffffff814de2b4>] cpuidle_idle_call+0xc4/0x190
[ 88.805254] [<ffffffff8101309b>] cpu_idle+0xcb/0x120
[ 88.805257] [<ffffffff816078eb>] start_secondary+0xde/0xe0
...
Gave up waiting for root device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
   - Check root= (did the system wait for the right device?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/mapper/trunks-lv_root does not exist. Dropping to a shell!

[Albert Chin (bugs-ubuntu-vendor)]

Well, this is weird. At the (initramfs) prompt, I entered "exit" and the system continued to boot. So, I brought up bond0, eth2, and eth3. I continue to see dropped packets:

# uname -a
Linux trunks 3.2.0-030200rc2-generic #201111151435 SMP Tue Nov 15 19:36:01 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
# ifconfig bond0
bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea
          inet addr:10.191.62.2 Bcast:10.191.62.255 Mask:255.255.255.0
          inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
          RX packets:200 errors:0 dropped:165 overruns:0 frame:0
          TX packets:213 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:22758 (22.7 KB) TX bytes:25950 (25.9 KB)

[Albert Chin (bugs-ubuntu-vendor)]

This system has four interfaces. The bonded interfaces comprise (2) Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01). The two interfaces that come with the system are Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02). I tried to bond the two 82575EB interfaces with the same result, dropped packets on 3.0.0-12-server:
  # ifconfig bond0
  bond0 Link encap:Ethernet HWaddr 00:1e:67:1c:fd:d4
            inet6 addr: fe80::21e:67ff:fe1c:fdd4/64 Scope:Link
            UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
            RX packets:457 errors:0 dropped:359 overruns:0 frame:0
            TX packets:444 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0
            RX bytes:50891 (50.8 KB) TX bytes:54504 (54.5 KB)

[Joseph Salisbury (jsalisbury)]

Hi Albert,

The apport data you attached shows that you are running kernel version: Ubuntu 2.6.32-28.55-generic 2.6.32.27+drm33.12

Were you having the dropped packets in that version of the kernel, or just the Oneiric kernel? Can you re-run apport-collect while booted into the Oneiric kernel:

apport-collect 890475

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

apport information

[Albert Chin (bugs-ubuntu-vendor)]

I installed Fedora 16 on this system with a 3.1.1-1.fc16.x86_64 kernel and see dropped packets there as well:
  $ ifconfig bond0
  bond0 Link encap:Ethernet HWaddr 00:1B:21:D3:F6:0A
            inet addr:10.191.62.2 Bcast:10.191.62.255 Mask:255.255.255.0
            inet6 addr: fe80::21b:21ff:fed3:f60a/64 Scope:Link
            UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
            RX packets:175 errors:0 dropped:34 overruns:0 frame:0
            TX packets:157 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0
            RX bytes:13089 (12.7 KiB) TX bytes:19016 (18.5 KiB)

The rate of dropped packets was the same as the 3.0.0-12-server kernel after I set lacp_rate=fast.

[Albert Chin (bugs-ubuntu-vendor)]

http://marc.info/?t=132185330900002&r=1&w=2

[penalvch (penalvch)]

Albert Chin, this bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? If so, could you please test for this with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/ .

If it remains an issue, could you please run the following command in the development release from a Terminal (Applications->Accessories->Terminal), as it will automatically gather and attach updated debug information to this report:

apport-collect -p linux <replace-with-bug-number>

Also, could you please test the latest upstream kernel available? It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please do not test the kernel in the daily folder, but the one all the way at the bottom. Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. As well, please comment on which kernel version specifically you tested.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream', and comment as to why specifically you were unable to test it.

Please let us know your results. Thanks in advance.

Helpful Bug Reporting Links:
https://help.ubuntu.com/community/ReportingBugs#Bug_Reporting_Etiquette
https://help.ubuntu.com/community/ReportingBugs#A3._Make_sure_the_bug_hasn.27t_already_been_reported
https://help.ubuntu.com/community/ReportingBugs#Adding_Apport_Debug_Information_to_an_Existing_Launchpad_Bug
https://help.ubuntu.com/community/ReportingBugs#Adding_Additional_Attachments_to_an_Existing_Launchpad_Bug

[Daniel Kendrick (p-danielk)]

This issue is also happening on 12.04 LTS

          inet addr:192.168.17.22 Bcast:192.168.17.255 Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:9612202 errors:0 dropped:1422061703 overruns:0 frame:0
          TX packets:383 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:10000
          RX bytes:13947785127 (13.9 GB) TX bytes:45380 (45.3 KB)
          Interrupt:48 Memory:d8000000-d8012800

3.2.0-29-generic #46-Ubuntu SMP Fri Jul 27 17:03:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

This isnt happening on all my servers, but a good portion. I have 20 machines and 5 are showing these symptoms.

01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
03:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

When bonded packets drop, when not bonded packets do not drop.

Ive ran a series of tests against the interfaces and network.

[Tim Stewart (tim-j-stewart)]

Could this bug be related to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/1041070

If so, it might just be a display error according to: https://bugs.launchpad.net/ubuntu/+source/bridge-utils/+bug/1041070/comments/8

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Arie Skliarouk (skliarie)]

@daniel-kendrick:
Could it be switch problem? We use two SGE2010 48-Port 10/100/1000 Ethernet Switches.

@Tim Stewart
No, it is not display error - we really have dropped packets on the network.

[penalvch (penalvch)]

Daniel Kendrick / Tim Stewart / Arie Skliarouk, if you have a bug in Ubuntu, the Ubuntu Kernel team, Ubuntu Bug Control team, and Ubuntu Bug Squad would like you to please file a new report by executing the following in a terminal:
ubuntu-bug linux

For more on this, please see the Ubuntu Kernel team article:
https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Filing_Kernel_Bug_reports

the Ubuntu Bug Control team and Ubuntu Bug Squad team article:
https://wiki.ubuntu.com/Bugs/BestPractices#X.2BAC8-Reporting.Focus_on_One_Issue

and Ubuntu Community article:
https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette

When opening up the new report, please feel free to subscribe me to it.

Please note, not filing a new report would delay your problem being addressed as quickly as possible.

Thank you for your understanding.