8086:10c9 bridged network interface dropping RX packets

[Tamas Papp]

This bug was created, because it was requested in the bug #787055.

br-eth0 Link encap:Ethernet HWaddr 00:25:90:58:37:7c
          inet addr:10.0.0.126 Bcast:10.0.1.255 Mask:255.255.254.0
          inet6 addr: fe80::225:90ff:fe58:377c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:397947 errors:0 dropped:78036 overruns:0 frame:0
          TX packets:153115 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:185053012 (185.0 MB) TX bytes:27029969 (27.0 MB)

# brctl show
bridge name bridge id STP enabled interfaces
br-eth0 8000.00259058377c yes eth0
                                                                       vethgKuosW
                                                                       vethmrHuzA
                                                                      vethoKe7HJ
                                                                      vethuIAK0P
                                                                      vnet0
                                                                      vnet1
virbr0 8000.000000000000 yes

So STP is enabled, but it doesn't count, the same problem if it's off.
Dropped packets number increasing continuosly.

That same happens if the system is lucid and backported kernel is running (maverick or above).

On this LAN there is multiple system with similar configuration and all produce this error.
On another network it works with no problem, but there is no more similar setup.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-23-generic 3.2.0-23.36
ProcVersionSignature: Ubuntu 3.2.0-23.36-generic 3.2.14
Uname: Linux 3.2.0-23-generic x86_64
AlsaDevices:
 total 0
 crw-rw---T 1 root audio 116, 1 Apr 19 23:46 seq
 crw-rw---T 1 root audio 116, 33 Apr 19 23:46 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Date: Fri Apr 20 09:33:34 2012
HibernationDevice: RESUME=UUID=4044e27d-f347-404b-9494-e863083578bc
IwConfig: Error: [Errno 2] No such file or directory
MachineType: Supermicro H8QG6
PciMultimedia:

ProcEnviron:
 TERM=xterm
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB:

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-23-generic root=/dev/mapper/vg0-root ro
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-23-generic N/A
 linux-backports-modules-3.2.0-23-generic N/A
 linux-firmware 1.79
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 09/26/2011
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 2.00
dmi.board.asset.tag: 1234567890
dmi.board.name: H8QG6
dmi.board.vendor: Supermicro
dmi.board.version: 1234567890
dmi.chassis.asset.tag: 1234567890
dmi.chassis.type: 17
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 1234567890
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.00:bd09/26/2011:svnSupermicro:pnH8QG6:pvr1234567890:rvnSupermicro:rnH8QG6:rvr1234567890:cvnSupermicro:ct17:cvr1234567890:
dmi.product.name: H8QG6
dmi.product.version: 1234567890
dmi.sys.vendor: Supermicro

[Tamas Papp (tompos)]

This is a VM host for libvirt (kvm) machines and LXC containers.

I see dropped packets on the VMs too.

Example #1:

tompos@lxc01:~$ uptime
 10:47:46 up 10:44, 2 users, load average: 0.16, 0.42, 0.48
tompos@lxc01:~$ uname -a
Linux lxc01 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

LXC HOST:
br-eth0 Link encap:Ethernet HWaddr 00:25:b3:82:fd:d8
          inet addr:10.0.0.54 Bcast:10.0.1.255 Mask:255.255.254.0
          inet6 addr: fe80::225:b3ff:fe82:fdd8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:400371 errors:0 dropped:84828 overruns:0 frame:0
          TX packets:94804 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:167501161 (167.5 MB) TX bytes:18564022 (18.5 MB)

Debian container:
eth0 Link encap:Ethernet HWaddr 52:54:00:00:00:00
          inet addr:10.0.0.55 Bcast:10.0.1.255 Mask:255.255.254.0
          inet6 addr: fe80::5054:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:17213802 errors:0 dropped:84822 overruns:0 frame:0
          TX packets:40391225 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5360613932 (4.9 GiB) TX bytes:12306335047 (11.4 GiB)

Debian container:
eth0 Link encap:Ethernet HWaddr 36:6d:6a:e4:86:ea
          inet addr:10.0.0.115 Bcast:10.0.1.255 Mask:255.255.254.0
          inet6 addr: fe80::346d:6aff:fee4:86ea/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:324344 errors:0 dropped:84822 overruns:0 frame:0
          TX packets:1030 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:143732907 (137.0 MiB) TX bytes:105055 (102.5 KiB)

Example #2:

LXC+libvirt KVM host:
tompos@virt101:~$ uname -a
Linux virt101 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
tompos@virt101:~$ uptime
 10:51:28 up 11:05, 1 user, load average: 0.26, 0.22, 0.23

br-eth0 Link encap:Ethernet HWaddr 00:25:90:58:37:7c
          inet addr:10.0.0.126 Bcast:10.0.1.255 Mask:255.255.254.0
          inet6 addr: fe80::225:90ff:fe58:377c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:502125 errors:0 dropped:87609 overruns:0 frame:0
          TX packets:170265 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:208178343 (208.1 MB) TX bytes:30995052 (30.9 MB)

Debian container:

eth0 Link encap:Ethernet HWaddr ae:7f:44:57:9f:0f
          inet addr:10.0.0.124 Bcast:10.0.1.255 Mask:255.255.254.0
          inet6 addr: fe80::ac7f:44ff:fe57:9f0f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:5638101 errors:0 dropped:87602 overruns:0 frame:0
          TX packets:7535345 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1896436495 (1.7 GiB) TX bytes:1321880127 (1.2 GiB)

KVM Lucid VM:
dropped packets number does not increasing co...

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.4kernel[1] (Not a kernel in the daily directory). Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag(Only that one tag, please leave the other tags). This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.4-rc3-precise/

[Tamas Papp (tompos)]

No joy:(

[penalvch (penalvch)]

Marking Triaged as mainline tested.

[penalvch (penalvch)]

Tamas Papp, please answer the following questions:
+ Regarding https://bugs.launchpad.net/ubuntu/+source/linux/+bug/986043/comments/4 which kernel specifically did you test?
+ Did you test the mainline kernel in the host only, guest only, and both?
+ Could you elaborate more on the LAN setup?
+ It is mentioned in the Bug Description that:
"On another network it works with no problem, but there is no more similar setup." Could you please describe the other network in full detail?

[Hjálmar Snorrason (hjallisnorra)]

I can confirm that this bug is still there..

uname -a 3.2.0-24-generic #37-Ubuntu SMP Wed Apr 25 08:43:22 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

dropping rx packages in bonded network using 4 network cards....

and i have a simpler interface setup, posted a question :

https://answers.launchpad.net/ubuntu/+source/gnome-nettool/+question/195870

was assigned to gnome-nettools, dont think it belongs there.

And I think this bug is more important than Medium...

[penalvch (penalvch)]

Hjálmar Snorrason, please execute the following via the Terminal and feel free to subscribe me to it:
ubuntu-bug linux

Thanks!

[Hjálmar Snorrason (hjallisnorra)]

Christopher M. Penalver,

tried to do ubuntu-bug linux, but the mashine froze.

dont know how to continue...
regards
Hjalmar

[penalvch (penalvch)]

Hjalmar Snorrason, you are welcome to file a bug using https://bugs.launchpad.net/ubuntu/+source/linux/+filebug and feel free to subscribe me to it. Thanks!

[Tamas Papp (tompos)]

Sorry for the late answer.

> + Regarding https://bugs.launchpad.net/ubuntu/+source/linux/+bug/986043/comments/4 which kernel specifically did you test?

I don't get it. What did I test? I did, what you asked for.

> Did you test the mainline kernel in the host only, guest only, and both?

The same, I don't get it. This is an LXC host, there is kernel only on the container.

> Could you elaborate more on the LAN setup?

Whaat do you need exactly? There is nothing unusual. This is a medium sized network with physical and virtual (ESX, KVM, LXC) machines. There are servers and desktops too with windows, linux, and OSX.

> It is mentioned in the Bug Description that:
"On another network it works with no problem, but there is no more similar setup." Could you please describe the other network in full detail?

I have access for two networks with similar setup.
One is in a hosting center with some servers (ESXs, physical and the LXC host).
The other one is a small office, with some servers (physical and one LXC hsot) and desktops. Both of them seem to be OK.

Please write me, what you need.

[hosh (hosh-n)]

Hallo

we have the same problem on a KVM host (the VMs have also drpped Packets)

br0 Link encap:Ethernet HWaddr 00:19:99:ab:33:77
          inet addr:10.25.133.74 Bcast:10.25.133.255 Mask:255.255.255.0
          inet6 addr: fe80::219:99ff:feab:3377/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:5348308 errors:0 dropped:2330840 overruns:0 frame:0
          TX packets:2959092 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:493375573 (493.3 MB) TX bytes:1714425852 (1.7 GB)

i add the bug report of this machine

we are currently testing some senarios (vm's migrated on physical machines etc)

we can see that without a bridgeinterface everything is fine! if we got more informations i will post them.

[hosh (hosh-n)]

We configured the bridge interface on the test machine without KVM and the dropping start

br0 Link encap:Ethernet HWaddr 00:26:2d:0a:7e:f6
          inet addr:10.25.133.78 Bcast:10.25.133.255 Mask:255.255.255.0
          inet6 addr: fe80::226:2dff:fe0a:7ef6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:84734 errors:0 dropped:8264 overruns:0 frame:0
          TX packets:78633 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:21683346 (21.6 MB) TX bytes:25454981 (25.4 MB)

[penalvch (penalvch)]

hosh, please do not post attachments to this report. For more on this please see https://help.ubuntu.com/community/ReportingBugs#Adding_Apport_Debug_Information_to_an_Existing_Launchpad_Bug .

If you are having a problem in Ubuntu, please execute the following via the Terminal and feel free to subscribe me to it:
ubuntu-bug linux
Thanks!

[hosh (hosh-n)]

i am sorry
i keep that in mind!

we tested a similar configuration and openvswitch - same behavior

[Narayan Desai (narayan-desai)]

We were experiencing a similar issue, and it ended up being an interaction between the bridging code and iptables. Setting
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-ip6tables = 0

Fixed the issues for us.

[Duncan Idaho (ghola)]

As Natayan mentioned above the sysctl fixes the issue, but for us creates another issue because we actually want to use netfilter on the bridges. Sadly this is breaking part of our OpenStack deployment.

[Tamas Papp (tompos)]

Axtually I cannot reproduce on the same system right now:

Linux virt101 3.2.0-34-generic #53-Ubuntu SMP Thu Nov 15 10:48:16 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

I will check it again after a few days.

[penalvch (penalvch)]

Duncan Idaho, if you have a bug in Ubuntu, could you please file a new report by executing the following in a terminal:
ubuntu-bug linux

For more on this, please see the Ubuntu Kernel team article:
https://wiki.ubuntu.com/KernelTeam/KernelTeamBugPolicies#Filing_Kernel_Bug_reports

the Ubuntu Bug Control team and Ubuntu Bug Squad team article:
https://wiki.ubuntu.com/Bugs/BestPractices#X.2BAC8-Reporting.Focus_on_One_Issue

and Ubuntu Community article:
https://help.ubuntu.com/community/ReportingBugs#Bug_reporting_etiquette

When opening up the new report, please feel free to subscribe me to it.

Please note, not filing a new report may delay your problem being addressed as quickly as possible.

Thank you for your understanding.