Bridge does not work i ubuntu 12.04 - kernel 3.5.0-37

[Søren Laursen]

We are using ganeti on a HP ProLiant DL380p Gen8.

We are running kvm as virtualization.

Then enable a new ganeti instance, the kernel will make a new network interface (TAPx). After a minut or two, the instance starts to drop packages. Ping is sufficent.

We have tried on different network cards on the server, and all fails.

Changing to different kernels has not solved the problems.

Currently we have 6 ProLiant DL380p Gen7 running on the same configuration with no problems at all.

Regards

Søren Laursen

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.5.0-37-generic 3.5.0-37.58~precise1
ProcVersionSignature: Ubuntu 3.5.0-37.58~precise1-generic 3.5.7.16
Uname: Linux 3.5.0-37-generic x86_64
AlsaDevices:
 total 0
 crw-rw---T 1 root audio 116, 1 Aug 16 11:08 seq
 crw-rw---T 1 root audio 116, 33 Aug 16 11:08 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu17.4
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Date: Mon Aug 19 13:05:33 2013
HibernationDevice: RESUME=UUID=eb9606b5-5c67-4221-82c4-13d0f88abba9
InstallationMedia: Ubuntu-Server 12.04.2 LTS "Precise Pangolin" - Release amd64 (20130214)
MachineType: HP ProLiant DL380p Gen8
MarkForUpload: True
PciMultimedia:

ProcEnviron:
 LANGUAGE=en_GB:en
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
ProcFB: 0 VESA VGA
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.5.0-37-generic root=UUID=3a6a448e-8900-4eb5-b203-f5ddd1fbe28d ro
RelatedPackageVersions:
 linux-restricted-modules-3.5.0-37-generic N/A
 linux-backports-modules-3.5.0-37-generic N/A
 linux-firmware 1.79.6
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/14/2012
dmi.bios.vendor: HP
dmi.bios.version: P70
dmi.chassis.type: 23
dmi.chassis.vendor: HP
dmi.modalias: dmi:bvnHP:bvrP70:bd12/14/2012:svnHP:pnProLiantDL380pGen8:pvr:cvnHP:ct23:cvr:
dmi.product.name: ProLiant DL380p Gen8
dmi.sys.vendor: HP

[Brad Figg (brad-figg)]

This change was made by a bot.

[Joseph Salisbury (jsalisbury)]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.11 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.11-rc6-saucy/

[Søren Laursen (sl-6)]

Have tried:
Linux kvm07 3.5.0-37-generic #58~precise1-Ubuntu SMP Wed Jul 10 17:48:11 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Linux kvm07 3.10.7-031007-generic #201308150319 SMP Thu Aug 15 07:20:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Linux kvm07 3.11.0-999-generic #201308190423 SMP Mon Aug 19 08:24:36 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Linux kvm07 3.11.0-031100rc6-generic #201308181835 SMP Sun Aug 18 22:35:54 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

The bridge still fails. However, I have narrowed it a bit down.

If you create a simple tap interface for example:
sudo tunctl -u root -t tap42
sudo brctl addif mybridte tap42
sudo ip link set tap42 up
sudo ip addr add 10.0.0.10 dev tap42

It will continue to work and response to ping etc.

If the bridge is created by ganeti/kvm it will fail, if you try to make more than "one" connection. For example, start a ping and the a ssh connection the network freeze.

Regards

Søren

[Søren Laursen (sl-6)]

Tried to make some reduced testcases:

The test is done on:
Linux kvm07 3.11.0-031100rc6-generic #201308181835 SMP Sun Aug 18 22:35:54 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Later I tested it on Linux kvm07 3.11.0-031100-generic #201309021735 SMP Mon Sep 2 21:36:21 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux, the problems was not solved.

eth3 - testlan
driver: tg3
version: 3.132
firmware-version: 5719-v1.31 NCSI v1.1.15.0
bus-info: 0000:03:00.3
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes

eth4 - vlan280
driver: be2net
version: 4.6.62.0u
firmware-version: 4.2.401.605
bus-info: 0000:24:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: no

Small kvm scripts:
sudo tunctl -u root -t tap42
sudo brctl addif vlan280 tap42
sudo ip link set tap42 up
sudo kvm -hda vdisk.img -cdrom /srv/ganeti/iso/linux-secure-13.04-64bit.iso -m 1024 -net nic -net tap,ifname=tap42,script=no -boot d

sudo tunctl -u root -t tap42
sudo brctl addif vlan280 tap42
sudo ip link set tap42 up
sudo kvm -hda vdisk.img -cdrom /srv/ganeti/iso/ubuntu-rescue-remix-11-10.iso -m 1024 -net nic -net tap,ifname=tap42,script=no -boot d

sudo tunctl -u root -t tap42
sudo brctl addif vlan280 tap42
sudo ip link set tap42 up
sudo kvm -hda vdisk.img -cdrom /srv/ganeti/iso/clonezilla-live-2.1.1-7-amd64.iso -m 1024 -net nic -net tap,ifname=tap42,script=no -boot d

Then I testet it on testlan (tg3 device), the testlan is on the same network at vlan280
All the test working
sudo tunctl -u root -t tap43
sudo brctl addif testlan tap43xit
sudo ip link set tap43 up
sudo kvm -hda vdisk.img -cdrom /srv/ganeti/iso/linux-secure-13.04-64bit.iso -m 1024 -net nic -net tap,ifname=tap43,script=no -boot d

sudo tunctl -u root -t tap43
sudo brctl addif testlan tap43
sudo ip link set tap43 up
sudo kvm -hda vdisk.img -cdrom /srv/ganeti/iso/ubuntu-rescue-remix-11-10.iso -m 1024 -net nic -net tap,ifname=tap43,script=no -boot d

sudo tunctl -u root -t tap43
sudo brctl addif testlan tap43
sudo ip link set tap43 up
sudo kvm -hda vdisk.img -cdrom /srv/ganeti/iso/clonezilla-live-2.1.1-7-amd64.iso -m 1024 -net nic -net tap,ifname=tap43,script=no -boot d

Change the two network cards, that is testlan moved to eth4, vlan280 moved to eth3

Nothing worked at vlan280 and nothing worked at testlan. We could not get at dhcp working on vlan280 (eth3 - tg3) driver.

The bridge setup is as followed, with or without the mac address setup, the result is the same. And I remember to change it when change the bridge_ports. The other 10Gbit interface is used to do drbd and is working without problems.

auto vlan280
iface vlan280 inet static
      address 10.230.80.26
      netmask 255.255.255.0
      gateway 10.230.80.1
      dns-search fcoo.dk
      dns-nameservers 10.230.80.11 10.230.80.12
      bridge_ports eth3
      bridge_maxwait 0
      bridge_fd 0
      bridge_stp off
      post-up ip link set vlan280 address ac:16:2d:7b:35:4f

#
# TEST VLAN
#
auto eth4
iface eth4 inet manual

#
# TEST VLAN bridge
#
auto testlan
iface testlan inet manual
     bridge_ports eth4
     bridge_maxwait 0
     bridge_fd 0
     bridge_stp of...

[Søren Laursen (sl-6)]

Looks a lot like this bug:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1013199

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Søren Laursen (sl-6)]

Friday we change the network card to a card 10GBit card from Intel. Everything work. Then we put another Emulex OneConnect 10Gbps NIC(be3) card, and the bridge still works. Now we will put the original Emulex OneConnect 10Gbps NIC(be3) back in the server to check if we have a sort of hardware fail.