SRU Justification:
Impact:
When running KVM with a few VM's with physical eth bridges and TAP interfaces connected between KVM (via libvirt) and the bridge, occasionally kernel panics will occur.
Fix: Cherry-pick from a504b86e718a425ea4a34e2f95b5cf0545ddfd8d. This is present in Precise onwards.
Testcase:
0) Download attached netperf.tgz testcase.
1) Launch 3 instances (for example Ubuntu Natty/Oneiric).
2) Assign a public IP to each instance.
3) Copy all files from netperf.tgz to /root on the 3 instances.
4) Run the following command on each instance:
$ /root/netserver -p 12865
5) Create a file named 'remote_hosts' with the following content and copy it to /root on the 1st of the 3 instances:
REMOTE_HOSTS[0]=<public IP of 2nd instance>
REMOTE_HOSTS[1]=<public IP of 3rd instance>
NUM_REMOTE_HOSTS=2
6) Run the following command on the 1st instance to start the test:
$ cd /root
$ PATH=$PATH:. ./runemomniaggdemo.sh | tee overall.log
7) The machine will eventually panic.
--
On multiple servers on which we have Ubuntu versions installed we periodically hit this kernel panic. in netfilter flowpath The console freezes & the only way is to physically reboot the servers.
We run KVM and few VM's on the physical servers. On the physical eth's bridges are created and TAP interfaces are setup into the bridges. These TAP interfaces are connected into KVM through libvirt (we primarily use openstack for the VM management).
This problem happened in Natty 2.6.38-8-server version. After looking at the below links, we thought this could be a problem on 2.6.38-8-server kernel that is bundled along with Natty & decided to move to oneiric running 3.0 kernel.
http://lkml.indiana.edu/hypermail/linux/kernel/1106.0/00755.html
https://lkml.org/lkml/2011/6/1/754
http://www.spinics.net/lists/netfilter-devel/msg17239.html
https://lkml.org/lkml/2011/2/3/147
However, this exact problem is repeating in oneiric kernels. We request you to pls look at this issue asap. I will be happy to provide you any additional information that you will need.
Thanks
[ 5445.359446] [<ffffffffa0395200>] ? br_nf_pre_routing_finish_bridge+0x60/0xd0 [bridge]
[ 5445.372671] [<ffffffffa0396808>] br_nf_pre_routing_finish+0x328/0x360 [bridge]
[ 5445.385984] [<ffffffffa0396b96>] br_nf_pre_routing+0x356/0x370 [bridge]
[ 5445.393255] [<ffffffff815165d5>] nf_iterate+0x85/0xc0
[ 5445.400827] [<ffffffffa038ffd0>] ? br_handle_local_finish+0x50/0x50 [bridge]
[ 5445.408643] [<ffffffff81516686>] nf_hook_slow+0x76/0x130
[ 5445.416438] [<ffffffffa038ffd0>] ? br_hanlde_local_finish+0x50/0x50 [bridge]
[ 5445.424202] [<ffffffffa0390424>] br_handle_frame+0x1b4/0x260 [bridge]
[ 5445.431824] [<ffffffffa0390270>] ? br_hanlde_frame_finish+0x2a0/0x2a0 [bridge]
[ 5445.446876] [<ffffffff814e9b8d>] __netif_receive_skb+0x17d/0x540
[ 5445.454815] [<ffffffff814ea191>] process_backlog+0xb1/0x190
[ 5445.462790] [<ffffffff814eb2f4>] net_rx_action+0x134/0x290
[ 5445.470445] [<ffffffff81065ea8>] __do_softirq+0xa8/0x210
[ 5445.478364] [<ffffffff8160831c>] call_softirq+0x1c/0x30
[ 5445.486145] <EOI> [<ffffffff8100c295>] ? do_softirq+0x65/0xa0
[ 5445.494076] [<ffffffff814eb7d8>] netif_rx_ni+0x28/0x30
[ 5445.501902] [<ffffffff8144c702>] tun_chr_aio_write+0x342/0x4f0
[ 5445.509771] [<ffffffff8144c3c0>] ? tun_chr_aio_read+0xd0/0xd0
[ 5445.517864] [<ffffffff81167db3>] do_sync_readv_writev+0xd3/0x110
[ 5445.526011] [<ffffffff81281ecc>] ? security_file_permission+0x2c/0xb0
[ 5445.534177] [<ffffffff811674b1>] ? rw_verify_area+0x61/0xf0
[ 5445.542339] [<ffffffff8116807d>] do_readv_write_v+0xcd/0x1d0
[ 5445.550227] [<ffffffff811681bc>] vfs_writev+0x3c/0x50
[ 5445.558337] [<ffffffff8116831a>] sys_writev+0x4a/0xb0
[ 5445.566288] [<ffffffff81607102>] system_call_fastpath+0x16/0x1b
[ 5445.574267] ---[ end trace f585024e7045bff8 ]---
ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: linux-image-3.0.0-13-server 3.0.0-13.22
ProcVersionSignature: Ubuntu 3.0.0-13.22-server 3.0.6
Uname: Linux 3.0.0-13-server x86_64
AlsaDevices: Error: command ['ls', '-l', '/dev/snd/'] failed with exit code 2: ls: cannot access /dev/snd/: No such file or directory
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
CRDA: Error: [Errno 2] No such file or directory
Date: Fri Dec 16 09:51:41 2011
HibernationDevice: RESUME=UUID=5d83c5ae-1432-4ed0-be62-0b5bb54d8989
IwConfig: Error: [Errno 2] No such file or directory
MachineType: Dell Inc. PowerEdge R710
PciMultimedia:
ProcEnviron:
LANGUAGE=en_US:
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-3.0.0-13-server root=UUID=6adff529-7713-48d3-9174-d7e8f7fe5053 ro splash quiet vt.handoff=7
RelatedPackageVersions:
linux-restricted-modules-3.0.0-13-server N/A
linux-backports-modules-3.0.0-13-server N/A
linux-firmware 1.60
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: Upgraded to oneiric on 2011-12-09 (6 days ago)
dmi.bios.date: 01/31/2011
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 3.0.0
dmi.board.name: 00NH4P
dmi.board.vendor: Dell Inc.
dmi.board.version: A12
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr3.0.0:bd01/31/2011:svnDellInc.:pnPowerEdgeR710:pvr:rvnDellInc.:rn00NH4P:rvrA12:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R710
dmi.sys.vendor: Dell Inc.
Pls note that the stack trace is slightly different between the reference links of 2.6.38-8 netfilter corruption & the ones we are hitting.
We are constantly hitting the ones as shown in the attachments (png files).