Crash from 26.12.2011.
-----------------------------------------------------------------------------------
[3058856.377469] divide error: 0000 [#1] SMP 
[3058856.377477] last sysfs file: /sys/devices/virtual/block/md2/md/mismatch_cnt
[3058856.377480] CPU 3 
[3058856.377482] Modules linked in: ipv6 raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid1 raid0 multipath linear md_mod
[3058856.377497] Pid: 2536, comm: postgres Not tainted 2.6.32-309-ec2 #18-Ubuntu 
[3058856.377499] RIP: e030:[<ffffffff8102cdec>]  [<ffffffff8102cdec>] update_sd_lb_stats+0x3ac/0x4f0
[3058856.377508] RSP: e02b:ffff880068273708  EFLAGS: 00010046
[3058856.377510] RAX: 0000000000000000 RBX: ffff8800682738f4 RCX: 0000000000000001
[3058856.377513] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[3058856.377515] RBP: ffff8800682737e8 R08: ffff880002d48bc8 R09: 0000000000000040
[3058856.377517] R10: 0000000000000000 R11: 0000000000000001 R12: 00000000ffffffff
[3058856.377519] R13: 000000000000a380 R14: ffffffffffffffff R15: 0000000000000000
[3058856.377527] FS:  00007f42cdf0e720(0000) GS:ffff880002d41000(0000) knlGS:0000000000000000
[3058856.377529] CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[3058856.377531] CR2: 00007f4107a359df CR3: 00000002f3216000 CR4: 0000000000002620
[3058856.377534] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3058856.377536] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
[3058856.377538] Process postgres (pid: 2536, threadinfo ffff880068272000, task ffff8800f5c0a040)
[3058856.377540] Stack:
[3058856.377542]  ffffffff81490248 ffff880068273788 0000880068273738 0000000000000000
[3058856.377545] <0> ffff880002d48aa0 0000000300000000 0000000000000003 0000000000000008
[3058856.377550] <0> 000000000000a380 000000000000a380 ffff880002d48bb0 000000000000a380
[3058856.377554] Call Trace:
[3058856.377561]  [<ffffffff81490248>] ? packet_rcv_spkt+0x48/0x190
[3058856.377565]  [<ffffffff810388cd>] find_busiest_group+0x4d/0x460
[3058856.377569]  [<ffffffff810393e4>] load_balance_newidle+0xa4/0x320
[3058856.377575]  [<ffffffff814ae3b3>] thread_return+0x3c2/0x41f
[3058856.377580]  [<ffffffff8143b1c0>] ? ip_local_out+0x20/0x30
[3058856.377584]  [<ffffffff814ae735>] schedule_timeout+0x1e5/0x2c0
[3058856.377588]  [<ffffffff8144f836>] ? tcp_current_mss+0x46/0x70
[3058856.377591]  [<ffffffff814b0841>] ? _spin_lock_bh+0x11/0x70
[3058856.377595]  [<ffffffff813fa669>] sk_wait_data+0xd9/0xe0
[3058856.377601]  [<ffffffff81059f70>] ? autoremove_wake_function+0x0/0x40
[3058856.377605]  [<ffffffff81445474>] tcp_recvmsg+0x274/0xac0
[3058856.377608]  [<ffffffff813f96f4>] sock_common_recvmsg+0x34/0x50
[3058856.377611]  [<ffffffff813f8229>] sock_recvmsg+0x119/0x150
[3058856.377613]  [<ffffffff813f7b66>] ? sock_sendmsg+0x106/0x130
[3058856.377616]  [<ffffffff81059f70>] ? autoremove_wake_function+0x0/0x40
[3058856.377621]  [<ffffffff81100827>] ? core_sys_select+0x1e7/0x2b0
[3058856.377627]  [<ffffffff8133d666>] ? rb_erase+0xd6/0x160
[3058856.377630]  [<ffffffff8102b05b>] ? __dequeue_entity+0x2b/0x50
[3058856.377633]  [<ffffffff813f86dc>] sys_recvfrom+0xdc/0x170
[3058856.377638]  [<ffffffff81063840>] ? getnstimeofday+0x60/0xf0
[3058856.377641]  [<ffffffff81063935>] ? do_gettimeofday+0x15/0x50
[3058856.377646]  [<ffffffff81009ba8>] system_call_fastpath+0x16/0x1b
[3058856.377649]  [<ffffffff81009b40>] ? system_call+0x0/0x52
[3058856.377650] Code: ff c7 85 54 ff ff ff 01 00 00 00 e9 cd fd ff ff 0f 1f 80 00 00 00 00 48 8b 95 70 ff ff ff 48 8b 45 a8 8b 72 08 48 c1 e0 0a 31 d2 <48> f7 f6 48 8b 75 b0 48 89 45 a0 31 c0 48 85 f6 74 0c 48 8b 45 
[3058856.377685] RIP  [<ffffffff8102cdec>] update_sd_lb_stats+0x3ac/0x4f0
[3058856.377689]  RSP <ffff880068273708>
[3058856.377694] ---[ end trace 307fc082a7597501 ]---



Crash from 05.12.2011.
-----------------------------------------------------------------------------------
[836170.345306] divide error: 0000 [#1] SMP 
[836170.345314] last sysfs file: /sys/devices/virtual/block/md2/md/metadata_version
[836170.345316] CPU 3 
[836170.345319] Modules linked in: ipv6 raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid1 raid0 multipath linear md_mod
[836170.345333] Pid: 666, comm: md2_raid10 Not tainted 2.6.32-309-ec2 #18-Ubuntu 
[836170.345335] RIP: e030:[<ffffffff8102cdec>]  [<ffffffff8102cdec>] update_sd_lb_stats+0x3ac/0x4f0
[836170.345344] RSP: e02b:ffff880889943a70  EFLAGS: 00010046
[836170.345346] RAX: 0000000000000000 RBX: ffff880889943c5c RCX: 0000000000000001
[836170.345348] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[836170.345351] RBP: ffff880889943b50 R08: ffff880002d48bc8 R09: 0000000000000040
[836170.345353] R10: 0000000000000000 R11: 0000000000000001 R12: 00000000ffffffff
[836170.345355] R13: 000000000000a380 R14: ffffffffffffffff R15: 0000000000000000
[836170.345362] FS:  00007f87292a5720(0000) GS:ffff880002d41000(0000) knlGS:0000000000000000
[836170.345365] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[836170.345367] CR2: 00007f85c3d3e9df CR3: 000000075b8ad000 CR4: 0000000000002620
[836170.345369] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[836170.345371] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
[836170.345374] Process md2_raid10 (pid: 666, threadinfo ffff880889942000, task ffff880889b041c0)
[836170.345376] Stack:
[836170.345378]  ffff880889943b20 ffff880889943af0 0000880889943a90 0000000000000000
[836170.345382] <0> ffff880002d48aa0 0000000300000000 0000000000000003 0000000000000008
[836170.345386] <0> 000000000000a380 000000000000a380 ffff880002d48bb0 000000000000a380
[836170.345391] Call Trace:
[836170.345396]  [<ffffffff810388cd>] find_busiest_group+0x4d/0x460
[836170.345402]  [<ffffffff8133d526>] ? __rb_erase_color+0x166/0x1d0
[836170.345405]  [<ffffffff810393e4>] load_balance_newidle+0xa4/0x320
[836170.345412]  [<ffffffff814ae3b3>] thread_return+0x3c2/0x41f
[836170.345417]  [<ffffffffa0096894>] ? put_buf+0x24/0x30 [raid10]
[836170.345420]  [<ffffffff8131e81f>] ? blk_unplug+0x2f/0x70
[836170.345424]  [<ffffffff814ae735>] schedule_timeout+0x1e5/0x2c0
[836170.345427]  [<ffffffffa009a1ab>] ? raid10d+0x33b/0x408 [raid10]
[836170.345433]  [<ffffffffa0004bca>] md_thread+0xea/0x130 [md_mod]
[836170.345439]  [<ffffffff81059f70>] ? autoremove_wake_function+0x0/0x40
[836170.345443]  [<ffffffffa0004ae0>] ? md_thread+0x0/0x130 [md_mod]
[836170.345446]  [<ffffffff81059a8e>] kthread+0x8e/0xa0
[836170.345451]  [<ffffffff8100a70a>] child_rip+0xa/0x20
[836170.345453]  [<ffffffff81059a00>] ? kthread+0x0/0xa0
[836170.345456]  [<ffffffff8100a700>] ? child_rip+0x0/0x20
[836170.345458] Code: ff c7 85 54 ff ff ff 01 00 00 00 e9 cd fd ff ff 0f 1f 80 00 00 00 00 48 8b 95 70 ff ff ff 48 8b 45 a8 8b 72 08 48 c1 e0 0a 31 d2 <48> f7 f6 48 8b 75 b0 48 89 45 a0 31 c0 48 85 f6 74 0c 48 8b 45 
[836170.345493] RIP  [<ffffffff8102cdec>] update_sd_lb_stats+0x3ac/0x4f0
[836170.345496]  RSP <ffff880889943a70>
[836170.345501] ---[ end trace 3be04222d1d3df08 ]---