Oct  3 13:50:11 db01.customer.com kernel: : [220174.277583] divide error: 0000 [#1] SMP 
Oct  3 13:50:11 db01.customer.com kernel: : [220174.277937] last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
Oct  3 13:50:11 db01.customer.com kernel: : [220174.278300] CPU 14 
Oct  3 13:50:11 db01.customer.com kernel: : [220174.278500] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:50:11 db01.customer.com kernel: : [220174.283799] Pid: 18188, comm: mysqld Not tainted 2.6.32-29-server #58-Ubuntu PowerEdge R710
Oct  3 13:50:11 db01.customer.com kernel: : [220174.284174] RIP: 0010:[<ffffffff81055edc>]  [<ffffffff81055edc>] find_busiest_group+0x63c/0x9a0
Oct  3 13:50:11 db01.customer.com kernel: : [220174.284661] RSP: 0018:ffff8802260517d8  EFLAGS: 00010046
Oct  3 13:50:11 db01.customer.com kernel: : [220174.284904] RAX: 0000000000000000 RBX: ffff8802260519a4 RCX: 0000000000000001
Oct  3 13:50:11 db01.customer.com kernel: : [220174.285246] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Oct  3 13:50:11 db01.customer.com kernel: : [220174.285587] RBP: ffff880226051948 R08: ffff880133cefd10 R09: 0000000000000100
Oct  3 13:50:12 db01.customer.com kernel: : [220174.285933] R10: ffff880226696800 R11: 0000000000000001 R12: 00000000ffffffff
Oct  3 13:50:12 db01.customer.com kernel: : [220174.286274] R13: 0000000000015dc0 R14: ffffffffffffffff R15: 0000000000000000
Oct  3 13:50:12 db01.customer.com kernel: : [220174.286616] FS:  00007f9a0a8d9700(0000) GS:ffff880133ce0000(0000) knlGS:0000000000000000
Oct  3 13:50:12 db01.customer.com kernel: : [220174.313251] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Oct  3 13:50:12 db01.customer.com kernel: : [220174.327021] CR2: 00007f99d22af348 CR3: 000000022a668000 CR4: 00000000000006e0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.354614] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct  3 13:50:12 db01.customer.com kernel: : [220174.382996] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383000] Process mysqld (pid: 18188, threadinfo ffff880226050000, task ffff88007a918000)
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383003] Stack:
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383005]  ffff8802260518e8 ffff880226051858 ffff880226051990 0000000000003ee8
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383010] <0> ffff880133cefba0 0000000ea0197d4e 000000000000000e 0000000000000008
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383015] <0> 0000000000015dc0 0000000000015dc0 ffff880133cefce0 0000000000015dc0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383020] Call Trace:
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383030]  [<ffffffff8105c420>] load_balance_newidle+0xc0/0x340
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383040]  [<ffffffff8155b7ba>] thread_return+0x35a/0x420
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383046]  [<ffffffff8155bc5d>] schedule_timeout+0x22d/0x300
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383052]  [<ffffffff814678ac>] ? sk_reset_timer+0x1c/0x30
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383060]  [<ffffffff810397f9>] ? default_spin_lock_flags+0x9/0x10
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383066]  [<ffffffff8155d7af>] ? _spin_lock_irqsave+0x2f/0x40
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383071]  [<ffffffff810397f9>] ? default_spin_lock_flags+0x9/0x10
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383076]  [<ffffffff81467a79>] sk_wait_data+0xd9/0xe0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383083]  [<ffffffff81085b40>] ? autoremove_wake_function+0x0/0x40
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383088]  [<ffffffff814673da>] ? release_sock+0xba/0xd0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383096]  [<ffffffff814b4d7e>] tcp_recvmsg+0x2ce/0xe90
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383104]  [<ffffffff81466a09>] sock_common_recvmsg+0x39/0x50
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383109]  [<ffffffff814665f9>] sock_aio_read+0x159/0x160
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383139]  [<ffffffffa00a8e9c>] ? xfs_iunlock+0xac/0xc0 [xfs]
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383148]  [<ffffffff811460ea>] do_sync_read+0xfa/0x140
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383154]  [<ffffffff81085b40>] ? autoremove_wake_function+0x0/0x40
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383160]  [<ffffffff8101184e>] ? __switch_to+0x26e/0x320
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383166]  [<ffffffff810582d0>] ? finish_task_switch+0x50/0xe0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383172]  [<ffffffff81254af6>] ? security_file_permission+0x16/0x20
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383178]  [<ffffffff81146aa1>] vfs_read+0x181/0x1a0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383183]  [<ffffffff81155b75>] ? do_fcntl+0xb5/0x3d0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383188]  [<ffffffff81146b91>] sys_read+0x51/0x80
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383192]  [<ffffffff81155f00>] ? sys_fcntl+0x70/0x90
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383198]  [<ffffffff810131b2>] system_call_fastpath+0x16/0x1b
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383200] Code: ff c7 85 c4 fe ff ff 01 00 00 00 e9 95 fb ff ff 0f 1f 80 00 00 00 00 48 8b 95 e0 fe ff ff 48 8b 45 a8 8b 72 08 48 c1 e0 0a 31 d2 <48> f7 f6 48 8b 75 b0 48 89 45 a0 31 c0 48 85 f6 74 0c 48 8b 45 
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383230] RIP  [<ffffffff81055edc>] find_busiest_group+0x63c/0x9a0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.383236]  RSP <ffff8802260517d8>
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393685] ---[ end trace d08e3a1f920f04e2 ]---
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393688] divide error: 0000 [#2] SMP 
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393691] last sysfs file: /sys/devices/system/cpu/cpu15/cache/index2/shared_cpu_map
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393694] CPU 6 
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393696] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393727] Pid: 246, comm: kondemand/6 Tainted: G      D    2.6.32-29-server #58-Ubuntu PowerEdge R710
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393731] RIP: 0010:[<ffffffff81055edc>]  [<ffffffff81055edc>] find_busiest_group+0x63c/0x9a0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393738] RSP: 0018:ffff88022a9dbbb0  EFLAGS: 00010046
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393741] RAX: 0000000000000000 RBX: ffff88022a9dbd7c RCX: 0000000000000000
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393745] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393748] RBP: ffff88022a9dbd20 R08: ffff880133cefd10 R09: 0000000000000100
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393751] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393755] R13: 0000000000015dc0 R14: 0000000000000000 R15: 0000000000000000
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393759] FS:  0000000000000000(0000) GS:ffff880133c60000(0000) knlGS:0000000000000000
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393763] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393766] CR2: 00007f99d22af348 CR3: 0000000001001000 CR4: 00000000000006e0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393770] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393773] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393777] Process kondemand/6 (pid: 246, threadinfo ffff88022a9da000, task ffff88022a9d2e00)
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393780] Stack:
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393782]  ffff88022a9dbcc0 ffff88022a9dbc30 ffff88022a9dbd68 000000000000ffff
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393786] <0> ffff880133c6fba0 00000006815606e6 00000000ffffffff 0000000000000008
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393791] <0> 0000000000015dc0 0000000000015dc0 ffff880133cefce0 0000000000015dc0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393795] Call Trace:
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393802]  [<ffffffff8105c420>] load_balance_newidle+0xc0/0x340
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393809]  [<ffffffff8155b7ba>] thread_return+0x35a/0x420
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393817]  [<ffffffff8108115a>] worker_thread+0xda/0x110
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393824]  [<ffffffff81085b40>] ? autoremove_wake_function+0x0/0x40
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393831]  [<ffffffff81081080>] ? worker_thread+0x0/0x110
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393838]  [<ffffffff810857c6>] kthread+0x96/0xa0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393845]  [<ffffffff810141ea>] child_rip+0xa/0x20
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393851]  [<ffffffff81085730>] ? kthread+0x0/0xa0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393857]  [<ffffffff810141e0>] ? child_rip+0x0/0x20
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393861] Code: ff c7 85 c4 fe ff ff 01 00 00 00 e9 95 fb ff ff 0f 1f 80 00 00 00 00 48 8b 95 e0 fe ff ff 48 8b 45 a8 8b 72 08 48 c1 e0 0a 31 d2 <48> f7 f6 48 8b 75 b0 48 89 45 a0 31 c0 48 85 f6 74 0c 48 8b 45 
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393911] RIP  [<ffffffff81055edc>] find_busiest_group+0x63c/0x9a0
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393918]  RSP <ffff88022a9dbbb0>
Oct  3 13:50:12 db01.customer.com kernel: : [220174.393922] ---[ end trace d08e3a1f920f04e3 ]---
Oct  3 13:51:18 db01.customer.com kernel: : [220240.554384] BUG: soft lockup - CPU#15 stuck for 61s! [mysqld:20778]
Oct  3 13:51:18 db01.customer.com kernel: : [220240.591043] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:18 db01.customer.com kernel: : [220240.708495] CPU 15:
Oct  3 13:51:18 db01.customer.com kernel: : [220240.726292] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:18 db01.customer.com kernel: : [220240.838844] Pid: 20778, comm: mysqld Tainted: G      D    2.6.32-29-server #58-Ubuntu PowerEdge R710
Oct  3 13:51:18 db01.customer.com kernel: : [220240.870803] RIP: 0010:[<ffffffff812c0a30>]  [<ffffffff812c0a30>] __bitmap_empty+0x0/0x90
Oct  3 13:51:18 db01.customer.com kernel: : [220240.901161] RSP: 0018:ffff8802274eddb0  EFLAGS: 00000246
Oct  3 13:51:18 db01.customer.com kernel: : [220240.916331] RAX: 0000000000000000 RBX: ffff8802274eddf8 RCX: 000000000000000f
Oct  3 13:51:18 db01.customer.com kernel: : [220240.946336] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffffffff819a22d8
Oct  3 13:51:18 db01.customer.com kernel: : [220240.976245] RBP: ffffffff81013cae R08: 0000000000000004 R09: 0000000000000100
Oct  3 13:51:18 db01.customer.com kernel: : [220241.005803] R10: 00000000008006d0 R11: 0000000000000000 R12: ffff8802274edda8
Oct  3 13:51:18 db01.customer.com kernel: : [220241.034640] R13: ffffffff81013cae R14: 0000000000011fa0 R15: ffff880000014918
Oct  3 13:51:18 db01.customer.com kernel: : [220241.061199] FS:  00007f9a03db6700(0000) GS:ffff8800058e0000(0000) knlGS:0000000000000000
Oct  3 13:51:18 db01.customer.com kernel: : [220241.087876] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  3 13:51:18 db01.customer.com kernel: : [220241.101811] CR2: 00007f99c5fd26d8 CR3: 000000022a668000 CR4: 00000000000006e0
Oct  3 13:51:18 db01.customer.com kernel: : [220241.129369] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct  3 13:51:18 db01.customer.com kernel: : [220241.157298] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct  3 13:51:18 db01.customer.com kernel: : [220241.185620] Call Trace:
Oct  3 13:51:18 db01.customer.com kernel: : [220241.199483]  [<ffffffff81045e87>] ? flush_tlb_others_ipi+0x127/0x130
Oct  3 13:51:18 db01.customer.com kernel: : [220241.213519]  [<ffffffff81045e9e>] ? native_flush_tlb_others+0xe/0x10
Oct  3 13:51:18 db01.customer.com kernel: : [220241.227501]  [<ffffffff8104604c>] ? flush_tlb_mm+0x5c/0xa0
Oct  3 13:51:18 db01.customer.com kernel: : [220241.241130]  [<ffffffff81116313>] ? zap_page_range+0xf3/0x120
Oct  3 13:51:18 db01.customer.com kernel: : [220241.254649]  [<ffffffff81111fcf>] ? madvise_vma+0x1cf/0x210
Oct  3 13:51:18 db01.customer.com kernel: : [220241.267987]  [<ffffffff8111216d>] ? sys_madvise+0x15d/0x290
Oct  3 13:51:18 db01.customer.com kernel: : [220241.281286]  [<ffffffff810131b2>] ? system_call_fastpath+0x16/0x1b
Oct  3 13:51:18 db01.customer.com kernel: : [220241.294527] BUG: soft lockup - CPU#1 stuck for 61s! [mysqld:30363]
Oct  3 13:51:18 db01.customer.com kernel: : [220241.307773] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:18 db01.customer.com kernel: : [220241.390837] CPU 1:
Oct  3 13:51:18 db01.customer.com kernel: : [220241.402943] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:18 db01.customer.com kernel: : [220241.482253] Pid: 30363, comm: mysqld Tainted: G      D    2.6.32-29-server #58-Ubuntu PowerEdge R710
Oct  3 13:51:18 db01.customer.com kernel: : [220241.505707] RIP: 0010:[<ffffffff81045e7a>]  [<ffffffff81045e7a>] flush_tlb_others_ipi+0x11a/0x130
Oct  3 13:51:18 db01.customer.com kernel: : [220241.528506] RSP: 0018:ffff88022776bdb8  EFLAGS: 00000246
Oct  3 13:51:18 db01.customer.com kernel: : [220241.539705] RAX: 0000000000000000 RBX: ffff88022776bdf8 RCX: 000000000000000f
Oct  3 13:51:18 db01.customer.com kernel: : [220241.561961] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffffffff819a2158
Oct  3 13:51:18 db01.customer.com kernel: : [220241.585084] RBP: ffffffff81013cae R08: 0000000000000004 R09: 0000000000000100
Oct  3 13:51:18 db01.customer.com kernel: : [220241.609097] R10: 0000000000800270 R11: 0000000000000000 R12: ffff88022776bdf8
Oct  3 13:51:18 db01.customer.com kernel: : [220241.633936] R13: ffffffff81013cae R14: 0000000000011fa0 R15: ffff880000014918
Oct  3 13:51:18 db01.customer.com kernel: : [220241.659491] FS:  00007f9a0a13b700(0000) GS:ffff880005800000(0000) knlGS:0000000000000000
Oct  3 13:51:18 db01.customer.com kernel: : [220241.685373] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  3 13:51:18 db01.customer.com kernel: : [220241.698837] CR2: 00007f99dc976b38 CR3: 000000022a668000 CR4: 00000000000006e0
Oct  3 13:51:18 db01.customer.com kernel: : [220241.725368] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct  3 13:51:18 db01.customer.com kernel: : [220241.752269] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct  3 13:51:18 db01.customer.com kernel: : [220241.779642] Call Trace:
Oct  3 13:51:18 db01.customer.com kernel: : [220241.793026]  [<ffffffff81045e87>] ? flush_tlb_others_ipi+0x127/0x130
Oct  3 13:51:18 db01.customer.com kernel: : [220241.806585]  [<ffffffff81045e9e>] ? native_flush_tlb_others+0xe/0x10
Oct  3 13:51:18 db01.customer.com kernel: : [220241.820090]  [<ffffffff8104604c>] ? flush_tlb_mm+0x5c/0xa0
Oct  3 13:51:18 db01.customer.com kernel: : [220241.833202]  [<ffffffff81116313>] ? zap_page_range+0xf3/0x120
Oct  3 13:51:18 db01.customer.com kernel: : [220241.846206]  [<ffffffff81111fcf>] ? madvise_vma+0x1cf/0x210
Oct  3 13:51:18 db01.customer.com kernel: : [220241.859070]  [<ffffffff8111216d>] ? sys_madvise+0x15d/0x290
Oct  3 13:51:18 db01.customer.com kernel: : [220241.871894]  [<ffffffff810131b2>] ? system_call_fastpath+0x16/0x1b
Oct  3 13:51:18 db01.customer.com kernel: : [220241.884747] BUG: soft lockup - CPU#7 stuck for 61s! [mysqld:11013]
Oct  3 13:51:19 db01.customer.com kernel: : [220241.898067] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:19 db01.customer.com kernel: : [220241.981296] CPU 7:
Oct  3 13:51:19 db01.customer.com kernel: : [220241.993410] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:19 db01.customer.com kernel: : [220242.072958] Pid: 11013, comm: mysqld Tainted: G      D    2.6.32-29-server #58-Ubuntu PowerEdge R710
Oct  3 13:51:19 db01.customer.com kernel: : [220242.096714] RIP: 0010:[<ffffffff81039769>]  [<ffffffff81039769>] __ticket_spin_lock+0x19/0x20
Oct  3 13:51:19 db01.customer.com kernel: : [220242.119920] RSP: 0018:ffff88007db0dd98  EFLAGS: 00000297
Oct  3 13:51:19 db01.customer.com kernel: : [220242.131501] RAX: 00000000000008ae RBX: ffff88007db0dd98 RCX: 0000000000000000
Oct  3 13:51:19 db01.customer.com kernel: : [220242.154514] RDX: 00000000000008ad RSI: ffff88022a406c80 RDI: ffffffff819a22d0
Oct  3 13:51:19 db01.customer.com kernel: : [220242.178399] RBP: ffffffff81013cae R08: ffff88022a406f40 R09: 0000000000000000
Oct  3 13:51:19 db01.customer.com kernel: : [220242.203276] R10: 0000000000800760 R11: 0000000000000000 R12: 0000000000000001
Oct  3 13:51:19 db01.customer.com kernel: : [220242.228905] R13: 0000000000000000 R14: ffffffff81114725 R15: ffff88007db0dd88
Oct  3 13:51:19 db01.customer.com kernel: : [220242.255295] FS:  00007f9a0375d700(0000) GS:ffff880005860000(0000) knlGS:0000000000000000
Oct  3 13:51:19 db01.customer.com kernel: : [220242.281984] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  3 13:51:19 db01.customer.com kernel: : [220242.295858] CR2: 00007f99a8a72648 CR3: 000000022a668000 CR4: 00000000000006e0
Oct  3 13:51:19 db01.customer.com kernel: : [220242.323206] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct  3 13:51:19 db01.customer.com kernel: : [220242.350928] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct  3 13:51:19 db01.customer.com kernel: : [220242.379131] Call Trace:
Oct  3 13:51:19 db01.customer.com kernel: : [220242.392926]  [<ffffffff81114f4a>] ? unmap_vmas+0x17a/0x310
Oct  3 13:51:19 db01.customer.com kernel: : [220242.406827]  [<ffffffff8155d88e>] ? _spin_lock+0xe/0x20
Oct  3 13:51:19 db01.customer.com kernel: : [220242.420731]  [<ffffffff81045dc6>] ? flush_tlb_others_ipi+0x66/0x130
Oct  3 13:51:19 db01.customer.com kernel: : [220242.434433]  [<ffffffff81045e9e>] ? native_flush_tlb_others+0xe/0x10
Oct  3 13:51:19 db01.customer.com kernel: : [220242.448072]  [<ffffffff8104604c>] ? flush_tlb_mm+0x5c/0xa0
Oct  3 13:51:19 db01.customer.com kernel: : [220242.461373]  [<ffffffff81116313>] ? zap_page_range+0xf3/0x120
Oct  3 13:51:19 db01.customer.com kernel: : [220242.474678]  [<ffffffff81111fcf>] ? madvise_vma+0x1cf/0x210
Oct  3 13:51:19 db01.customer.com kernel: : [220242.487913]  [<ffffffff8111216d>] ? sys_madvise+0x15d/0x290
Oct  3 13:51:19 db01.customer.com kernel: : [220242.501027]  [<ffffffff810131b2>] ? system_call_fastpath+0x16/0x1b
Oct  3 13:51:19 db01.customer.com kernel: : [220242.514036] BUG: soft lockup - CPU#8 stuck for 61s! [mysqld:31749]
Oct  3 13:51:20 db01.customer.com kernel: : [220242.527120] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:20 db01.customer.com kernel: : [220242.608698] CPU 8:
Oct  3 13:51:20 db01.customer.com kernel: : [220242.620535] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:20 db01.customer.com kernel: : [220242.698498] Pid: 31749, comm: mysqld Tainted: G      D    2.6.32-29-server #58-Ubuntu PowerEdge R710
Oct  3 13:51:20 db01.customer.com kernel: : [220242.721714] RIP: 0010:[<ffffffff812c0a30>]  [<ffffffff812c0a30>] __bitmap_empty+0x0/0x90
Oct  3 13:51:20 db01.customer.com kernel: : [220242.744356] RSP: 0018:ffff88001bae9db0  EFLAGS: 00000246
Oct  3 13:51:20 db01.customer.com kernel: : [220242.755926] RAX: 0000000000000000 RBX: ffff88001bae9df8 RCX: 000000000000000f
Oct  3 13:51:20 db01.customer.com kernel: : [220242.779079] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffffffff819a2118
Oct  3 13:51:20 db01.customer.com kernel: : [220242.802913] RBP: ffffffff81013cae R08: 0000000000000004 R09: 0000000000000100
Oct  3 13:51:20 db01.customer.com kernel: : [220242.827644] R10: 0000000000800a60 R11: 0000000000000000 R12: ffff88001bae9da8
Oct  3 13:51:20 db01.customer.com kernel: : [220242.853078] R13: ffffffff81013cae R14: 0000000000011fa0 R15: ffff880130003928
Oct  3 13:51:20 db01.customer.com kernel: : [220242.879272] FS:  00007f9a09c68700(0000) GS:ffff880133c80000(0000) knlGS:0000000000000000
Oct  3 13:51:20 db01.customer.com kernel: : [220242.905757] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  3 13:51:20 db01.customer.com kernel: : [220242.919572] CR2: 00007f99d22af348 CR3: 000000022a668000 CR4: 00000000000006e0
Oct  3 13:51:20 db01.customer.com kernel: : [220242.946802] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct  3 13:51:20 db01.customer.com kernel: : [220242.974405] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct  3 13:51:20 db01.customer.com kernel: : [220243.002490] Call Trace:
Oct  3 13:51:20 db01.customer.com kernel: : [220243.016231]  [<ffffffff81045e87>] ? flush_tlb_others_ipi+0x127/0x130
Oct  3 13:51:20 db01.customer.com kernel: : [220243.030146]  [<ffffffff81045e9e>] ? native_flush_tlb_others+0xe/0x10
Oct  3 13:51:20 db01.customer.com kernel: : [220243.044005]  [<ffffffff8104604c>] ? flush_tlb_mm+0x5c/0xa0
Oct  3 13:51:20 db01.customer.com kernel: : [220243.057465]  [<ffffffff81116313>] ? zap_page_range+0xf3/0x120
Oct  3 13:51:20 db01.customer.com kernel: : [220243.070814]  [<ffffffff81111fcf>] ? madvise_vma+0x1cf/0x210
Oct  3 13:51:20 db01.customer.com kernel: : [220243.083938]  [<ffffffff8111216d>] ? sys_madvise+0x15d/0x290
Oct  3 13:51:20 db01.customer.com kernel: : [220243.097025]  [<ffffffff810131b2>] ? system_call_fastpath+0x16/0x1b
Oct  3 13:51:20 db01.customer.com kernel: : [220243.110054] BUG: soft lockup - CPU#12 stuck for 61s! [mysqld:18908]
Oct  3 13:51:20 db01.customer.com kernel: : [220243.123324] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:20 db01.customer.com kernel: : [220243.206100] CPU 12:
Oct  3 13:51:20 db01.customer.com kernel: : [220243.218162] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:20 db01.customer.com kernel: : [220243.297514] Pid: 18908, comm: mysqld Tainted: G      D    2.6.32-29-server #58-Ubuntu PowerEdge R710
Oct  3 13:51:20 db01.customer.com kernel: : [220243.321129] RIP: 0010:[<ffffffff812c0ab3>]  [<ffffffff812c0ab3>] __bitmap_empty+0x83/0x90
Oct  3 13:51:20 db01.customer.com kernel: : [220243.344137] RSP: 0018:ffff880226969db0  EFLAGS: 00000246
Oct  3 13:51:20 db01.customer.com kernel: : [220243.355588] RAX: 0000000000000000 RBX: ffff880226969df8 RCX: 000000000000000f
Oct  3 13:51:20 db01.customer.com kernel: : [220243.378261] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffffffff819a2218
Oct  3 13:51:20 db01.customer.com kernel: : [220243.401804] RBP: ffffffff81013cae R08: 0000000000000004 R09: 0000000000000100
Oct  3 13:51:20 db01.customer.com kernel: : [220243.426326] R10: 0000000000800070 R11: 0000000000000000 R12: ffff880226969df8
Oct  3 13:51:20 db01.customer.com kernel: : [220243.451683] R13: ffffffff81013cae R14: 0000000000011fa0 R15: ffff880130003928
Oct  3 13:51:20 db01.customer.com kernel: : [220243.477704] FS:  00007f9a0913c700(0000) GS:ffff880133cc0000(0000) knlGS:0000000000000000
Oct  3 13:51:20 db01.customer.com kernel: : [220243.503927] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  3 13:51:20 db01.customer.com kernel: : [220243.517606] CR2: 00007f99c0938d38 CR3: 000000022a668000 CR4: 00000000000006e0
Oct  3 13:51:20 db01.customer.com kernel: : [220243.544652] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct  3 13:51:20 db01.customer.com kernel: : [220243.572067] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct  3 13:51:20 db01.customer.com kernel: : [220243.599874] Call Trace:
Oct  3 13:51:20 db01.customer.com kernel: : [220243.613475]  [<ffffffff81045e87>] ? flush_tlb_others_ipi+0x127/0x130
Oct  3 13:51:20 db01.customer.com kernel: : [220243.627296]  [<ffffffff81045e9e>] ? native_flush_tlb_others+0xe/0x10
Oct  3 13:51:20 db01.customer.com kernel: : [220243.641062]  [<ffffffff8104604c>] ? flush_tlb_mm+0x5c/0xa0
Oct  3 13:51:20 db01.customer.com kernel: : [220243.654433]  [<ffffffff81116313>] ? zap_page_range+0xf3/0x120
Oct  3 13:51:20 db01.customer.com kernel: : [220243.667696]  [<ffffffff81111fcf>] ? madvise_vma+0x1cf/0x210
Oct  3 13:51:20 db01.customer.com kernel: : [220243.680780]  [<ffffffff8111216d>] ? sys_madvise+0x15d/0x290
Oct  3 13:51:20 db01.customer.com kernel: : [220243.693824]  [<ffffffff810131b2>] ? system_call_fastpath+0x16/0x1b
Oct  3 13:51:20 db01.customer.com kernel: : [220243.706812] BUG: soft lockup - CPU#2 stuck for 61s! [mysqld:17466]
Oct  3 13:51:21 db01.customer.com kernel: : [220243.720160] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:21 db01.customer.com kernel: : [220243.803847] CPU 2:
Oct  3 13:51:21 db01.customer.com kernel: : [220243.816037] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:21 db01.customer.com kernel: : [220243.895988] Pid: 17466, comm: mysqld Tainted: G      D    2.6.32-29-server #58-Ubuntu PowerEdge R710
Oct  3 13:51:21 db01.customer.com kernel: : [220243.919681] RIP: 0010:[<ffffffff81045e7a>]  [<ffffffff81045e7a>] flush_tlb_others_ipi+0x11a/0x130
Oct  3 13:51:21 db01.customer.com kernel: : [220243.942759] RSP: 0018:ffff880069e7ddb8  EFLAGS: 00000246
Oct  3 13:51:21 db01.customer.com kernel: : [220243.954117] RAX: 0000000000000000 RBX: ffff880069e7ddf8 RCX: 0000000000000010
Oct  3 13:51:21 db01.customer.com kernel: : [220243.976594] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffffffff819a2198
Oct  3 13:51:21 db01.customer.com kernel: : [220243.999853] RBP: ffffffff81013cae R08: 0000000000000004 R09: 0000000000000100
Oct  3 13:51:21 db01.customer.com kernel: : [220244.024102] R10: 0000000000800330 R11: 0000000000000000 R12: ffff880069e7ddf8
Oct  3 13:51:21 db01.customer.com kernel: : [220244.049187] R13: ffffffff81013cae R14: 0000000000011fa0 R15: ffff880130003928
Oct  3 13:51:21 db01.customer.com kernel: : [220244.075033] FS:  00007f99b7a28700(0000) GS:ffff880133c20000(0000) knlGS:0000000000000000
Oct  3 13:51:21 db01.customer.com kernel: : [220244.101256] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  3 13:51:21 db01.customer.com kernel: : [220244.114986] CR2: 00007f99e88e0a78 CR3: 000000022a668000 CR4: 00000000000006e0
Oct  3 13:51:21 db01.customer.com kernel: : [220244.142121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct  3 13:51:21 db01.customer.com kernel: : [220244.169630] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct  3 13:51:21 db01.customer.com kernel: : [220244.197627] Call Trace:
Oct  3 13:51:21 db01.customer.com kernel: : [220244.211320]  [<ffffffff81045e87>] ? flush_tlb_others_ipi+0x127/0x130
Oct  3 13:51:21 db01.customer.com kernel: : [220244.225238]  [<ffffffff81045e9e>] ? native_flush_tlb_others+0xe/0x10
Oct  3 13:51:21 db01.customer.com kernel: : [220244.239100]  [<ffffffff8104604c>] ? flush_tlb_mm+0x5c/0xa0
Oct  3 13:51:21 db01.customer.com kernel: : [220244.252560]  [<ffffffff81116313>] ? zap_page_range+0xf3/0x120
Oct  3 13:51:21 db01.customer.com kernel: : [220244.265914]  [<ffffffff81111fcf>] ? madvise_vma+0x1cf/0x210
Oct  3 13:51:21 db01.customer.com kernel: : [220244.279084]  [<ffffffff8111216d>] ? sys_madvise+0x15d/0x290
Oct  3 13:51:21 db01.customer.com kernel: : [220244.292215]  [<ffffffff810131b2>] ? system_call_fastpath+0x16/0x1b
Oct  3 13:51:21 db01.customer.com kernel: : [220244.305328] BUG: soft lockup - CPU#10 stuck for 61s! [mysqld:30417]
Oct  3 13:51:22 db01.customer.com kernel: : [220244.318923] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:22 db01.customer.com kernel: : [220244.404356] CPU 10:
Oct  3 13:51:22 db01.customer.com kernel: : [220244.416838] Modules linked in: ip6table_filter ip6_tables ipt_MASQUERADE xt_tcpudp xt_limit xt_state ipt_REJECT nf_conntrack_ftp iptable_mangle iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 iptable_filter ip_tables x_tables fbcon tileblit font bitblit softcursor bnx2 dell_wmi psmouse lp vga16fb vgastate serio_raw power_meter dcdbas joydev parport xfs exportfs usbhid hid ses enclosure usb_storage megaraid_sas
Oct  3 13:51:22 db01.customer.com kernel: : [220244.498373] Pid: 30417, comm: mysqld Tainted: G      D    2.6.32-29-server #58-Ubuntu PowerEdge R710
Oct  3 13:51:22 db01.customer.com kernel: : [220244.522473] RIP: 0010:[<ffffffff81039769>]  [<ffffffff81039769>] __ticket_spin_lock+0x19/0x20
Oct  3 13:51:22 db01.customer.com kernel: : [220244.545939] RSP: 0018:ffff8802265b9ba8  EFLAGS: 00000297
Oct  3 13:51:22 db01.customer.com kernel: : [220244.557586] RAX: 00000000000001d7 RBX: ffff8802265b9ba8 RCX: 0000000000000000
Oct  3 13:51:22 db01.customer.com kernel: : [220244.580642] RDX: 00000000000001d6 RSI: ffff88022a406c80 RDI: ffffffff819a2190
Oct  3 13:51:22 db01.customer.com kernel: : [220244.604487] RBP: ffffffff81013cae R08: ffff88022a406f40 R09: 0000000000000000
Oct  3 13:51:22 db01.customer.com kernel: : [220244.629324] R10: 00000000003705b0 R11: 0000000000000000 R12: ffffffff8155dd55
Oct  3 13:51:22 db01.customer.com kernel: : [220244.655004] R13: ffff8802265b9c08 R14: 0000000000000000 R15: 0000000000001000
Oct  3 13:51:22 db01.customer.com kernel: : [220244.681316] FS:  00007f9a03c30700(0000) GS:ffff880133ca0000(0000) knlGS:0000000000000000
Oct  3 13:51:22 db01.customer.com kernel: : [220244.707695] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Oct  3 13:51:22 db01.customer.com kernel: : [220244.721460] CR2: 00007f99c8377a5f CR3: 000000022a668000 CR4: 00000000000006e0
Oct  3 13:51:22 db01.customer.com kernel: : [220244.748587] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Oct  3 13:51:22 db01.customer.com kernel: : [220244.776087] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Oct  3 13:51:22 db01.customer.com kernel: : [220244.804070] Call Trace:
Oct  3 13:51:22 db01.customer.com kernel: : [220244.817756]  [<ffffffff8155d88e>] ? _spin_lock+0xe/0x20
Oct  3 13:51:22 db01.customer.com kernel: : [220244.831524]  [<ffffffff81045dc6>] ? flush_tlb_others_ipi+0x66/0x130
Oct  3 13:51:22 db01.customer.com kernel: : [220244.845350]  [<ffffffff81045e9e>] ? native_flush_tlb_others+0xe/0x10
Oct  3 13:51:22 db01.customer.com kernel: : [220244.858870]  [<ffffffff8104604c>] ? flush_tlb_mm+0x5c/0xa0
Oct  3 13:51:22 db01.customer.com kernel: : [220244.872187]  [<ffffffff8111373b>] ? zap_pte_range+0x2fb/0x400
Oct  3 13:51:22 db01.customer.com kernel: : [220244.885260]  [<ffffffff81114725>] ? unmap_page_range+0x1c5/0x2a0
Oct  3 13:51:22 db01.customer.com kernel: : [220244.898409]  [<ffffffffa00a8e9c>] ? xfs_iunlock+0xac/0xc0 [xfs]
Oct  3 13:51:22 db01.customer.com kernel: : [220244.911480]  [<ffffffff81114f4a>] ? unmap_vmas+0x17a/0x310
Oct  3 13:51:22 db01.customer.com kernel: : [220244.924439]  [<ffffffff811162d8>] ? zap_page_range+0xb8/0x120
Oct  3 13:51:22 db01.customer.com kernel: : [220244.937296]  [<ffffffff812bab16>] ? rb_erase+0xd6/0x160
Oct  3 13:51:22 db01.customer.com kernel: : [220244.949936]  [<ffffffff8101178c>] ? __switch_to+0x1ac/0x320
Oct  3 13:51:22 db01.customer.com kernel: : [220244.962355]  [<ffffffff81111fcf>] ? madvise_vma+0x1cf/0x210
Oct  3 13:51:22 db01.customer.com kernel: : [220244.974651]  [<ffffffff8155b4a8>] ? thread_return+0x48/0x420
Oct  3 13:51:22 db01.customer.com kernel: : [220244.986717]  [<ffffffff8111216d>] ? sys_madvise+0x15d/0x290
Oct  3 13:51:22 db01.customer.com kernel: : [220244.998497]  [<ffffffff810131b2>] ? system_call_fastpath+0x16/0x1b