May  2 12:47:46 mainserver kernel: [307624.885499] BUG: soft lockup - CPU#3 stuck for 61s! [dd:2251]
May  2 12:47:46 mainserver kernel: [307624.885502] Modules linked in: i915 drm binfmt_misc rfcomm bnep sco l2cap bluetooth tun ipt_MASQUERADE iptable_nat $
May  2 12:47:46 mainserver kernel: ner snd_page_alloc ext3 jbd mbcache sr_mod cdrom sd_mod crc_t10dif sg ata_generic ata_piix pata_it8213 pata_acpi libata$
May  2 12:47:46 mainserver kernel: [307624.885502] CPU 3:
May  2 12:47:46 mainserver kernel: [307624.885502] Modules linked in: i915 drm binfmt_misc rfcomm bnep sco l2cap bluetooth tun ipt_MASQUERADE iptable_nat $
May  2 12:47:46 mainserver kernel: ner snd_page_alloc ext3 jbd mbcache sr_mod cdrom sd_mod crc_t10dif sg ata_generic ata_piix pata_it8213 pata_acpi libata$
May  2 12:47:46 mainserver kernel: [307624.885502] Pid: 2251, comm: dd Not tainted 2.6.27-11-generic #1
May  2 12:47:46 mainserver kernel: [307624.885502] RIP: 0010:[<ffffffff802abf51>]  [<ffffffff802abf51>] find_get_pages+0x71/0x110
May  2 12:47:46 mainserver kernel: [307624.885502] RSP: 0018:ffff88019b40d378  EFLAGS: 00000282
May  2 12:47:46 mainserver kernel: [307624.885502] RAX: ffff88018b6fb288 RBX: ffff88019b40d3b8 RCX: 0000000000000003
May  2 12:47:46 mainserver kernel: [307624.885502] RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffffe20000ecbd40
May  2 12:47:46 mainserver kernel: [307624.885502] RBP: ffff88019b40d528 R08: ffffe20000e19848 R09: 0000000000000006
May  2 12:47:46 mainserver kernel: [307624.885502] R10: 0000000000000009 R11: 000000000055938a R12: ffffffffa04a5880
May  2 12:47:46 mainserver kernel: [307624.885502] R13: ffff8801808fb000 R14: 01ff880134872920 R15: 0000000000601b99
May  2 12:47:46 mainserver kernel: [307624.885502] FS:  00007f3c1cb506e0(0000) GS:ffff88023f06a080(0000) knlGS:0000000000000000
May  2 12:47:46 mainserver kernel: [307624.885502] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May  2 12:47:46 mainserver kernel: [307624.885502] CR2: 00007f84ac3a6b00 CR3: 000000010f15c000 CR4: 00000000000026e0
May  2 12:47:46 mainserver kernel: [307624.885502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May  2 12:47:46 mainserver kernel: [307624.885502] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May  2 12:47:46 mainserver kernel: [307624.885502]
May  2 12:47:46 mainserver kernel: [307624.885502] Call Trace:
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802abf23>] ? find_get_pages+0x43/0x110
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802b6ae4>] ? pagevec_lookup+0x24/0x30
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffffa04c906d>] ? xfs_cluster_write+0xad/0x180 [xfs]
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffffa04c95d8>] ? xfs_page_state_convert+0x498/0x760 [xfs]
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffffa04c9a01>] ? xfs_vm_writepage+0x71/0x120 [xfs]
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802b93c4>] ? pageout+0x124/0x280
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ab07a>] ? page_waitqueue+0xa/0x90
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802b99cd>] ? shrink_page_list+0x34d/0x530
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802b9d52>] ? shrink_inactive_list+0x1a2/0x4b0
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802b4934>] ? get_dirty_limits+0x14/0x2b0
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ba0db>] ? shrink_zone+0x7b/0x160
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ba24d>] ? shrink_zones+0x8d/0x150
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ba396>] ? do_try_to_free_pages+0x86/0x2e0
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ba6e7>] ? try_to_free_pages+0x67/0x70
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802b91f0>] ? isolate_pages_global+0x0/0x50
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802b29d9>] ? __alloc_pages_internal+0x239/0x520
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802d5a9d>] ? alloc_pages_current+0xad/0x110
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ac4b7>] ? __page_cache_alloc+0x67/0x80
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ad0f3>] ? __grab_cache_page+0x63/0xb0
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff80316fa9>] ? block_write_begin+0x89/0xf0
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffffa04c84ca>] ? xfs_vm_write_begin+0x2a/0x30 [xfs]
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffffa04c8090>] ? xfs_get_blocks+0x0/0x20 [xfs]
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ab7dc>] ? generic_perform_write+0xbc/0x1c0
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ad542>] ? generic_file_buffered_write+0x92/0x170
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ad542>] ? generic_file_buffered_write+0x92/0x170
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffffa04d12c3>] ? xfs_write+0x6b3/0x9b0 [xfs]
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ae241>] ? generic_file_aio_read+0xd1/0x1b0
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffffa04ccc78>] ? xfs_file_aio_write+0x58/0x60 [xfs]
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802e9999>] ? do_sync_write+0xf9/0x140
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff803a7e01>] ? __rb_erase_color+0x101/0x1d0
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff80267070>] ? autoremove_wake_function+0x0/0x40
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff803870d1>] ? aa_file_permission+0x21/0xf0
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff803871f8>] ? apparmor_file_permission+0x28/0x30
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff80361cf6>] ? security_file_permission+0x16/0x20
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ea05b>] ? vfs_write+0xcb/0x130
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff802ea1b5>] ? sys_write+0x55/0x90
May  2 12:47:46 mainserver kernel: [307624.885502]  [<ffffffff8021285a>] ? system_call_fastpath+0x16/0x1b
May  2 12:48:52 mainserver kernel: [307690.385499] BUG: soft lockup - CPU#3 stuck for 61s! [dd:2251]
May  2 12:48:52 mainserver kernel: [307690.385502] Modules linked in: i915 drm binfmt_misc rfcomm bnep sco l2cap bluetooth tun ipt_MASQUERADE iptable_nat $
May  2 12:48:52 mainserver kernel: ner snd_page_alloc ext3 jbd mbcache sr_mod cdrom sd_mod crc_t10dif sg ata_generic ata_piix pata_it8213 pata_acpi libata$
May  2 12:48:52 mainserver kernel: [307690.385502] CPU 3:
May  2 12:48:52 mainserver kernel: [307690.385502] Modules linked in: i915 drm binfmt_misc rfcomm bnep sco l2cap bluetooth tun ipt_MASQUERADE iptable_nat $
May  2 12:48:52 mainserver kernel: ner snd_page_alloc ext3 jbd mbcache sr_mod cdrom sd_mod crc_t10dif sg ata_generic ata_piix pata_it8213 pata_acpi libata$
May  2 12:48:52 mainserver kernel: [307690.385502] Pid: 2251, comm: dd Not tainted 2.6.27-11-generic #1
May  2 12:48:52 mainserver kernel: [307690.385502] RIP: 0010:[<ffffffff802abf4c>]  [<ffffffff802abf4c>] find_get_pages+0x6c/0x110
May  2 12:48:52 mainserver kernel: [307690.385502] RSP: 0018:ffff88019b40d378  EFLAGS: 00000246
May  2 12:48:52 mainserver kernel: [307690.385502] RAX: ffff88018b6fb288 RBX: ffff88019b40d3b8 RCX: 0000000000000003
May  2 12:48:52 mainserver kernel: [307690.385502] RDX: 0000000000000004 RSI: 0000000000000000 RDI: ffffe20000ecbd40
May  2 12:48:52 mainserver kernel: [307690.385502] RBP: ffff88019b40d528 R08: ffffe20000e19848 R09: 0000000000000006
May  2 12:48:52 mainserver kernel: [307690.385502] R10: 0000000000000009 R11: 000000000055938a R12: ffffffffa04a5880
May  2 12:48:52 mainserver kernel: [307690.385502] R13: ffff8801808fb000 R14: 01ff880134872920 R15: 0000000000601b99
May  2 12:48:52 mainserver kernel: [307690.385502] FS:  00007f3c1cb506e0(0000) GS:ffff88023f06a080(0000) knlGS:0000000000000000
May  2 12:48:52 mainserver kernel: [307690.385502] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
May  2 12:48:52 mainserver kernel: [307690.385502] CR2: 00007f84ac3a6b00 CR3: 000000010f15c000 CR4: 00000000000026e0
May  2 12:48:52 mainserver kernel: [307690.385502] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
May  2 12:48:52 mainserver kernel: [307690.385502] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
May  2 12:48:52 mainserver kernel: [307690.385502]
May  2 12:48:52 mainserver kernel: [307690.385502] Call Trace:
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802abf23>] ? find_get_pages+0x43/0x110
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802b6ae4>] ? pagevec_lookup+0x24/0x30
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffffa04c906d>] ? xfs_cluster_write+0xad/0x180 [xfs]
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffffa04c95d8>] ? xfs_page_state_convert+0x498/0x760 [xfs]
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffffa04c9a01>] ? xfs_vm_writepage+0x71/0x120 [xfs]
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802b93c4>] ? pageout+0x124/0x280
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802ab07a>] ? page_waitqueue+0xa/0x90
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802b99cd>] ? shrink_page_list+0x34d/0x530
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802b9d52>] ? shrink_inactive_list+0x1a2/0x4b0
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802b4934>] ? get_dirty_limits+0x14/0x2b0
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802ba0db>] ? shrink_zone+0x7b/0x160
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802ba24d>] ? shrink_zones+0x8d/0x150
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802ba396>] ? do_try_to_free_pages+0x86/0x2e0
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802ba6e7>] ? try_to_free_pages+0x67/0x70
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802b91f0>] ? isolate_pages_global+0x0/0x50
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802b29d9>] ? __alloc_pages_internal+0x239/0x520
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802d5a9d>] ? alloc_pages_current+0xad/0x110
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802ac4b7>] ? __page_cache_alloc+0x67/0x80
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802ad0f3>] ? __grab_cache_page+0x63/0xb0
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff80316fa9>] ? block_write_begin+0x89/0xf0
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffffa04c84ca>] ? xfs_vm_write_begin+0x2a/0x30 [xfs]
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffffa04c8090>] ? xfs_get_blocks+0x0/0x20 [xfs]
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802ab7dc>] ? generic_perform_write+0xbc/0x1c0
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802ad542>] ? generic_file_buffered_write+0x92/0x170
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffffa04d12c3>] ? xfs_write+0x6b3/0x9b0 [xfs]
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802ae241>] ? generic_file_aio_read+0xd1/0x1b0
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffffa04ccc78>] ? xfs_file_aio_write+0x58/0x60 [xfs]
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff802e9999>] ? do_sync_write+0xf9/0x140
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff803a7e01>] ? __rb_erase_color+0x101/0x1d0
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff80267070>] ? autoremove_wake_function+0x0/0x40
May  2 12:48:52 mainserver kernel: [307690.385502]  [<ffffffff803870d1>] ? aa_file_permission+0x21/0xf0
May  2 12:49:57 mainserver kernel: [307755.885502] Call Trace:
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802abf23>] ? find_get_pages+0x43/0x110
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802b6ae4>] ? pagevec_lookup+0x24/0x30
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffffa04c906d>] ? xfs_cluster_write+0xad/0x180 [xfs]
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffffa04c95d8>] ? xfs_page_state_convert+0x498/0x760 [xfs]
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffffa04c9a01>] ? xfs_vm_writepage+0x71/0x120 [xfs]
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802b93c4>] ? pageout+0x124/0x280
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ab07a>] ? page_waitqueue+0xa/0x90
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802b99cd>] ? shrink_page_list+0x34d/0x530
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802b9d52>] ? shrink_inactive_list+0x1a2/0x4b0
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802b4934>] ? get_dirty_limits+0x14/0x2b0
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ba0db>] ? shrink_zone+0x7b/0x160
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ba24d>] ? shrink_zones+0x8d/0x150
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ba396>] ? do_try_to_free_pages+0x86/0x2e0
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ba6e7>] ? try_to_free_pages+0x67/0x70
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802b91f0>] ? isolate_pages_global+0x0/0x50
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802b29d9>] ? __alloc_pages_internal+0x239/0x520
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802d5a9d>] ? alloc_pages_current+0xad/0x110
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ac4b7>] ? __page_cache_alloc+0x67/0x80
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ad0f3>] ? __grab_cache_page+0x63/0xb0
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff80316fa9>] ? block_write_begin+0x89/0xf0
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffffa04c84ca>] ? xfs_vm_write_begin+0x2a/0x30 [xfs]
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffffa04c8090>] ? xfs_get_blocks+0x0/0x20 [xfs]
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ab7dc>] ? generic_perform_write+0xbc/0x1c0
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ad542>] ? generic_file_buffered_write+0x92/0x170
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffffa04d12c3>] ? xfs_write+0x6b3/0x9b0 [xfs]
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ae241>] ? generic_file_aio_read+0xd1/0x1b0
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffffa04ccc78>] ? xfs_file_aio_write+0x58/0x60 [xfs]
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802e9999>] ? do_sync_write+0xf9/0x140
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff803a7e01>] ? __rb_erase_color+0x101/0x1d0
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff80267070>] ? autoremove_wake_function+0x0/0x40
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff803870d1>] ? aa_file_permission+0x21/0xf0
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff803871f8>] ? apparmor_file_permission+0x28/0x30
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff80361cf6>] ? security_file_permission+0x16/0x20
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ea05b>] ? vfs_write+0xcb/0x130
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff802ea1b5>] ? sys_write+0x55/0x90
May  2 12:49:57 mainserver kernel: [307755.885502]  [<ffffffff8021285a>] ? system_call_fastpath+0x16/0x1b