Jun 14 12:07:15 jo kernel: [595546.669221] lvm invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Jun 14 12:07:15 jo kernel: [595546.669226] Pid: 18021, comm: lvm Not tainted 2.6.28-19-server #61-Ubuntu
Jun 14 12:07:15 jo kernel: [595546.669228] Call Trace:
Jun 14 12:07:15 jo kernel: [595546.669237]  [<ffffffff802b3845>] oom_kill_process+0x95/0x240
Jun 14 12:07:15 jo kernel: [595546.669241]  [<ffffffff802b3fff>] ? select_bad_process+0xef/0x130
Jun 14 12:07:15 jo kernel: [595546.669244]  [<ffffffff802b40f4>] out_of_memory+0xb4/0x150
Jun 14 12:07:15 jo kernel: [595546.669248]  [<ffffffff802b6c69>] __alloc_pages_internal+0x4a9/0x4f0
Jun 14 12:07:15 jo kernel: [595546.669251]  [<ffffffff802b9a9a>] __do_page_cache_readahead+0xda/0x210
Jun 14 12:07:15 jo kernel: [595546.669255]  [<ffffffff802b9c2e>] do_page_cache_readahead+0x5e/0x90
Jun 14 12:07:15 jo kernel: [595546.669258]  [<ffffffff802b203a>] filemap_fault+0x34a/0x430
Jun 14 12:07:15 jo kernel: [595546.669262]  [<ffffffff802c6030>] __do_fault+0x50/0x520
Jun 14 12:07:15 jo kernel: [595546.669267]  [<ffffffff8069ae19>] ? _spin_lock+0x9/0x10
Jun 14 12:07:15 jo upsmon[3099]: upsmon parent: read
Jun 14 12:07:15 jo kernel: [595546.669270]  [<ffffffff802fb624>] ? __d_lookup+0x114/0x150
Jun 14 12:07:15 jo kernel: [595546.669275]  [<ffffffff8023dd90>] ? update_curr+0xf0/0x170
Jun 14 12:07:15 jo kernel: [595546.669278]  [<ffffffff802c71b9>] handle_mm_fault+0x1e9/0x470
Jun 14 12:07:15 jo kernel: [595546.669282]  [<ffffffff80210579>] ? __switch_to+0x109/0x490
Jun 14 12:07:15 jo kernel: [595546.669285]  [<ffffffff8069dc37>] do_page_fault+0x307/0x780
Jun 14 12:07:15 jo kernel: [595546.669289]  [<ffffffff8026c4ec>] ? lock_hrtimer_base+0x2c/0x60
Jun 14 12:07:15 jo kernel: [595546.669292]  [<ffffffff8026c5af>] ? hrtimer_try_to_cancel+0x3f/0x90
Jun 14 12:07:15 jo kernel: [595546.669295]  [<ffffffff8026c61a>] ? hrtimer_cancel+0x1a/0x30
Jun 14 12:07:15 jo kernel: [595546.669298]  [<ffffffff80699e90>] ? do_nanosleep+0x40/0xc0
Jun 14 12:07:15 jo kernel: [595546.669301]  [<ffffffff8026cc59>] ? hrtimer_nanosleep+0xa9/0x130
Jun 14 12:07:15 jo kernel: [595546.669304]  [<ffffffff8026bf00>] ? hrtimer_wakeup+0x0/0x30
Jun 14 12:07:15 jo kernel: [595546.669307]  [<ffffffff80699ece>] ? do_nanosleep+0x7e/0xc0
Jun 14 12:07:15 jo kernel: [595546.669310]  [<ffffffff8069b21a>] error_exit+0x0/0x70
Jun 14 12:07:15 jo kernel: [595546.669312] Mem-Info:
Jun 14 12:07:15 jo kernel: [595546.669314] DMA per-cpu:
Jun 14 12:07:15 jo kernel: [595546.669316] CPU    0: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595546.669318] CPU    1: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595546.669320] DMA32 per-cpu:
Jun 14 12:07:15 jo kernel: [595546.669321] CPU    0: hi:  186, btch:  31 usd: 161
Jun 14 12:07:15 jo kernel: [595546.669323] CPU    1: hi:  186, btch:  31 usd: 155
Jun 14 12:07:15 jo kernel: [595546.669325] Normal per-cpu:
Jun 14 12:07:15 jo kernel: [595546.669326] CPU    0: hi:  186, btch:  31 usd: 165
Jun 14 12:07:15 jo kernel: [595546.669328] CPU    1: hi:  186, btch:  31 usd: 183
Jun 14 12:07:15 jo kernel: [595546.669331] Active_anon:707190 active_file:228 inactive_anon:235733
Jun 14 12:07:15 jo kernel: [595546.669332]  inactive_file:164 unevictable:0 dirty:0 writeback:0 unstable:0
Jun 14 12:07:15 jo kernel: [595546.669333]  free:5608 slab:5907 mapped:1 pagetables:5999 bounce:0
Jun 14 12:07:15 jo kernel: [595546.669337] DMA free:6700kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:5560kB pages_scanned:0 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595546.669340] lowmem_reserve[]: 0 1967 3924 3924
Jun 14 12:07:15 jo kernel: [595546.669345] DMA32 free:11780kB min:4016kB low:5020kB high:6024kB active_anon:1364084kB inactive_anon:454540kB active_file:16kB inactive_file:0kB unevictable:0kB present:2015008kB pages_scanned:2871454 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595546.669348] lowmem_reserve[]: 0 0 1956 1956
Jun 14 12:07:15 jo kernel: [595546.669354] Normal free:3952kB min:3992kB low:4988kB high:5988kB active_anon:1464676kB inactive_anon:488392kB active_file:896kB inactive_file:656kB unevictable:0kB present:2003840kB pages_scanned:11486476 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595546.669357] lowmem_reserve[]: 0 0 0 0
Jun 14 12:07:15 jo kernel: [595546.669360] DMA: 5*4kB 3*8kB 6*16kB 5*32kB 4*64kB 4*128kB 2*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB = 6700kB
Jun 14 12:07:15 jo kernel: [595546.669370] DMA32: 2143*4kB 1*8kB 10*16kB 11*32kB 14*64kB 6*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 11780kB
Jun 14 12:07:15 jo kernel: [595546.669379] Normal: 118*4kB 1*8kB 7*16kB 7*32kB 9*64kB 4*128kB 4*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3952kB
Jun 14 12:07:15 jo kernel: [595546.669388] 405 total pagecache pages
Jun 14 12:07:15 jo kernel: [595546.669389] 0 pages in swap cache
Jun 14 12:07:15 jo kernel: [595546.669391] Swap cache stats: add 978835, delete 978835, find 30294/30639
Jun 14 12:07:15 jo kernel: [595546.669393] Free swap  = 0kB
Jun 14 12:07:15 jo kernel: [595546.669394] Total swap = 3903752kB
Jun 14 12:07:15 jo kernel: [595546.691384] 1556464 pages RAM
Jun 14 12:07:15 jo kernel: [595546.691386] 583861 pages reserved
Jun 14 12:07:15 jo kernel: [595546.691388] 3466 pages shared
Jun 14 12:07:15 jo kernel: [595546.691389] 962551 pages non-shared
Jun 14 12:07:15 jo kernel: [595546.691393] Out of memory: kill process 2930 (exim4) score 2744 or a child
Jun 14 12:07:15 jo kernel: [595546.696633] Killed process 2930 (exim4)
Jun 14 12:07:15 jo kernel: [595548.077060] lvm invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Jun 14 12:07:15 jo kernel: [595548.077064] Pid: 22573, comm: lvm Not tainted 2.6.28-19-server #61-Ubuntu
Jun 14 12:07:15 jo kernel: [595548.077066] Call Trace:
Jun 14 12:07:15 jo kernel: [595548.077075]  [<ffffffff802b3845>] oom_kill_process+0x95/0x240
Jun 14 12:07:15 jo kernel: [595548.077079]  [<ffffffff802b3fff>] ? select_bad_process+0xef/0x130
Jun 14 12:07:15 jo kernel: [595548.077082]  [<ffffffff802b40f4>] out_of_memory+0xb4/0x150
Jun 14 12:07:15 jo kernel: [595548.077086]  [<ffffffff802b6c69>] __alloc_pages_internal+0x4a9/0x4f0
Jun 14 12:07:15 jo kernel: [595548.077089]  [<ffffffff802b9a9a>] __do_page_cache_readahead+0xda/0x210
Jun 14 12:07:15 jo kernel: [595548.077093]  [<ffffffff802b9c2e>] do_page_cache_readahead+0x5e/0x90
Jun 14 12:07:15 jo kernel: [595548.077096]  [<ffffffff802b203a>] filemap_fault+0x34a/0x430
Jun 14 12:07:15 jo kernel: [595548.077100]  [<ffffffff802c6030>] __do_fault+0x50/0x520
Jun 14 12:07:15 jo kernel: [595548.077104]  [<ffffffff80242462>] ? enqueue_entity+0x122/0x2b0
Jun 14 12:07:15 jo kernel: [595548.077108]  [<ffffffff8023dd90>] ? update_curr+0xf0/0x170
Jun 14 12:07:15 jo kernel: [595548.077111]  [<ffffffff802c71b9>] handle_mm_fault+0x1e9/0x470
Jun 14 12:07:15 jo kernel: [595548.077115]  [<ffffffff80210579>] ? __switch_to+0x109/0x490
Jun 14 12:07:15 jo kernel: [595548.077119]  [<ffffffff8069dc37>] do_page_fault+0x307/0x780
Jun 14 12:07:15 jo kernel: [595548.077123]  [<ffffffff8026c4ec>] ? lock_hrtimer_base+0x2c/0x60
Jun 14 12:07:15 jo kernel: [595548.077126]  [<ffffffff8026c5af>] ? hrtimer_try_to_cancel+0x3f/0x90
Jun 14 12:07:15 jo kernel: [595548.077129]  [<ffffffff8026c61a>] ? hrtimer_cancel+0x1a/0x30
Jun 14 12:07:15 jo kernel: [595548.077132]  [<ffffffff80699e90>] ? do_nanosleep+0x40/0xc0
Jun 14 12:07:15 jo kernel: [595548.077135]  [<ffffffff8026cc59>] ? hrtimer_nanosleep+0xa9/0x130
Jun 14 12:07:15 jo kernel: [595548.077138]  [<ffffffff8026bf00>] ? hrtimer_wakeup+0x0/0x30
Jun 14 12:07:15 jo kernel: [595548.077141]  [<ffffffff80699ece>] ? do_nanosleep+0x7e/0xc0
Jun 14 12:07:15 jo kernel: [595548.077144]  [<ffffffff8069b21a>] error_exit+0x0/0x70
Jun 14 12:07:15 jo kernel: [595548.077146] Mem-Info:
Jun 14 12:07:15 jo kernel: [595548.077148] DMA per-cpu:
Jun 14 12:07:15 jo kernel: [595548.077150] CPU    0: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595548.077152] CPU    1: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595548.077153] DMA32 per-cpu:
Jun 14 12:07:15 jo kernel: [595548.077155] CPU    0: hi:  186, btch:  31 usd: 161
Jun 14 12:07:15 jo kernel: [595548.077157] CPU    1: hi:  186, btch:  31 usd: 155
Jun 14 12:07:15 jo kernel: [595548.077159] Normal per-cpu:
Jun 14 12:07:15 jo kernel: [595548.077160] CPU    0: hi:  186, btch:  31 usd: 159
Jun 14 12:07:15 jo kernel: [595548.077162] CPU    1: hi:  186, btch:  31 usd: 177
Jun 14 12:07:15 jo kernel: [595548.077165] Active_anon:706805 active_file:177 inactive_anon:235609
Jun 14 12:07:15 jo kernel: [595548.077166]  inactive_file:202 unevictable:0 dirty:0 writeback:0 unstable:0
Jun 14 12:07:15 jo kernel: [595548.077167]  free:5618 slab:5904 mapped:2 pagetables:5974 bounce:0
Jun 14 12:07:15 jo kernel: [595548.077171] DMA free:6700kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:5560kB pages_scanned:0 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595548.077173] lowmem_reserve[]: 0 1967 3924 3924
Jun 14 12:07:15 jo kernel: [595548.077179] DMA32 free:11780kB min:4016kB low:5020kB high:6024kB active_anon:1363956kB inactive_anon:454684kB active_file:0kB inactive_file:16kB unevictable:0kB present:2015008kB pages_scanned:3608062 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595548.077182] lowmem_reserve[]: 0 0 1956 1956
Jun 14 12:07:15 jo kernel: [595548.077188] Normal free:3992kB min:3992kB low:4988kB high:5988kB active_anon:1463140kB inactive_anon:487752kB active_file:708kB inactive_file:792kB unevictable:0kB present:2003840kB pages_scanned:1421740 all_unreclaimable? no
Jun 14 12:07:15 jo kernel: [595548.077191] lowmem_reserve[]: 0 0 0 0
Jun 14 12:07:15 jo kernel: [595548.077194] DMA: 5*4kB 3*8kB 6*16kB 5*32kB 4*64kB 4*128kB 2*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB = 6700kB
Jun 14 12:07:15 jo kernel: [595548.077204] DMA32: 2143*4kB 1*8kB 10*16kB 11*32kB 14*64kB 6*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 11780kB
Jun 14 12:07:15 jo kernel: [595548.077213] Normal: 118*4kB 1*8kB 7*16kB 7*32kB 9*64kB 4*128kB 4*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3952kB
Jun 14 12:07:15 jo kernel: [595548.077221] 406 total pagecache pages
Jun 14 12:07:15 jo kernel: [595548.077223] 0 pages in swap cache
Jun 14 12:07:15 jo kernel: [595548.077225] Swap cache stats: add 978960, delete 978960, find 30295/30644
Jun 14 12:07:15 jo kernel: [595548.077227] Free swap  = 0kB
Jun 14 12:07:15 jo kernel: [595548.077228] Total swap = 3903752kB
Jun 14 12:07:15 jo kernel: [595548.098981] 1556464 pages RAM
Jun 14 12:07:15 jo kernel: [595548.098984] 583861 pages reserved
Jun 14 12:07:15 jo kernel: [595548.098985] 3924 pages shared
Jun 14 12:07:15 jo kernel: [595548.098986] 962105 pages non-shared
Jun 14 12:07:15 jo kernel: [595548.098989] Out of memory: kill process 28389 (rsync) score 1514 or a child
Jun 14 12:07:15 jo kernel: [595548.104167] Killed process 28389 (rsync)
Jun 14 12:07:15 jo kernel: [595553.461573] lvm invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Jun 14 12:07:15 jo kernel: [595553.461578] Pid: 24741, comm: lvm Not tainted 2.6.28-19-server #61-Ubuntu
Jun 14 12:07:15 jo kernel: [595553.461580] Call Trace:
Jun 14 12:07:15 jo kernel: [595553.461589]  [<ffffffff802b3845>] oom_kill_process+0x95/0x240
Jun 14 12:07:15 jo kernel: [595553.461593]  [<ffffffff802b3fff>] ? select_bad_process+0xef/0x130
Jun 14 12:07:15 jo kernel: [595553.461596]  [<ffffffff802b40f4>] out_of_memory+0xb4/0x150
Jun 14 12:07:15 jo kernel: [595553.461599]  [<ffffffff802b6c69>] __alloc_pages_internal+0x4a9/0x4f0
Jun 14 12:07:15 jo kernel: [595553.461603]  [<ffffffff802b9a9a>] __do_page_cache_readahead+0xda/0x210
Jun 14 12:07:15 jo kernel: [595553.461607]  [<ffffffff802b9c2e>] do_page_cache_readahead+0x5e/0x90
Jun 14 12:07:15 jo kernel: [595553.461610]  [<ffffffff802b203a>] filemap_fault+0x34a/0x430
Jun 14 12:07:15 jo kernel: [595553.461614]  [<ffffffff802c6030>] __do_fault+0x50/0x520
Jun 14 12:07:15 jo kernel: [595553.461619]  [<ffffffff8069ae19>] ? _spin_lock+0x9/0x10
Jun 14 12:07:15 jo kernel: [595553.461622]  [<ffffffff802fb624>] ? __d_lookup+0x114/0x150
Jun 14 12:07:15 jo kernel: [595553.461626]  [<ffffffff8023dd90>] ? update_curr+0xf0/0x170
Jun 14 12:07:15 jo kernel: [595553.461630]  [<ffffffff802c71b9>] handle_mm_fault+0x1e9/0x470
Jun 14 12:07:15 jo kernel: [595553.461634]  [<ffffffff80210579>] ? __switch_to+0x109/0x490
Jun 14 12:07:15 jo kernel: [595553.461637]  [<ffffffff8069dc37>] do_page_fault+0x307/0x780
Jun 14 12:07:15 jo kernel: [595553.461641]  [<ffffffff8026c4ec>] ? lock_hrtimer_base+0x2c/0x60
Jun 14 12:07:15 jo kernel: [595553.461644]  [<ffffffff8026c5af>] ? hrtimer_try_to_cancel+0x3f/0x90
Jun 14 12:07:15 jo kernel: [595553.461647]  [<ffffffff8026c61a>] ? hrtimer_cancel+0x1a/0x30
Jun 14 12:07:15 jo kernel: [595553.461650]  [<ffffffff80699e90>] ? do_nanosleep+0x40/0xc0
Jun 14 12:07:15 jo kernel: [595553.461653]  [<ffffffff8026cc59>] ? hrtimer_nanosleep+0xa9/0x130
Jun 14 12:07:15 jo kernel: [595553.461656]  [<ffffffff8026bf00>] ? hrtimer_wakeup+0x0/0x30
Jun 14 12:07:15 jo kernel: [595553.461659]  [<ffffffff80699ece>] ? do_nanosleep+0x7e/0xc0
Jun 14 12:07:15 jo kernel: [595553.461662]  [<ffffffff8069b21a>] error_exit+0x0/0x70
Jun 14 12:07:15 jo kernel: [595553.461664] Mem-Info:
Jun 14 12:07:15 jo kernel: [595553.461666] DMA per-cpu:
Jun 14 12:07:15 jo kernel: [595553.461668] CPU    0: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595553.461670] CPU    1: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595553.461672] DMA32 per-cpu:
Jun 14 12:07:15 jo kernel: [595553.461674] CPU    0: hi:  186, btch:  31 usd: 163
Jun 14 12:07:15 jo kernel: [595553.461676] CPU    1: hi:  186, btch:  31 usd:  39
Jun 14 12:07:15 jo kernel: [595553.461677] Normal per-cpu:
Jun 14 12:07:15 jo kernel: [595553.461679] CPU    0: hi:  186, btch:  31 usd: 164
Jun 14 12:07:15 jo kernel: [595553.461680] CPU    1: hi:  186, btch:  31 usd:  37
Jun 14 12:07:15 jo kernel: [595553.461684] Active_anon:706780 active_file:217 inactive_anon:235448
Jun 14 12:07:15 jo kernel: [595553.461685]  inactive_file:228 unevictable:0 dirty:0 writeback:0 unstable:0
Jun 14 12:07:15 jo kernel: [595553.461686]  free:6252 slab:5904 mapped:73 pagetables:5923 bounce:0
Jun 14 12:07:15 jo kernel: [595553.461689] DMA free:6700kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:5560kB pages_scanned:0 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595553.461692] lowmem_reserve[]: 0 1967 3924 3924
Jun 14 12:07:15 jo kernel: [595553.461698] DMA32 free:13156kB min:4016kB low:5020kB high:6024kB active_anon:1361728kB inactive_anon:453628kB active_file:0kB inactive_file:64kB unevictable:0kB present:2015008kB pages_scanned:115634 all_unreclaimable? no
Jun 14 12:07:15 jo kernel: [595553.461701] lowmem_reserve[]: 0 0 1956 1956
Jun 14 12:07:15 jo kernel: [595553.461706] Normal free:5152kB min:3992kB low:4988kB high:5988kB active_anon:1465516kB inactive_anon:488164kB active_file:932kB inactive_file:848kB unevictable:0kB present:2003840kB pages_scanned:82185 all_unreclaimable? no
Jun 14 12:07:15 jo kernel: [595553.461709] lowmem_reserve[]: 0 0 0 0
Jun 14 12:07:15 jo kernel: [595553.461713] DMA: 5*4kB 3*8kB 6*16kB 5*32kB 4*64kB 4*128kB 2*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB = 6700kB
Jun 14 12:07:15 jo kernel: [595553.461722] DMA32: 2400*4kB 36*8kB 14*16kB 11*32kB 14*64kB 6*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 13152kB
Jun 14 12:07:15 jo kernel: [595553.461731] Normal: 390*4kB 21*8kB 7*16kB 7*32kB 9*64kB 4*128kB 4*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 5200kB
Jun 14 12:07:15 jo kernel: [595553.461741] 465 total pagecache pages
Jun 14 12:07:15 jo kernel: [595553.461742] 0 pages in swap cache
Jun 14 12:07:15 jo kernel: [595553.461744] Swap cache stats: add 978960, delete 978960, find 30295/30644
Jun 14 12:07:15 jo kernel: [595553.461746] Free swap  = 0kB
Jun 14 12:07:15 jo kernel: [595553.461748] Total swap = 3903752kB
Jun 14 12:07:15 jo kernel: [595553.485940] 1556464 pages RAM
Jun 14 12:07:15 jo kernel: [595553.485943] 583861 pages reserved
Jun 14 12:07:15 jo kernel: [595553.485944] 3804 pages shared
Jun 14 12:07:15 jo kernel: [595553.485945] 961821 pages non-shared
Jun 14 12:07:15 jo kernel: [595553.485948] Out of memory: kill process 12689 (sshd) score 1245 or a child
Jun 14 12:07:15 jo kernel: [595553.491428] Killed process 12691 (bash)
Jun 14 12:07:15 jo kernel: [595556.231338] lvm invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Jun 14 12:07:15 jo kernel: [595556.231343] Pid: 16738, comm: lvm Not tainted 2.6.28-19-server #61-Ubuntu
Jun 14 12:07:15 jo kernel: [595556.231345] Call Trace:
Jun 14 12:07:15 jo kernel: [595556.231353]  [<ffffffff802b3845>] oom_kill_process+0x95/0x240
Jun 14 12:07:15 jo kernel: [595556.231357]  [<ffffffff802b3fff>] ? select_bad_process+0xef/0x130
Jun 14 12:07:15 jo kernel: [595556.231360]  [<ffffffff802b40f4>] out_of_memory+0xb4/0x150
Jun 14 12:07:15 jo kernel: [595556.231364]  [<ffffffff802b6c69>] __alloc_pages_internal+0x4a9/0x4f0
Jun 14 12:07:15 jo kernel: [595556.231367]  [<ffffffff802b9a9a>] __do_page_cache_readahead+0xda/0x210
Jun 14 12:07:15 jo kernel: [595556.231371]  [<ffffffff802b9c2e>] do_page_cache_readahead+0x5e/0x90
Jun 14 12:07:15 jo kernel: [595556.231374]  [<ffffffff802b203a>] filemap_fault+0x34a/0x430
Jun 14 12:07:15 jo kernel: [595556.231378]  [<ffffffff802c6030>] __do_fault+0x50/0x520
Jun 14 12:07:15 jo kernel: [595556.231383]  [<ffffffff8069ae19>] ? _spin_lock+0x9/0x10
Jun 14 12:07:15 jo kernel: [595556.231387]  [<ffffffff802fb624>] ? __d_lookup+0x114/0x150
Jun 14 12:07:15 jo kernel: [595556.231391]  [<ffffffff8023dd90>] ? update_curr+0xf0/0x170
Jun 14 12:07:15 jo kernel: [595556.231394]  [<ffffffff802c71b9>] handle_mm_fault+0x1e9/0x470
Jun 14 12:07:15 jo kernel: [595556.231399]  [<ffffffff80210620>] ? __switch_to+0x1b0/0x490
Jun 14 12:07:15 jo kernel: [595556.231402]  [<ffffffff8069dc37>] do_page_fault+0x307/0x780
Jun 14 12:07:15 jo kernel: [595556.231405]  [<ffffffff8026c4ec>] ? lock_hrtimer_base+0x2c/0x60
Jun 14 12:07:15 jo kernel: [595556.231408]  [<ffffffff8026c5af>] ? hrtimer_try_to_cancel+0x3f/0x90
Jun 14 12:07:15 jo kernel: [595556.231411]  [<ffffffff8026c61a>] ? hrtimer_cancel+0x1a/0x30
Jun 14 12:07:15 jo kernel: [595556.231414]  [<ffffffff80699e90>] ? do_nanosleep+0x40/0xc0
Jun 14 12:07:15 jo kernel: [595556.231417]  [<ffffffff8026cc59>] ? hrtimer_nanosleep+0xa9/0x130
Jun 14 12:07:15 jo kernel: [595556.231420]  [<ffffffff8026bf00>] ? hrtimer_wakeup+0x0/0x30
Jun 14 12:07:15 jo kernel: [595556.231423]  [<ffffffff80699ece>] ? do_nanosleep+0x7e/0xc0
Jun 14 12:07:15 jo kernel: [595556.231426]  [<ffffffff8069b21a>] error_exit+0x0/0x70
Jun 14 12:07:15 jo kernel: [595556.231428] Mem-Info:
Jun 14 12:07:15 jo kernel: [595556.231431] DMA per-cpu:
Jun 14 12:07:15 jo kernel: [595556.231433] CPU    0: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595556.231435] CPU    1: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595556.231436] DMA32 per-cpu:
Jun 14 12:07:15 jo kernel: [595556.231438] CPU    0: hi:  186, btch:  31 usd: 125
Jun 14 12:07:15 jo kernel: [595556.231440] CPU    1: hi:  186, btch:  31 usd: 177
Jun 14 12:07:15 jo kernel: [595556.231441] Normal per-cpu:
Jun 14 12:07:15 jo kernel: [595556.231443] CPU    0: hi:  186, btch:  31 usd: 170
Jun 14 12:07:15 jo kernel: [595556.231445] CPU    1: hi:  186, btch:  31 usd: 176
Jun 14 12:07:15 jo kernel: [595556.231448] Active_anon:706959 active_file:345 inactive_anon:235474
Jun 14 12:07:15 jo kernel: [595556.231449]  inactive_file:518 unevictable:0 dirty:0 writeback:0 unstable:0
Jun 14 12:07:15 jo kernel: [595556.231450]  free:5841 slab:5901 mapped:97 pagetables:5871 bounce:0
Jun 14 12:07:15 jo kernel: [595556.231453] DMA free:6700kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:5560kB pages_scanned:0 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595556.231456] lowmem_reserve[]: 0 1967 3924 3924
Jun 14 12:07:15 jo kernel: [595556.231462] DMA32 free:12672kB min:4016kB low:5020kB high:6024kB active_anon:1360484kB inactive_anon:453156kB active_file:48kB inactive_file:448kB unevictable:0kB present:2015008kB pages_scanned:122144 all_unreclaimable? no
Jun 14 12:07:15 jo kernel: [595556.231465] lowmem_reserve[]: 0 0 1956 1956
Jun 14 12:07:15 jo kernel: [595556.231470] Normal free:3992kB min:3992kB low:4988kB high:5988kB active_anon:1467352kB inactive_anon:488740kB active_file:1332kB inactive_file:1624kB unevictable:0kB present:2003840kB pages_scanned:307744 all_unreclaimable? no
Jun 14 12:07:15 jo kernel: [595556.231473] lowmem_reserve[]: 0 0 0 0
Jun 14 12:07:15 jo kernel: [595556.231477] DMA: 5*4kB 3*8kB 6*16kB 5*32kB 4*64kB 4*128kB 2*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB = 6700kB
Jun 14 12:07:15 jo kernel: [595556.231486] DMA32: 2230*4kB 33*8kB 27*16kB 11*32kB 14*64kB 6*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 12656kB
Jun 14 12:07:15 jo kernel: [595556.231495] Normal: 116*4kB 5*8kB 8*16kB 7*32kB 9*64kB 4*128kB 4*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3992kB
Jun 14 12:07:15 jo kernel: [595556.231504] 883 total pagecache pages
Jun 14 12:07:15 jo kernel: [595556.231506] 0 pages in swap cache
Jun 14 12:07:15 jo kernel: [595556.231508] Swap cache stats: add 979574, delete 979574, find 30358/30745
Jun 14 12:07:15 jo kernel: [595556.231509] Free swap  = 0kB
Jun 14 12:07:15 jo kernel: [595556.231511] Total swap = 3903752kB
Jun 14 12:07:15 jo kernel: [595556.255881] 1556464 pages RAM
Jun 14 12:07:15 jo kernel: [595556.255883] 583861 pages reserved
Jun 14 12:07:15 jo kernel: [595556.255884] 4120 pages shared
Jun 14 12:07:15 jo kernel: [595556.255885] 962294 pages non-shared
Jun 14 12:07:15 jo kernel: [595556.255888] Out of memory: kill process 3101 (upsmon) score 1166 or a child
Jun 14 12:07:15 jo kernel: [595556.261591] Killed process 3101 (upsmon)
Jun 14 12:07:15 jo kernel: [595576.044103] lvm invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Jun 14 12:07:15 jo kernel: [595576.044108] Pid: 24937, comm: lvm Not tainted 2.6.28-19-server #61-Ubuntu
Jun 14 12:07:15 jo kernel: [595576.044110] Call Trace:
Jun 14 12:07:15 jo kernel: [595576.044119]  [<ffffffff802b3845>] oom_kill_process+0x95/0x240
Jun 14 12:07:15 jo kernel: [595576.044123]  [<ffffffff802b3fff>] ? select_bad_process+0xef/0x130
Jun 14 12:07:15 jo kernel: [595576.044126]  [<ffffffff802b40f4>] out_of_memory+0xb4/0x150
Jun 14 12:07:15 jo kernel: [595576.044130]  [<ffffffff802b6c69>] __alloc_pages_internal+0x4a9/0x4f0
Jun 14 12:07:15 jo kernel: [595576.044134]  [<ffffffff802b9a9a>] __do_page_cache_readahead+0xda/0x210
Jun 14 12:07:15 jo kernel: [595576.044137]  [<ffffffff802b9c2e>] do_page_cache_readahead+0x5e/0x90
Jun 14 12:07:15 jo kernel: [595576.044140]  [<ffffffff802b203a>] filemap_fault+0x34a/0x430
Jun 14 12:07:15 jo kernel: [595576.044144]  [<ffffffff802c6030>] __do_fault+0x50/0x520
Jun 14 12:07:15 jo kernel: [595576.044149]  [<ffffffff8069ae19>] ? _spin_lock+0x9/0x10
Jun 14 12:07:15 jo kernel: [595576.044153]  [<ffffffff802fb624>] ? __d_lookup+0x114/0x150
Jun 14 12:07:15 jo kernel: [595576.044157]  [<ffffffff8023dd90>] ? update_curr+0xf0/0x170
Jun 14 12:07:15 jo kernel: [595576.044160]  [<ffffffff802c71b9>] handle_mm_fault+0x1e9/0x470
Jun 14 12:07:15 jo kernel: [595576.044165]  [<ffffffff80210579>] ? __switch_to+0x109/0x490
Jun 14 12:07:15 jo kernel: [595576.044168]  [<ffffffff8069dc37>] do_page_fault+0x307/0x780
Jun 14 12:07:15 jo kernel: [595576.044171]  [<ffffffff8026c4ec>] ? lock_hrtimer_base+0x2c/0x60
Jun 14 12:07:15 jo kernel: [595576.044174]  [<ffffffff8026c5af>] ? hrtimer_try_to_cancel+0x3f/0x90
Jun 14 12:07:15 jo kernel: [595576.044177]  [<ffffffff8026c61a>] ? hrtimer_cancel+0x1a/0x30
Jun 14 12:07:15 jo kernel: [595576.044180]  [<ffffffff80699e90>] ? do_nanosleep+0x40/0xc0
Jun 14 12:07:15 jo kernel: [595576.044183]  [<ffffffff8026cc59>] ? hrtimer_nanosleep+0xa9/0x130
Jun 14 12:07:15 jo kernel: [595576.044186]  [<ffffffff8026bf00>] ? hrtimer_wakeup+0x0/0x30
Jun 14 12:07:15 jo kernel: [595576.044189]  [<ffffffff80699ece>] ? do_nanosleep+0x7e/0xc0
Jun 14 12:07:15 jo kernel: [595576.044192]  [<ffffffff8069b21a>] error_exit+0x0/0x70
Jun 14 12:07:15 jo kernel: [595576.044194] Mem-Info:
Jun 14 12:07:15 jo kernel: [595576.044196] DMA per-cpu:
Jun 14 12:07:15 jo kernel: [595576.044198] CPU    0: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595576.044199] CPU    1: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595576.044201] DMA32 per-cpu:
Jun 14 12:07:15 jo kernel: [595576.044203] CPU    0: hi:  186, btch:  31 usd: 163
Jun 14 12:07:15 jo kernel: [595576.044205] CPU    1: hi:  186, btch:  31 usd: 185
Jun 14 12:07:15 jo kernel: [595576.044206] Normal per-cpu:
Jun 14 12:07:15 jo kernel: [595576.044208] CPU    0: hi:  186, btch:  31 usd: 150
Jun 14 12:07:15 jo kernel: [595576.044209] CPU    1: hi:  186, btch:  31 usd: 156
Jun 14 12:07:15 jo kernel: [595576.044213] Active_anon:709400 active_file:224 inactive_anon:235350
Jun 14 12:07:15 jo kernel: [595576.044214]  inactive_file:233 unevictable:0 dirty:0 writeback:0 unstable:0
Jun 14 12:07:15 jo kernel: [595576.044215]  free:5641 slab:5849 mapped:1 pagetables:5881 bounce:0
Jun 14 12:07:15 jo kernel: [595576.044218] DMA free:6700kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:5560kB pages_scanned:0 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595576.044221] lowmem_reserve[]: 0 1967 3924 3924
Jun 14 12:07:15 jo kernel: [595576.044226] DMA32 free:11736kB min:4016kB low:5020kB high:6024kB active_anon:1366396kB inactive_anon:451008kB active_file:0kB inactive_file:24kB unevictable:0kB present:2015008kB pages_scanned:4250480 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595576.044230] lowmem_reserve[]: 0 0 1956 1956
Jun 14 12:07:15 jo kernel: [595576.044235] Normal free:4128kB min:3992kB low:4988kB high:5988kB active_anon:1471080kB inactive_anon:490392kB active_file:896kB inactive_file:908kB unevictable:0kB present:2003840kB pages_scanned:4392213 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595576.044238] lowmem_reserve[]: 0 0 0 0
Jun 14 12:07:15 jo kernel: [595576.044241] DMA: 5*4kB 3*8kB 6*16kB 5*32kB 4*64kB 4*128kB 2*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB = 6700kB
Jun 14 12:07:15 jo kernel: [595576.044251] DMA32: 2114*4kB 6*8kB 6*16kB 12*32kB 15*64kB 6*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 11736kB
Jun 14 12:07:15 jo kernel: [595576.044260] Normal: 146*4kB 8*8kB 1*16kB 7*32kB 10*64kB 4*128kB 4*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 4088kB
Jun 14 12:07:15 jo kernel: [595576.044269] 470 total pagecache pages
Jun 14 12:07:15 jo kernel: [595576.044271] 0 pages in swap cache
Jun 14 12:07:15 jo kernel: [595576.044273] Swap cache stats: add 979646, delete 979646, find 30362/30755
Jun 14 12:07:15 jo kernel: [595576.044274] Free swap  = 0kB
Jun 14 12:07:15 jo kernel: [595576.044275] Total swap = 3903752kB
Jun 14 12:07:15 jo kernel: [595576.066499] 1556464 pages RAM
Jun 14 12:07:15 jo kernel: [595576.066501] 583861 pages reserved
Jun 14 12:07:15 jo kernel: [595576.066503] 1882 pages shared
Jun 14 12:07:15 jo kernel: [595576.066504] 964121 pages non-shared
Jun 14 12:07:15 jo kernel: [595576.066507] Out of memory: kill process 3150 (atd) score 1032 or a child
Jun 14 12:07:15 jo kernel: [595576.072206] Killed process 3150 (atd)
Jun 14 12:07:15 jo kernel: [595579.991348] lvm invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Jun 14 12:07:15 jo kernel: [595579.991352] Pid: 23333, comm: lvm Not tainted 2.6.28-19-server #61-Ubuntu
Jun 14 12:07:15 jo kernel: [595579.991355] Call Trace:
Jun 14 12:07:15 jo kernel: [595579.991364]  [<ffffffff802b3845>] oom_kill_process+0x95/0x240
Jun 14 12:07:15 jo kernel: [595579.991368]  [<ffffffff802b3fff>] ? select_bad_process+0xef/0x130
Jun 14 12:07:15 jo kernel: [595579.991371]  [<ffffffff802b40f4>] out_of_memory+0xb4/0x150
Jun 14 12:07:15 jo kernel: [595579.991374]  [<ffffffff802b6c69>] __alloc_pages_internal+0x4a9/0x4f0
Jun 14 12:07:15 jo kernel: [595579.991378]  [<ffffffff802b9a9a>] __do_page_cache_readahead+0xda/0x210
Jun 14 12:07:15 jo kernel: [595579.991382]  [<ffffffff802b9c2e>] do_page_cache_readahead+0x5e/0x90
Jun 14 12:07:15 jo kernel: [595579.991385]  [<ffffffff802b203a>] filemap_fault+0x34a/0x430
Jun 14 12:07:15 jo kernel: [595579.991389]  [<ffffffff802c6030>] __do_fault+0x50/0x520
Jun 14 12:07:15 jo kernel: [595579.991394]  [<ffffffff8069ae19>] ? _spin_lock+0x9/0x10
Jun 14 12:07:15 jo kernel: [595579.991397]  [<ffffffff802fb624>] ? __d_lookup+0x114/0x150
Jun 14 12:07:15 jo kernel: [595579.991401]  [<ffffffff8023dd90>] ? update_curr+0xf0/0x170
Jun 14 12:07:15 jo kernel: [595579.991404]  [<ffffffff802c71b9>] handle_mm_fault+0x1e9/0x470
Jun 14 12:07:15 jo kernel: [595579.991409]  [<ffffffff80210620>] ? __switch_to+0x1b0/0x490
Jun 14 12:07:15 jo kernel: [595579.991412]  [<ffffffff8069dc37>] do_page_fault+0x307/0x780
Jun 14 12:07:15 jo kernel: [595579.991416]  [<ffffffff8026c4ec>] ? lock_hrtimer_base+0x2c/0x60
Jun 14 12:07:15 jo kernel: [595579.991419]  [<ffffffff8026c5af>] ? hrtimer_try_to_cancel+0x3f/0x90
Jun 14 12:07:15 jo kernel: [595579.991422]  [<ffffffff8026c61a>] ? hrtimer_cancel+0x1a/0x30
Jun 14 12:07:15 jo kernel: [595579.991425]  [<ffffffff80699e90>] ? do_nanosleep+0x40/0xc0
Jun 14 12:07:15 jo kernel: [595579.991428]  [<ffffffff8026cc59>] ? hrtimer_nanosleep+0xa9/0x130
Jun 14 12:07:15 jo kernel: [595579.991431]  [<ffffffff8026bf00>] ? hrtimer_wakeup+0x0/0x30
Jun 14 12:07:15 jo kernel: [595579.991434]  [<ffffffff80699ece>] ? do_nanosleep+0x7e/0xc0
Jun 14 12:07:15 jo kernel: [595579.991437]  [<ffffffff8069b21a>] error_exit+0x0/0x70
Jun 14 12:07:15 jo kernel: [595579.991439] Mem-Info:
Jun 14 12:07:15 jo kernel: [595579.991441] DMA per-cpu:
Jun 14 12:07:15 jo kernel: [595579.991443] CPU    0: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595579.991445] CPU    1: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595579.991447] DMA32 per-cpu:
Jun 14 12:07:15 jo kernel: [595579.991448] CPU    0: hi:  186, btch:  31 usd: 158
Jun 14 12:07:15 jo kernel: [595579.991450] CPU    1: hi:  186, btch:  31 usd: 157
Jun 14 12:07:15 jo kernel: [595579.991452] Normal per-cpu:
Jun 14 12:07:15 jo kernel: [595579.991453] CPU    0: hi:  186, btch:  31 usd: 173
Jun 14 12:07:15 jo kernel: [595579.991455] CPU    1: hi:  186, btch:  31 usd: 172
Jun 14 12:07:15 jo kernel: [595579.991459] Active_anon:708497 active_file:240 inactive_anon:236124
Jun 14 12:07:15 jo kernel: [595579.991460]  inactive_file:263 unevictable:0 dirty:0 writeback:0 unstable:0
Jun 14 12:07:15 jo kernel: [595579.991461]  free:5600 slab:5848 mapped:17 pagetables:5869 bounce:0
Jun 14 12:07:15 jo kernel: [595579.991464] DMA free:6700kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:5560kB pages_scanned:0 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595579.991467] lowmem_reserve[]: 0 1967 3924 3924
Jun 14 12:07:15 jo kernel: [595579.991473] DMA32 free:11860kB min:4016kB low:5020kB high:6024kB active_anon:1362412kB inactive_anon:453832kB active_file:0kB inactive_file:24kB unevictable:0kB present:2015008kB pages_scanned:1090418 all_unreclaimable? no
Jun 14 12:07:15 jo kernel: [595579.991476] lowmem_reserve[]: 0 0 1956 1956
Jun 14 12:07:15 jo kernel: [595579.991481] Normal free:3840kB min:3992kB low:4988kB high:5988kB active_anon:1471576kB inactive_anon:490664kB active_file:960kB inactive_file:1028kB unevictable:0kB present:2003840kB pages_scanned:4799733 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595579.991484] lowmem_reserve[]: 0 0 0 0
Jun 14 12:07:15 jo kernel: [595579.991488] DMA: 5*4kB 3*8kB 6*16kB 5*32kB 4*64kB 4*128kB 2*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB = 6700kB
Jun 14 12:07:15 jo kernel: [595579.991498] DMA32: 2127*4kB 13*8kB 7*16kB 12*32kB 15*64kB 6*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 11860kB
Jun 14 12:07:15 jo kernel: [595579.991508] Normal: 94*4kB 3*8kB 1*16kB 7*32kB 10*64kB 4*128kB 4*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3840kB
Jun 14 12:07:15 jo kernel: [595579.991517] 516 total pagecache pages
Jun 14 12:07:15 jo kernel: [595579.991519] 0 pages in swap cache
Jun 14 12:07:15 jo kernel: [595579.991521] Swap cache stats: add 979680, delete 979680, find 30362/30755
Jun 14 12:07:15 jo kernel: [595579.991522] Free swap  = 0kB
Jun 14 12:07:15 jo kernel: [595579.991524] Total swap = 3903752kB
Jun 14 12:07:15 jo kernel: [595580.015593] 1556464 pages RAM
Jun 14 12:07:15 jo kernel: [595580.015595] 583861 pages reserved
Jun 14 12:07:15 jo kernel: [595580.015597] 2016 pages shared
Jun 14 12:07:15 jo kernel: [595580.015598] 964061 pages non-shared
Jun 14 12:07:15 jo kernel: [595580.015601] Out of memory: kill process 2278 (rpc.statd) score 907 or a child
Jun 14 12:07:15 jo kernel: [595580.021633] Killed process 2278 (rpc.statd)
Jun 14 12:07:15 jo kernel: [595582.231347] lvm invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Jun 14 12:07:15 jo kernel: [595582.231352] Pid: 28199, comm: lvm Not tainted 2.6.28-19-server #61-Ubuntu
Jun 14 12:07:15 jo kernel: [595582.231354] Call Trace:
Jun 14 12:07:15 jo kernel: [595582.231363]  [<ffffffff802b3845>] oom_kill_process+0x95/0x240
Jun 14 12:07:15 jo kernel: [595582.231367]  [<ffffffff802b3fff>] ? select_bad_process+0xef/0x130
Jun 14 12:07:15 jo kernel: [595582.231370]  [<ffffffff802b40f4>] out_of_memory+0xb4/0x150
Jun 14 12:07:15 jo kernel: [595582.231373]  [<ffffffff802b6c69>] __alloc_pages_internal+0x4a9/0x4f0
Jun 14 12:07:15 jo kernel: [595582.231377]  [<ffffffff802b9a9a>] __do_page_cache_readahead+0xda/0x210
Jun 14 12:07:15 jo kernel: [595582.231382]  [<ffffffff802689f0>] ? wake_bit_function+0x0/0x40
Jun 14 12:07:15 jo kernel: [595582.231385]  [<ffffffff802b9c2e>] do_page_cache_readahead+0x5e/0x90
Jun 14 12:07:15 jo kernel: [595582.231389]  [<ffffffff802b203a>] filemap_fault+0x34a/0x430
Jun 14 12:07:15 jo kernel: [595582.231393]  [<ffffffff802c6030>] __do_fault+0x50/0x520
Jun 14 12:07:15 jo kernel: [595582.231396]  [<ffffffff802d1ab1>] ? vmalloc+0x21/0x30
Jun 14 12:07:15 jo kernel: [595582.231399]  [<ffffffff802c71b9>] handle_mm_fault+0x1e9/0x470
Jun 14 12:07:15 jo kernel: [595582.231403]  [<ffffffff8022f669>] ? default_spin_lock_flags+0x9/0x10
Jun 14 12:07:15 jo kernel: [595582.231408]  [<ffffffff8069dc37>] do_page_fault+0x307/0x780
Jun 14 12:07:15 jo kernel: [595582.231412]  [<ffffffff80578850>] ? table_status+0x0/0x90
Jun 14 12:07:15 jo kernel: [595582.231415]  [<ffffffff802d0485>] ? vfree+0x25/0x30
Jun 14 12:07:15 jo kernel: [595582.231418]  [<ffffffff80579cd0>] ? ctl_ioctl+0x1f0/0x260
Jun 14 12:07:15 jo kernel: [595582.231421]  [<ffffffff802eba13>] ? cp_new_stat+0xe3/0xf0
Jun 14 12:07:15 jo kernel: [595582.231424]  [<ffffffff80579d4e>] ? dm_ctl_ioctl+0xe/0x20
Jun 14 12:07:15 jo kernel: [595582.231428]  [<ffffffff802f64f1>] ? vfs_ioctl+0x31/0xa0
Jun 14 12:07:15 jo kernel: [595582.231431]  [<ffffffff802f68a5>] ? do_vfs_ioctl+0x75/0x230
Jun 14 12:07:15 jo kernel: [595582.231435]  [<ffffffff8069b21a>] error_exit+0x0/0x70
Jun 14 12:07:15 jo kernel: [595582.231437] Mem-Info:
Jun 14 12:07:15 jo kernel: [595582.231439] DMA per-cpu:
Jun 14 12:07:15 jo kernel: [595582.231442] CPU    0: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595582.231443] CPU    1: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595582.231445] DMA32 per-cpu:
Jun 14 12:07:15 jo kernel: [595582.231447] CPU    0: hi:  186, btch:  31 usd: 178
Jun 14 12:07:15 jo kernel: [595582.231449] CPU    1: hi:  186, btch:  31 usd: 174
Jun 14 12:07:15 jo kernel: [595582.231450] Normal per-cpu:
Jun 14 12:07:15 jo kernel: [595582.231452] CPU    0: hi:  186, btch:  31 usd: 129
Jun 14 12:07:15 jo kernel: [595582.231453] CPU    1: hi:  186, btch:  31 usd: 176
Jun 14 12:07:15 jo kernel: [595582.231457] Active_anon:708580 active_file:187 inactive_anon:236066
Jun 14 12:07:15 jo kernel: [595582.231458]  inactive_file:353 unevictable:0 dirty:0 writeback:0 unstable:0
Jun 14 12:07:15 jo kernel: [595582.231459]  free:5630 slab:5847 mapped:7 pagetables:5856 bounce:0
Jun 14 12:07:15 jo kernel: [595582.231463] DMA free:6700kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:5560kB pages_scanned:0 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595582.231465] lowmem_reserve[]: 0 1967 3924 3924
Jun 14 12:07:15 jo kernel: [595582.231471] DMA32 free:11820kB min:4016kB low:5020kB high:6024kB active_anon:1363000kB inactive_anon:453984kB active_file:20kB inactive_file:96kB unevictable:0kB present:2015008kB pages_scanned:655113 all_unreclaimable? no
Jun 14 12:07:15 jo kernel: [595582.231474] lowmem_reserve[]: 0 0 1956 1956
Jun 14 12:07:15 jo kernel: [595582.231479] Normal free:4000kB min:3992kB low:4988kB high:5988kB active_anon:1471444kB inactive_anon:490280kB active_file:728kB inactive_file:1316kB unevictable:0kB present:2003840kB pages_scanned:2670 all_unreclaimable? no
Jun 14 12:07:15 jo kernel: [595582.231482] lowmem_reserve[]: 0 0 0 0
Jun 14 12:07:15 jo kernel: [595582.231486] DMA: 5*4kB 3*8kB 6*16kB 5*32kB 4*64kB 4*128kB 2*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB = 6700kB
Jun 14 12:07:15 jo kernel: [595582.231496] DMA32: 2127*4kB 13*8kB 7*16kB 12*32kB 15*64kB 6*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 11860kB
Jun 14 12:07:15 jo kernel: [595582.231505] Normal: 125*4kB 3*8kB 1*16kB 7*32kB 10*64kB 4*128kB 4*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3964kB
Jun 14 12:07:15 jo kernel: [595582.231514] 551 total pagecache pages
Jun 14 12:07:15 jo kernel: [595582.231516] 0 pages in swap cache
Jun 14 12:07:15 jo kernel: [595582.231518] Swap cache stats: add 979718, delete 979718, find 30362/30755
Jun 14 12:07:15 jo kernel: [595582.231519] Free swap  = 0kB
Jun 14 12:07:15 jo kernel: [595582.231521] Total swap = 3903752kB
Jun 14 12:07:15 jo kernel: [595582.256576] 1556464 pages RAM
Jun 14 12:07:15 jo kernel: [595582.256578] 583861 pages reserved
Jun 14 12:07:15 jo kernel: [595582.256580] 1901 pages shared
Jun 14 12:07:15 jo kernel: [595582.256581] 964064 pages non-shared
Jun 14 12:07:15 jo kernel: [595582.256584] Out of memory: kill process 28837 (rsync) score 854 or a child
Jun 14 12:07:15 jo kernel: [595582.262825] Killed process 28838 (ssh)
Jun 14 12:07:15 jo kernel: [595596.492167] lvm invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Jun 14 12:07:15 jo kernel: [595596.492171] Pid: 15611, comm: lvm Not tainted 2.6.28-19-server #61-Ubuntu
Jun 14 12:07:15 jo kernel: [595596.492174] Call Trace:
Jun 14 12:07:15 jo kernel: [595596.492182]  [<ffffffff802b3845>] oom_kill_process+0x95/0x240
Jun 14 12:07:15 jo kernel: [595596.492186]  [<ffffffff802b3fff>] ? select_bad_process+0xef/0x130
Jun 14 12:07:15 jo kernel: [595596.492189]  [<ffffffff802b40f4>] out_of_memory+0xb4/0x150
Jun 14 12:07:15 jo kernel: [595596.492193]  [<ffffffff802b6c69>] __alloc_pages_internal+0x4a9/0x4f0
Jun 14 12:07:15 jo kernel: [595596.492197]  [<ffffffff802b9a9a>] __do_page_cache_readahead+0xda/0x210
Jun 14 12:07:15 jo kernel: [595596.492200]  [<ffffffff802b9c2e>] do_page_cache_readahead+0x5e/0x90
Jun 14 12:07:15 jo kernel: [595596.492203]  [<ffffffff802b203a>] filemap_fault+0x34a/0x430
Jun 14 12:07:15 jo kernel: [595596.492207]  [<ffffffff802c6030>] __do_fault+0x50/0x520
Jun 14 12:07:15 jo kernel: [595596.492211]  [<ffffffff802c71b9>] handle_mm_fault+0x1e9/0x470
Jun 14 12:07:15 jo kernel: [595596.492215]  [<ffffffff8069dc37>] do_page_fault+0x307/0x780
Jun 14 12:07:15 jo kernel: [595596.492220]  [<ffffffff802689b0>] ? autoremove_wake_function+0x0/0x40
Jun 14 12:07:15 jo kernel: [595596.492224]  [<ffffffff80318038>] ? inotify_inode_queue_event+0xd8/0x100
Jun 14 12:07:15 jo kernel: [595596.492229]  [<ffffffff802e7569>] ? fsnotify_access+0x69/0x80
Jun 14 12:07:15 jo kernel: [595596.492232]  [<ffffffff802e82da>] ? vfs_read+0xda/0x130
Jun 14 12:07:15 jo kernel: [595596.492237]  [<ffffffff8069b21a>] error_exit+0x0/0x70
Jun 14 12:07:15 jo kernel: [595596.492239] Mem-Info:
Jun 14 12:07:15 jo kernel: [595596.492240] DMA per-cpu:
Jun 14 12:07:15 jo kernel: [595596.492242] CPU    0: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595596.492244] CPU    1: hi:    0, btch:   1 usd:   0
Jun 14 12:07:15 jo kernel: [595596.492246] DMA32 per-cpu:
Jun 14 12:07:15 jo kernel: [595596.492248] CPU    0: hi:  186, btch:  31 usd: 171
Jun 14 12:07:15 jo kernel: [595596.492249] CPU    1: hi:  186, btch:  31 usd: 180
Jun 14 12:07:15 jo kernel: [595596.492251] Normal per-cpu:
Jun 14 12:07:15 jo kernel: [595596.492252] CPU    0: hi:  186, btch:  31 usd: 159
Jun 14 12:07:15 jo kernel: [595596.492254] CPU    1: hi:  186, btch:  31 usd: 160
Jun 14 12:07:15 jo kernel: [595596.492257] Active_anon:708536 active_file:228 inactive_anon:236154
Jun 14 12:07:15 jo kernel: [595596.492258]  inactive_file:232 unevictable:0 dirty:0 writeback:0 unstable:0
Jun 14 12:07:15 jo kernel: [595596.492260]  free:5618 slab:5847 mapped:16 pagetables:5830 bounce:0
Jun 14 12:07:15 jo kernel: [595596.492263] DMA free:6700kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:5560kB pages_scanned:0 all_unreclaimable? yes
Jun 14 12:07:15 jo kernel: [595596.492266] lowmem_reserve[]: 0 1967 3924 3924
Jun 14 12:07:15 jo kernel: [595596.492272] DMA32 free:11808kB min:4016kB low:5020kB high:6024kB active_anon:1363404kB inactive_anon:454368kB active_file:0kB inactive_file:164kB unevictable:0kB present:2015008kB pages_scanned:832 all_unreclaimable? no
Jun 14 12:07:15 jo kernel: [595596.492275] lowmem_reserve[]: 0 0 1956 1956
Jun 14 12:07:15 jo kernel: [595596.492280] Normal free:3964kB min:3992kB low:4988kB high:5988kB active_anon:1470740kB inactive_anon:490248kB active_file:912kB inactive_file:764kB unevictable:0kB present:2003840kB pages_scanned:10382 all_unreclaimable? no
Jun 14 12:07:15 jo kernel: [595596.492283] lowmem_reserve[]: 0 0 0 0
Jun 14 12:07:15 jo kernel: [595596.492286] DMA: 5*4kB 3*8kB 6*16kB 5*32kB 4*64kB 4*128kB 2*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB = 6700kB
Jun 14 12:07:15 jo kernel: [595596.492296] DMA32: 2151*4kB 3*8kB 6*16kB 12*32kB 15*64kB 6*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 11860kB
Jun 14 12:07:15 jo kernel: [595596.492305] Normal: 125*4kB 3*8kB 1*16kB 7*32kB 10*64kB 4*128kB 4*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3964kB
Jun 14 12:07:15 jo kernel: [595596.492314] 495 total pagecache pages
Jun 14 12:07:15 jo kernel: [595596.492316] 0 pages in swap cache
Jun 14 12:07:15 jo kernel: [595596.492317] Swap cache stats: add 979736, delete 979736, find 30362/30757
Jun 14 12:07:15 jo kernel: [595596.492319] Free swap  = 0kB
Jun 14 12:07:15 jo kernel: [595596.492321] Total swap = 3903752kB
Jun 14 12:07:15 jo kernel: [595596.516691] 1556464 pages RAM
Jun 14 12:07:15 jo kernel: [595596.516693] 583861 pages reserved
Jun 14 12:07:15 jo kernel: [595596.516694] 1871 pages shared
Jun 14 12:07:15 jo kernel: [595596.516696] 964114 pages non-shared
Jun 14 12:07:15 jo kernel: [595596.516699] Out of memory: kill process 14703 (pvmove) score 819 or a child
Jun 14 12:07:15 jo kernel: [595596.523002] Killed process 14703 (pvmove)
Jun 14 12:07:43 jo kernel: [595637.432688] lvm invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=-17
Jun 14 12:07:43 jo kernel: [595637.432692] Pid: 24294, comm: lvm Not tainted 2.6.28-19-server #61-Ubuntu
Jun 14 12:07:43 jo kernel: [595637.432694] Call Trace:
Jun 14 12:07:43 jo kernel: [595637.432703]  [<ffffffff802b3845>] oom_kill_process+0x95/0x240
Jun 14 12:07:43 jo kernel: [595637.432707]  [<ffffffff802b3fff>] ? select_bad_process+0xef/0x130
Jun 14 12:07:43 jo kernel: [595637.432710]  [<ffffffff802b40f4>] out_of_memory+0xb4/0x150
Jun 14 12:07:43 jo kernel: [595637.432713]  [<ffffffff802b6c69>] __alloc_pages_internal+0x4a9/0x4f0
Jun 14 12:07:43 jo kernel: [595637.432717]  [<ffffffff802b9a9a>] __do_page_cache_readahead+0xda/0x210
Jun 14 12:07:43 jo kernel: [595637.432721]  [<ffffffff802b9c2e>] do_page_cache_readahead+0x5e/0x90
Jun 14 12:07:43 jo kernel: [595637.432724]  [<ffffffff802b203a>] filemap_fault+0x34a/0x430
Jun 14 12:07:43 jo kernel: [595637.432728]  [<ffffffff802c6030>] __do_fault+0x50/0x520
Jun 14 12:07:43 jo kernel: [595637.432731]  [<ffffffff802fb624>] ? __d_lookup+0x114/0x150
Jun 14 12:07:43 jo kernel: [595637.432735]  [<ffffffff8023dd90>] ? update_curr+0xf0/0x170
Jun 14 12:07:43 jo kernel: [595637.432739]  [<ffffffff802c71b9>] handle_mm_fault+0x1e9/0x470
Jun 14 12:07:43 jo kernel: [595637.432743]  [<ffffffff80210579>] ? __switch_to+0x109/0x490
Jun 14 12:07:43 jo kernel: [595637.432748]  [<ffffffff8069dc37>] do_page_fault+0x307/0x780
Jun 14 12:07:43 jo kernel: [595637.432752]  [<ffffffff8026c4ec>] ? lock_hrtimer_base+0x2c/0x60
Jun 14 12:07:43 jo kernel: [595637.432755]  [<ffffffff8026c5af>] ? hrtimer_try_to_cancel+0x3f/0x90
Jun 14 12:07:43 jo kernel: [595637.432757]  [<ffffffff8026c61a>] ? hrtimer_cancel+0x1a/0x30
Jun 14 12:07:43 jo kernel: [595637.432761]  [<ffffffff80699e90>] ? do_nanosleep+0x40/0xc0
Jun 14 12:07:43 jo kernel: [595637.432764]  [<ffffffff8026cc59>] ? hrtimer_nanosleep+0xa9/0x130
Jun 14 12:07:43 jo kernel: [595637.432768]  [<ffffffff8026bf00>] ? hrtimer_wakeup+0x0/0x30
Jun 14 12:07:43 jo kernel: [595637.432770]  [<ffffffff80699ece>] ? do_nanosleep+0x7e/0xc0
Jun 14 12:07:43 jo kernel: [595637.432774]  [<ffffffff8069b21a>] error_exit+0x0/0x70
Jun 14 12:07:43 jo kernel: [595637.432776] Mem-Info:
Jun 14 12:07:43 jo kernel: [595637.432778] DMA per-cpu:
Jun 14 12:07:43 jo kernel: [595637.432780] CPU    0: hi:    0, btch:   1 usd:   0
Jun 14 12:07:43 jo kernel: [595637.432782] CPU    1: hi:    0, btch:   1 usd:   0
Jun 14 12:07:43 jo kernel: [595637.432784] DMA32 per-cpu:
Jun 14 12:07:43 jo kernel: [595637.432785] CPU    0: hi:  186, btch:  31 usd: 174
Jun 14 12:07:43 jo kernel: [595637.432787] CPU    1: hi:  186, btch:  31 usd:  63
Jun 14 12:07:43 jo kernel: [595637.432789] Normal per-cpu:
Jun 14 12:07:43 jo kernel: [595637.432790] CPU    0: hi:  186, btch:  31 usd: 165
Jun 14 12:07:43 jo kernel: [595637.432792] CPU    1: hi:  186, btch:  31 usd: 168
Jun 14 12:07:43 jo kernel: [595637.432795] Active_anon:709411 active_file:237 inactive_anon:236435
Jun 14 12:07:43 jo kernel: [595637.432796]  inactive_file:290 unevictable:0 dirty:13 writeback:0 unstable:0
Jun 14 12:07:43 jo kernel: [595637.432797]  free:5673 slab:5860 mapped:18 pagetables:5843 bounce:0
Jun 14 12:07:43 jo kernel: [595637.432801] DMA free:6700kB min:8kB low:8kB high:12kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB present:5560kB pages_scanned:0 all_unreclaimable? yes
Jun 14 12:07:43 jo kernel: [595637.432804] lowmem_reserve[]: 0 1967 3924 3924
Jun 14 12:07:43 jo kernel: [595637.432809] DMA32 free:12124kB min:4016kB low:5020kB high:6024kB active_anon:1365040kB inactive_anon:454736kB active_file:180kB inactive_file:292kB unevictable:0kB present:2015008kB pages_scanned:2048 all_unreclaimable? no
Jun 14 12:07:43 jo kernel: [595637.432812] lowmem_reserve[]: 0 0 1956 1956
Jun 14 12:07:43 jo kernel: [595637.432818] Normal free:3868kB min:3992kB low:4988kB high:5988kB active_anon:1472604kB inactive_anon:490892kB active_file:768kB inactive_file:868kB unevictable:0kB present:2003840kB pages_scanned:5082473 all_unreclaimable? yes
Jun 14 12:07:43 jo kernel: [595637.432821] lowmem_reserve[]: 0 0 0 0
Jun 14 12:07:43 jo kernel: [595637.432824] DMA: 5*4kB 3*8kB 6*16kB 5*32kB 4*64kB 4*128kB 2*256kB 2*512kB 0*1024kB 0*2048kB 1*4096kB = 6700kB
Jun 14 12:07:43 jo kernel: [595637.432834] DMA32: 2208*4kB 14*8kB 6*16kB 13*32kB 15*64kB 6*128kB 0*256kB 2*512kB 0*1024kB 0*2048kB 0*4096kB = 12208kB
Jun 14 12:07:43 jo kernel: [595637.432843] Normal: 77*4kB 3*8kB 5*16kB 6*32kB 9*64kB 5*128kB 4*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 3868kB
Jun 14 12:07:43 jo kernel: [595637.432852] 515 total pagecache pages
Jun 14 12:07:43 jo kernel: [595637.432854] 0 pages in swap cache
Jun 14 12:07:43 jo kernel: [595637.432855] Swap cache stats: add 986698, delete 986698, find 30409/30876
Jun 14 12:07:43 jo kernel: [595637.432857] Free swap  = 0kB
Jun 14 12:07:43 jo kernel: [595637.432859] Total swap = 3903752kB
Jun 14 12:07:43 jo kernel: [595637.455000] 1556464 pages RAM
Jun 14 12:07:43 jo kernel: [595637.455002] 583861 pages reserved
Jun 14 12:07:43 jo kernel: [595637.455004] 1104 pages shared
Jun 14 12:07:43 jo kernel: [595637.455005] 964946 pages non-shared
Jun 14 12:07:43 jo kernel: [595637.455008] Out of memory: kill process 28843 (rsync) score 944 or a child
Jun 14 12:07:43 jo kernel: [595637.461432] Killed process 28844 (rsync)