[25205.564390] sshd invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=-1000 [25205.564393] sshd cpuset=/ mems_allowed=0 [25205.564398] CPU: 3 PID: 1521 Comm: sshd Not tainted 4.4.0-59-generic #80~14.04.1-Ubuntu [25205.564400] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 1106 10/17/2011 [25205.564402] 0000000000000000 ffff88081a283b38 ffffffff813dbd6c ffff88081a283cf0 [25205.564404] 0000000000000000 ffff88081a283bc8 ffffffff811fafc6 ffff88081a283bb8 [25205.564406] ffffffff8118bf5a ffff88083fad7d88 ffff88081a283bc0 ffff88083fdde500 [25205.564408] Call Trace: [25205.564414] [] dump_stack+0x63/0x87 [25205.564418] [] dump_header+0x5b/0x1d5 [25205.564422] [] ? free_pcppages_bulk+0x11a/0x430 [25205.564424] [] oom_kill_process+0x205/0x3d0 [25205.564426] [] out_of_memory+0x40b/0x460 [25205.564429] [] __alloc_pages_slowpath.constprop.87+0x742/0x7ad [25205.564432] [] __alloc_pages_nodemask+0x237/0x240 [25205.564434] [] alloc_kmem_pages_node+0x4d/0xd0 [25205.564437] [] copy_process+0x185/0x1ce0 [25205.564440] [] ? apparmor_file_alloc_security+0x5c/0x220 [25205.564442] [] ? kmem_cache_alloc+0x1e2/0x200 [25205.564445] [] ? security_file_alloc+0x33/0x50 [25205.564448] [] ? __fd_install+0x24/0xc0 [25205.564450] [] _do_fork+0x8a/0x310 [25205.564452] [] ? fd_install+0x25/0x30 [25205.564456] [] ? SyS_socketpair+0x16f/0x220 [25205.564458] [] SyS_clone+0x19/0x20 [25205.564462] [] entry_SYSCALL_64_fastpath+0x16/0x75 [25205.564464] Mem-Info: [25205.564468] active_anon:88138 inactive_anon:141265 isolated_anon:0 [25205.564468] active_file:1877472 inactive_file:1795676 isolated_file:0 [25205.564468] unevictable:8 dirty:480 writeback:0 unstable:0 [25205.564468] slab_reclaimable:4099672 slab_unreclaimable:9560 [25205.564468] mapped:7573 shmem:299 pagetables:2295 bounce:0 [25205.564468] free:174664 free_pcp:0 free_cma:0 [25205.564471] Node 0 DMA free:15328kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15360kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes [25205.564476] lowmem_reserve[]: 0 2865 32045 32045 32045 [25205.564478] Node 0 DMA32 free:124584kB min:6040kB low:7548kB high:9060kB active_anon:48812kB inactive_anon:68932kB active_file:816564kB inactive_file:609252kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3040872kB managed:2960440kB mlocked:0kB dirty:208kB writeback:0kB mapped:4648kB shmem:340kB slab_reclaimable:1284288kB slab_unreclaimable:3232kB kernel_stack:288kB pagetables:644kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:36 all_unreclaimable? no [25205.564483] lowmem_reserve[]: 0 0 29180 29180 29180 [25205.564485] Node 0 Normal free:558744kB min:61512kB low:76888kB high:92268kB active_anon:303740kB inactive_anon:496128kB active_file:6693324kB inactive_file:6573452kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:30406656kB managed:29880396kB mlocked:32kB dirty:1712kB writeback:0kB mapped:25644kB shmem:856kB slab_reclaimable:15114400kB slab_unreclaimable:34976kB kernel_stack:4208kB pagetables:8536kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:256 all_unreclaimable? no [25205.564490] lowmem_reserve[]: 0 0 0 0 0 [25205.564492] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (M) = 15328kB [25205.564500] Node 0 DMA32: 12719*4kB (UME) 9268*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 125020kB [25205.564507] Node 0 Normal: 115485*4kB (UMEH) 12138*8kB (UMEH) 3*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 559092kB [25205.564514] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [25205.564515] 3676319 total pagecache pages [25205.564517] 2851 pages in swap cache [25205.564518] Swap cache stats: add 109305, delete 106454, find 2709511/2727408 [25205.564519] Free swap = 33356160kB [25205.564520] Total swap = 33553332kB [25205.564521] 8365878 pages RAM [25205.564521] 0 pages HighMem/MovableOnly [25205.564522] 151829 pages reserved [25205.564523] 0 pages cma reserved [25205.564524] 0 pages hwpoisoned [25205.564525] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [25205.564530] [ 461] 0 461 4870 416 14 3 60 0 upstart-udev-br [25205.564532] [ 472] 0 472 12974 558 29 3 227 -1000 systemd-udevd [25205.564534] [ 679] 0 679 3816 374 12 3 48 0 upstart-socket- [25205.564536] [ 937] 0 937 3820 399 13 4 35 0 upstart-file-br [25205.564538] [ 1060] 0 1060 5450 439 13 3 36 0 getty [25205.564540] [ 1062] 0 1062 5450 425 13 3 35 0 getty [25205.564541] [ 1066] 0 1066 5450 461 12 3 13 0 getty [25205.564543] [ 1067] 0 1067 5450 434 12 3 36 0 getty [25205.564545] [ 1069] 0 1069 5450 429 13 3 39 0 getty [25205.564546] [ 1082] 0 1082 1093 360 8 3 34 0 acpid [25205.564548] [ 1115] 0 1115 5914 515 17 3 41 0 cron [25205.564550] [ 1158] 0 1158 4799 402 13 3 31 0 irqbalance [25205.564551] [ 1233] 102 1233 9779 508 23 3 62 0 dbus-daemon [25205.564553] [ 1251] 0 1251 31393 892 32 4 112 0 thermald [25205.564555] [ 1275] 101 1275 63961 630 27 4 130 0 rsyslogd [25205.564557] [ 1305] 0 1305 6085 481 16 3 117 0 screen [25205.564558] [ 1309] 0 1309 5073 705 15 3 609 0 bash [25205.564560] [ 1336] 0 1336 5073 715 15 3 646 0 bash [25205.564562] [ 1413] 0 1413 33792 18529 66 3 6575 0 waveclean [25205.564564] [ 1415] 0 1415 33807 320 49 4 399 0 nginx [25205.564565] [ 1416] 33 1416 34843 1427 54 4 1183 0 nginx [25205.564567] [ 1417] 33 1417 34835 1511 53 4 1131 0 nginx [25205.564569] [ 1418] 33 1418 34835 1508 53 4 1115 0 nginx [25205.564570] [ 1419] 33 1419 34829 1515 54 4 1091 0 nginx [25205.564572] [ 1420] 33 1420 34824 1496 55 4 1105 0 nginx [25205.564574] [ 1421] 33 1421 34831 1524 54 4 1102 0 nginx [25205.564575] [ 1422] 33 1422 34861 1480 53 4 1169 0 nginx [25205.564577] [ 1423] 33 1423 34836 1531 54 4 1098 0 nginx [25205.564579] [ 1521] 0 1521 15346 1197 36 3 141 -1000 sshd [25205.564580] [ 1620] 111 1620 124174 923 43 3 218 0 icinga2 [25205.564582] [ 1629] 111 1629 587460 2154 83 5 1171 0 icinga2 [25205.564584] [ 1645] 0 1645 6572 763 17 3 358 0 screen [25205.564585] [ 1647] 0 1647 5073 699 15 3 637 0 bash [25205.564587] [ 1665] 0 1665 5073 699 15 3 649 0 bash [25205.564588] [ 1725] 0 1725 3340 263 11 3 41 0 mcelog [25205.564590] [ 1733] 0 1733 2383 544 9 3 52 0 wave.sh [25205.564592] [ 1737] 0 1737 5073 815 15 3 540 0 bash [25205.564594] [ 1750] 0 1750 355846 163625 602 4 23794 0 wave [25205.564595] [ 1766] 0 1766 1097 166 8 3 2 0 tail [25205.564597] [ 1770] 0 1770 5073 818 15 3 529 0 bash [25205.564598] [ 1786] 0 1786 1097 142 8 3 18 0 tail [25205.564600] [ 1790] 0 1790 5073 776 16 3 585 0 bash [25205.564602] [ 1806] 0 1806 1097 145 8 3 18 0 tail [25205.564603] [ 1811] 0 1811 5073 727 15 3 592 0 bash [25205.564605] [ 1832] 0 1832 105625 16874 105 3 554 0 mcollectived [25205.564607] [ 1956] 0 1956 6336 536 17 3 53 0 master [25205.564609] [ 1986] 105 1986 6893 524 18 3 53 0 qmgr [25205.564610] [ 2365] 0 2365 48142 14642 89 3 743 0 puppet [25205.564612] [ 2442] 0 2442 5598 416 15 3 178 0 smartd [25205.564614] [ 2703] 0 2703 3352 381 10 3 20 0 mdadm [25205.564615] [ 2789] 0 2789 66396 2673 33 3 680 0 fail2ban-server [25205.564617] [ 2825] 105 2825 10045 742 25 3 107 0 tlsmgr [25205.564619] [ 2843] 0 2843 5450 431 13 3 39 0 getty [25205.564620] [ 3272] 106 3272 7865 912 20 3 110 0 ntpd [25205.564622] [14789] 111 14789 5835 489 17 3 125 0 nrpe [25205.564624] [14082] 1000 14082 320018 10658 128 5 0 0 collectd [25205.564626] [14103] 1000 14103 9189 2521 24 3 0 0 nginx_stats.pl [25205.564627] [15423] 105 15423 6852 677 17 4 0 0 pickup [25205.564629] Out of memory: Kill process 1750 (wave) score 10 or sacrifice child [25205.564674] Killed process 1750 (wave) total-vm:1423384kB, anon-rss:648920kB, file-rss:5580kB