[91220.596255] icinga2 invoked oom-killer: gfp_mask=0x27000c0(GFP_KERNEL_ACCOUNT|__GFP_NOTRACK), order=2, oom_score_adj=0 [91220.596257] icinga2 cpuset=/ mems_allowed=0 [91220.596261] CPU: 1 PID: 2009 Comm: icinga2 Tainted: G I 4.8.0-36-generic #36~16.04.1-Ubuntu [91220.596262] Hardware name: MSI MS-7522/MSI X58 Pro-E (MS-7522), BIOS V8.15 03/19/2011 [91220.596264] 0000000000000286 00000000e4ac99ea ffffa2859d5f7b38 ffffffff8dc2d7b3 [91220.596266] ffffa2859d5f7cc8 ffffa2859bd65cc0 ffffa2859d5f7ba0 ffffffff8da2e9bb [91220.596268] ffffa2859d5f7b40 0000000000000000 0000000000000000 0000000000000000 [91220.596270] Call Trace: [91220.596277] [] dump_stack+0x63/0x90 [91220.596282] [] dump_header+0x5c/0x1dc [91220.596287] [] oom_kill_process+0x226/0x3f0 [91220.596289] [] out_of_memory+0x35a/0x3f0 [91220.596291] [] __alloc_pages_slowpath+0x9fb/0xa20 [91220.596292] [] __alloc_pages_nodemask+0x2ba/0x300 [91220.596297] [] copy_process.part.30+0x146/0x1b50 [91220.596299] [] ? sock_recvmsg+0x3d/0x50 [91220.596301] [] ? SYSC_recvfrom+0xda/0x150 [91220.596302] [] _do_fork+0xe7/0x3f0 [91220.596304] [] ? __sys_recvmsg+0x51/0x90 [91220.596306] [] SyS_clone+0x19/0x20 [91220.596308] [] do_syscall_64+0x5e/0xc0 [91220.596312] [] entry_SYSCALL64_slow_path+0x25/0x25 [91220.596313] Mem-Info: [91220.596318] active_anon:293899 inactive_anon:133600 isolated_anon:0 active_file:2512730 inactive_file:251284 isolated_file:0 unevictable:8 dirty:459 writeback:0 unstable:0 slab_reclaimable:2899682 slab_unreclaimable:15044 mapped:8295 shmem:1470 pagetables:2530 bounce:0 free:41854 free_pcp:92 free_cma:0 [91220.596322] Node 0 active_anon:1175596kB inactive_anon:534400kB active_file:10050920kB inactive_file:1005136kB unevictable:32kB isolated(anon):0kB isolated(file):0kB mapped:33180kB dirty:1836kB writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 438272kB anon_thp: 5880kB writeback_tmp:0kB unstable:0kB pages_scanned:128 all_unreclaimable? no [91220.596322] Node 0 DMA free:15904kB min:40kB low:52kB high:64kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [91220.596326] lowmem_reserve[]: 0 2948 24047 24047 24047 [91220.596328] Node 0 DMA32 free:92412kB min:8276kB low:11292kB high:14308kB active_anon:83396kB inactive_anon:0kB active_file:689840kB inactive_file:11672kB unevictable:0kB writepending:56kB present:3120640kB managed:3055072kB mlocked:0kB slab_reclaimable:2174620kB slab_unreclaimable:2460kB kernel_stack:48kB pagetables:504kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [91220.596332] lowmem_reserve[]: 0 0 21099 21099 21099 [91220.596334] Node 0 Normal free:59100kB min:59260kB low:80864kB high:102468kB active_anon:1091728kB inactive_anon:534400kB active_file:9361176kB inactive_file:993604kB unevictable:32kB writepending:1780kB present:22020096kB managed:21608872kB mlocked:32kB slab_reclaimable:9424108kB slab_unreclaimable:57716kB kernel_stack:4656kB pagetables:9616kB bounce:0kB free_pcp:368kB local_pcp:0kB free_cma:0kB [91220.596338] lowmem_reserve[]: 0 0 0 0 0 [91220.596340] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB [91220.596348] Node 0 DMA32: 5060*4kB (UME) 8799*8kB (UME) 3*16kB (H) 5*32kB (H) 5*64kB (H) 3*128kB (H) 4*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 92568kB [91220.596356] Node 0 Normal: 14757*4kB (UMEH) 24*8kB (UEH) 11*16kB (H) 7*32kB (H) 9*64kB (H) 2*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60708kB [91220.596365] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [91220.596365] 2765564 total pagecache pages [91220.596366] 20 pages in swap cache [91220.596367] Swap cache stats: add 661, delete 641, find 0/2 [91220.596368] Free swap = 2094492kB [91220.596368] Total swap = 2097084kB [91220.596369] 6289181 pages RAM [91220.596369] 0 pages HighMem/MovableOnly [91220.596370] 119219 pages reserved [91220.596370] 0 pages cma reserved [91220.596370] 0 pages hwpoisoned [91220.596371] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [91220.596377] [ 439] 0 439 9582 550 24 3 40 0 cgmanager [91220.596378] [ 504] 0 504 7487 184 18 3 41 0 upstart-udev-br [91220.596380] [ 523] 0 523 11486 761 24 3 277 -1000 systemd-udevd [91220.596382] [ 776] 0 776 7471 71 18 3 0 0 upstart-socket- [91220.596384] [ 1252] 102 1252 10724 548 24 3 0 0 dbus-daemon [91220.596385] [ 1255] 0 1255 7475 70 20 3 0 0 upstart-file-br [91220.596387] [ 1267] 101 1267 64099 591 28 4 0 0 rsyslogd [91220.596388] [ 1298] 0 1298 6502 1011 17 3 0 0 screen [91220.596390] [ 1300] 0 1300 4959 1160 15 3 0 0 bash [91220.596391] [ 1313] 0 1313 4959 1218 14 3 0 0 bash [91220.596392] [ 1353] 0 1353 6235 727 17 3 0 0 screen [91220.596394] [ 1355] 0 1355 4958 1163 15 3 0 0 bash [91220.596395] [ 1400] 0 1400 4958 1224 14 3 0 0 bash [91220.596397] [ 1435] 1000 1435 302700 9356 124 4 0 0 collectd [91220.596398] [ 1450] 1000 1450 7685 1776 19 3 0 0 nginx_stats.pl [91220.596400] [ 1459] 0 1459 2396 634 10 3 0 0 wave.sh [91220.596401] [ 1460] 0 1460 431730 337709 710 4 0 0 wave [91220.596402] [ 1463] 0 1463 4959 1197 15 3 0 0 bash [91220.596404] [ 1517] 0 1517 33853 25055 67 3 0 0 waveclean [91220.596405] [ 1526] 0 1526 43185 804 52 4 0 0 nginx [91220.596407] [ 1527] 33 1527 44283 2724 58 4 0 0 nginx [91220.596408] [ 1528] 33 1528 44240 2680 57 4 0 0 nginx [91220.596409] [ 1529] 33 1529 44283 2723 58 4 0 0 nginx [91220.596411] [ 1530] 33 1530 44374 2740 60 4 0 0 nginx [91220.596412] [ 1531] 33 1531 44261 2694 58 4 0 0 nginx [91220.596413] [ 1532] 33 1532 44266 2706 57 4 0 0 nginx [91220.596415] [ 1533] 33 1533 44266 2707 58 4 0 0 nginx [91220.596416] [ 1534] 33 1534 44262 2722 59 4 0 0 nginx [91220.596417] [ 1563] 0 1563 1103 186 8 3 0 0 tail [91220.596418] [ 1567] 0 1567 4959 1225 14 3 0 0 bash [91220.596420] [ 1590] 0 1590 3985 388 13 3 0 0 getty [91220.596421] [ 1592] 0 1592 3985 416 13 3 0 0 getty [91220.596422] [ 1597] 0 1597 3985 403 11 3 0 0 getty [91220.596424] [ 1598] 0 1598 3985 400 13 3 0 0 getty [91220.596425] [ 1600] 0 1600 3985 395 13 3 0 0 getty [91220.596427] [ 1629] 0 1629 1100 29 8 3 0 0 acpid [91220.596428] [ 1631] 0 1631 16380 1451 37 3 0 -1000 sshd [91220.596430] [ 1632] 0 1632 6517 602 18 3 0 0 cron [91220.596431] [ 1660] 0 1660 4868 63 14 3 0 0 irqbalance [91220.596432] [ 1661] 0 1661 25742 69 17 3 0 0 lvmetad [91220.596434] [ 1724] 0 1724 1103 169 8 3 0 0 tail [91220.596435] [ 1729] 0 1729 4959 1227 16 3 0 0 bash [91220.596437] [ 1751] 0 1751 106530 19900 114 3 0 0 mcollectived [91220.596438] [ 1769] 0 1769 1103 167 8 3 0 0 tail [91220.596440] [ 1777] 0 1777 4959 1171 16 3 0 0 bash [91220.596441] [ 1821] 0 1821 70268 3621 39 3 0 0 fail2ban-server [91220.596442] [ 2009] 108 2009 125766 1207 44 3 0 0 icinga2 [91220.596444] [ 2018] 108 2018 539747 2870 81 5 0 0 icinga2 [91220.596445] [ 2127] 0 2127 3224 37 11 3 0 0 mcelog [91220.596446] [ 2140] 0 2140 3422 499 11 3 0 0 mdadm [91220.596448] [ 2311] 0 2311 7137 618 20 3 0 0 systemd-logind [91220.596449] [ 2396] 104 2396 27509 1078 24 3 0 0 ntpd [91220.596451] [ 2416] 0 2416 16352 894 24 3 0 0 master [91220.596452] [ 2439] 105 2439 16910 915 25 3 0 0 qmgr [91220.596453] [ 2531] 0 2531 49440 15538 94 3 0 0 puppet [91220.596455] [ 2617] 0 2617 6370 587 16 3 0 0 smartd [91220.596456] [ 2863] 0 2863 3985 405 12 3 0 0 getty [91220.596457] [21506] 0 21506 24877 1558 53 3 0 0 sshd [91220.596459] [21689] 1005 21689 24877 991 51 3 0 0 sshd [91220.596460] [ 3573] 105 3573 20176 1136 31 3 0 0 tlsmgr [91220.596462] [ 1278] 105 1278 16869 1110 25 3 0 0 pickup [91220.596463] Out of memory: Kill process 1460 (wave) score 49 or sacrifice child [91220.598169] Killed process 1460 (wave) total-vm:1726920kB, anon-rss:1343488kB, file-rss:5260kB, shmem-rss:4728kB