[340449.041963] lsof invoked oom-killer: gfp_mask=0x27000c0(GFP_KERNEL_ACCOUNT|__GFP_NOTRACK), order=2, oom_score_adj=0 [340449.041966] lsof cpuset=/ mems_allowed=0 [340449.041971] CPU: 6 PID: 5035 Comm: lsof Not tainted 4.8.0-36-generic #36~16.04.1-Ubuntu [340449.041972] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 1106 10/17/2011 [340449.041974] 0000000000000286 000000004a965fc5 ffff998ec77a7b38 ffffffff9e42d7b3 [340449.041976] ffff998ec77a7cc8 ffff99969a590e80 ffff998ec77a7ba0 ffffffff9e22e9bb [340449.041978] ffff998ec77a7b40 0000000000000000 0000000000000000 0000000000000000 [340449.041980] Call Trace: [340449.041987] [] dump_stack+0x63/0x90 [340449.041991] [] dump_header+0x5c/0x1dc [340449.041994] [] oom_kill_process+0x226/0x3f0 [340449.041996] [] out_of_memory+0x35a/0x3f0 [340449.041997] [] __alloc_pages_slowpath+0x9fb/0xa20 [340449.041999] [] __alloc_pages_nodemask+0x2ba/0x300 [340449.042002] [] copy_process.part.30+0x146/0x1b50 [340449.042005] [] ? apparmor_file_alloc_security+0x5c/0x270 [340449.042008] [] ? kmem_cache_alloc+0xd7/0x1b0 [340449.042009] [] _do_fork+0xe7/0x3f0 [340449.042011] [] ? __fd_install+0x33/0xe0 [340449.042013] [] SyS_clone+0x19/0x20 [340449.042015] [] do_syscall_64+0x5e/0xc0 [340449.042018] [] entry_SYSCALL64_slow_path+0x25/0x25 [340449.042019] Mem-Info: [340449.042023] active_anon:355084 inactive_anon:132082 isolated_anon:1 [340449.042023] active_file:3770445 inactive_file:314070 isolated_file:0 [340449.042023] unevictable:8 dirty:1383 writeback:1 unstable:0 [340449.042023] slab_reclaimable:3550601 slab_unreclaimable:17373 [340449.042023] mapped:9444 shmem:1460 pagetables:2581 bounce:0 [340449.042023] free:49708 free_pcp:302 free_cma:0 [340449.042026] Node 0 active_anon:1420336kB inactive_anon:528328kB active_file:15081780kB inactive_file:1256280kB unevictable:32kB isolated(anon):4kB isolated(file):0kB mapped:37776kB dirty:5532kB writeback:4kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 878592kB anon_thp: 5840kB writeback_tmp:0kB unstable:0kB pages_scanned:363 all_unreclaimable? no [340449.042027] Node 0 DMA free:15328kB min:28kB low:40kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15984kB managed:15360kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [340449.042030] lowmem_reserve[]: 0 2870 32031 32031 32031 [340449.042032] Node 0 DMA32 free:122280kB min:6052kB low:8988kB high:11924kB active_anon:147520kB inactive_anon:0kB active_file:491624kB inactive_file:97620kB unevictable:0kB writepending:4kB present:3040872kB managed:2975304kB mlocked:0kB slab_reclaimable:2112076kB slab_unreclaimable:2956kB kernel_stack:80kB pagetables:180kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB [340449.042035] lowmem_reserve[]: 0 0 29161 29161 29161 [340449.042037] Node 0 Normal free:61224kB min:61496kB low:91356kB high:121216kB active_anon:1271612kB inactive_anon:528328kB active_file:14590812kB inactive_file:1158536kB unevictable:32kB writepending:5532kB present:30406656kB managed:29864896kB mlocked:32kB slab_reclaimable:12090328kB slab_unreclaimable:66504kB kernel_stack:4640kB pagetables:10144kB bounce:0kB free_pcp:1464kB local_pcp:0kB free_cma:0kB [340449.042040] lowmem_reserve[]: 0 0 0 0 0 [340449.042042] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (M) = 15328kB [340449.042049] Node 0 DMA32: 18568*4kB (UME) 6000*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 122272kB [340449.042055] Node 0 Normal: 14997*4kB (UMEH) 22*8kB (MH) 4*16kB (H) 14*32kB (H) 6*64kB (H) 1*128kB (H) 0*256kB 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 61700kB [340449.042062] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [340449.042063] 4086143 total pagecache pages [340449.042064] 33 pages in swap cache [340449.042065] Swap cache stats: add 1323, delete 1290, find 36/49 [340449.042065] Free swap = 33548176kB [340449.042066] Total swap = 33553332kB [340449.042066] 8365878 pages RAM [340449.042067] 0 pages HighMem/MovableOnly [340449.042067] 151988 pages reserved [340449.042067] 0 pages cma reserved [340449.042068] 0 pages hwpoisoned [340449.042068] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [340449.042072] [ 474] 0 474 9582 643 24 3 39 0 cgmanager [340449.042074] [ 491] 0 491 7487 358 18 3 51 0 upstart-udev-br [340449.042076] [ 498] 0 498 11444 766 23 3 255 -1000 systemd-udevd [340449.042077] [ 719] 0 719 7471 29 18 3 43 0 upstart-socket- [340449.042078] [ 1220] 0 1220 7475 27 17 3 45 0 upstart-file-br [340449.042080] [ 1262] 0 1262 6502 909 17 3 125 0 screen [340449.042081] [ 1263] 0 1263 6234 644 18 3 107 0 screen [340449.042082] [ 1266] 0 1266 4968 1154 15 3 71 0 bash [340449.042083] [ 1267] 0 1267 4968 1093 15 3 64 0 bash [340449.042084] [ 1285] 0 1285 4968 1183 14 3 44 0 bash [340449.042085] [ 1287] 0 1287 4968 1210 15 3 38 0 bash [340449.042087] [ 1357] 102 1357 10727 544 23 3 51 0 dbus-daemon [340449.042088] [ 1359] 101 1359 64099 644 28 3 20 0 rsyslogd [340449.042089] [ 1459] 0 1459 33853 25109 67 3 0 0 waveclean [340449.042091] [ 1460] 0 1460 2396 622 10 3 0 0 wave.sh [340449.042092] [ 1468] 0 1468 505710 396448 829 5 0 0 wave [340449.042093] [ 1470] 0 1470 4968 1208 16 3 0 0 bash [340449.042094] [ 1512] 0 1512 43186 945 53 4 0 0 nginx [340449.042095] [ 1513] 33 1513 44293 2763 60 5 0 0 nginx [340449.042097] [ 1514] 33 1514 44303 2768 61 5 0 0 nginx [340449.042098] [ 1515] 33 1515 44261 2731 59 5 0 0 nginx [340449.042099] [ 1516] 33 1516 44261 2749 60 5 0 0 nginx [340449.042100] [ 1517] 33 1517 44280 2750 60 5 0 0 nginx [340449.042102] [ 1518] 33 1518 44239 2719 61 5 0 0 nginx [340449.042103] [ 1519] 33 1519 44280 2763 59 5 0 0 nginx [340449.042104] [ 1520] 33 1520 44310 2781 61 5 0 0 nginx [340449.042105] [ 1538] 0 1538 4367 420 13 3 0 0 getty [340449.042106] [ 1540] 0 1540 4367 401 12 3 0 0 getty [340449.042108] [ 1541] 0 1541 40878 1013 34 3 0 0 thermald [340449.042109] [ 1545] 0 1545 4367 404 13 3 0 0 getty [340449.042110] [ 1546] 0 1546 4367 385 13 3 0 0 getty [340449.042111] [ 1548] 0 1548 4367 394 13 3 0 0 getty [340449.042112] [ 1559] 0 1559 1100 30 8 3 0 0 acpid [340449.042114] [ 1568] 0 1568 16380 1482 36 3 0 -1000 sshd [340449.042115] [ 1573] 0 1573 6517 629 18 3 0 0 cron [340449.042116] [ 1591] 0 1591 25742 73 18 3 0 0 lvmetad [340449.042117] [ 1652] 0 1652 4892 458 14 3 0 0 irqbalance [340449.042118] [ 1703] 0 1703 1103 154 8 3 0 0 tail [340449.042120] [ 1708] 0 1708 4968 1205 14 3 0 0 bash [340449.042121] [ 1733] 0 1733 1103 178 8 3 0 0 tail [340449.042122] [ 1737] 0 1737 4968 1228 15 3 0 0 bash [340449.042123] [ 1776] 0 1776 106562 21064 112 3 0 0 mcollectived [340449.042124] [ 1836] 0 1836 1103 155 8 3 0 0 tail [340449.042125] [ 1840] 0 1840 4968 1214 15 3 0 0 bash [340449.042126] [ 1878] 0 1878 70365 3618 39 4 0 0 fail2ban-server [340449.042128] [ 1899] 0 1899 7137 633 18 3 0 0 systemd-logind [340449.042129] [ 2057] 110 2057 125766 1185 43 3 0 0 icinga2 [340449.042130] [ 2066] 110 2066 589094 4302 83 5 0 0 icinga2 [340449.042132] [ 2175] 0 2175 3353 41 11 3 0 0 mcelog [340449.042133] [ 2188] 0 2188 3423 487 11 3 0 0 mdadm [340449.042134] [ 2364] 0 2364 16352 907 23 3 0 0 master [340449.042135] [ 2395] 105 2395 16910 719 25 3 0 0 qmgr [340449.042136] [ 2489] 106 2489 27509 986 24 3 0 0 ntpd [340449.042137] [ 2537] 0 2537 49452 15607 94 3 0 0 puppet [340449.042139] [ 2587] 0 2587 6389 636 16 3 0 0 smartd [340449.042140] [ 2827] 0 2827 4367 387 14 3 0 0 getty [340449.042141] [26139] 105 26139 20176 958 31 4 0 0 tlsmgr [340449.042142] [21012] 1000 21012 302696 8335 122 4 0 0 collectd [340449.042143] [21027] 1000 21027 7691 2153 21 3 0 0 nginx_stats.pl [340449.042145] [20199] 105 20199 16869 1128 24 3 0 0 pickup [340449.042146] [ 5035] 1000 5035 3355 272 12 3 0 0 lsof [340449.042148] Out of memory: Kill process 1468 (wave) score 23 or sacrifice child [340449.042767] Killed process 1468 (wave) total-vm:2022840kB, anon-rss:1579304kB, file-rss:5456kB, shmem-rss:4728kB