Before it dies: 1 [|||| 2.7%] Tasks: 43, 109 thr; 1 running 2 [|||||||| 6.2%] Load average: 1.86 1.98 1.80 3 [||||||||||||||||||| 16.9%] Uptime: 83 days, 23:12:45 4 [|||||||||||| 10.1%] Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||45.7G/47.0G] Swp[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||4.00G/4.00G] PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 25704 root 20 0 131M 1236 20 S 0.0 0.0 0:00.18 │ └─ sshd: linko [priv] 25707 linko 20 0 131M 1240 0 S 0.0 0.0 0:00.57 │ └─ sshd: linko@pts/1 25708 linko 20 0 112M 396 0 S 0.0 0.0 0:00.01 │ └─ -bash 25731 root 20 0 178M 640 12 S 0.0 0.0 0:00.00 │ └─ sudo mc 25732 root 20 0 167M 1848 0 S 0.0 0.0 0:00.25 │ └─ mc 25734 root 20 0 112M 416 4 S 0.0 0.0 0:00.00 │ └─ bash -rcfile .bashrc 25757 root 20 0 50472 704 0 S 0.0 0.0 2:41.92 │ ├─ nc blackbox-03.mysql 7000 25756 root 20 0 27.5G 25.5G 256 S 2.0 54.2 25:48.73 │ └─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./ 25769 root 20 0 27.5G 25.5G 256 S 0.7 54.2 22:40.48 │ ├─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./ 25768 root 20 0 27.5G 25.5G 256 S 0.0 54.2 0:00.00 │ ├─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./ 25767 root 20 0 27.5G 25.5G 256 S 0.0 54.2 0:00.00 │ ├─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./ 25766 root 20 0 27.5G 25.5G 256 S 0.0 54.2 0:00.00 │ ├─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./ 25765 root 20 0 27.5G 25.5G 256 S 0.0 54.2 0:00.00 │ ├─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./ 25764 root 20 0 27.5G 25.5G 256 S 1.4 54.2 3:06.15 │ └─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./ /var/log/messages: Dec 21 07:16:43 blackbox-02.mysql kernel: sh invoked oom-killer: gfp_mask=0x84d0, order=0, oom_score_adj=0 Dec 21 07:16:43 blackbox-02.mysql kernel: sh cpuset=/ mems_allowed=0 Dec 21 07:16:43 blackbox-02.mysql kernel: CPU: 3 PID: 29964 Comm: sh Not tainted 3.10.0-229.el7.x86_64 #1 Dec 21 07:16:43 blackbox-02.mysql kernel: Hardware name: Red Hat KVM, BIOS seabios-1.7.5-11.el7 04/01/2014 Dec 21 07:16:43 blackbox-02.mysql kernel: ffff88003687b8e0 000000007e8cc3bf ffff8805eabbb858 ffffffff81604b0a Dec 21 07:16:43 blackbox-02.mysql kernel: ffff8805eabbb8e8 ffffffff815ffaaf 0000000000000001 00000000000a0008 Dec 21 07:16:43 blackbox-02.mysql kernel: 0000000000000206 ffffffffffffff10 ffffffff8160bbdb 0000000000000010 Dec 21 07:16:43 blackbox-02.mysql kernel: Call Trace: Dec 21 07:16:43 blackbox-02.mysql kernel: [] dump_stack+0x19/0x1b Dec 21 07:16:43 blackbox-02.mysql kernel: [] dump_header+0x8e/0x214 Dec 21 07:16:43 blackbox-02.mysql kernel: [] ? _raw_spin_unlock_irqrestore+0x1b/0x40 Dec 21 07:16:43 blackbox-02.mysql kernel: [] oom_kill_process+0x24e/0x3b0 Dec 21 07:16:43 blackbox-02.mysql kernel: [] ? has_capability_noaudit+0x1e/0x30 Dec 21 07:16:43 blackbox-02.mysql kernel: [] out_of_memory+0x4b6/0x4f0 Dec 21 07:16:43 blackbox-02.mysql kernel: [] __alloc_pages_nodemask+0xa95/0xb90 Dec 21 07:16:43 blackbox-02.mysql kernel: [] alloc_pages_current+0xa9/0x170 Dec 21 07:16:43 blackbox-02.mysql kernel: [] __pmd_alloc+0x24/0x110 Dec 21 07:16:43 blackbox-02.mysql kernel: [] move_page_tables+0x640/0x6e0 Dec 21 07:16:43 blackbox-02.mysql kernel: [] ? down_write+0x12/0x30 Dec 21 07:16:43 blackbox-02.mysql kernel: [] shift_arg_pages+0xcc/0x1e0 Dec 21 07:16:43 blackbox-02.mysql kernel: [] ? selinux_inode_permission+0xaf/0x170 Dec 21 07:16:43 blackbox-02.mysql kernel: [] setup_arg_pages+0x1fb/0x230 Dec 21 07:16:43 blackbox-02.mysql kernel: [] load_elf_binary+0x44c/0xdc0 Dec 21 07:16:43 blackbox-02.mysql kernel: [] ? load_misc_binary+0x64/0x460 [binfmt_misc] Dec 21 07:16:43 blackbox-02.mysql kernel: [] ? ima_get_action+0x23/0x30 Dec 21 07:16:43 blackbox-02.mysql kernel: [] ? process_measurement+0x8e/0x250 Dec 21 07:16:43 blackbox-02.mysql kernel: [] ? ima_bprm_check+0x49/0x50 Dec 21 07:16:43 blackbox-02.mysql kernel: [] ? load_elf_library+0x230/0x230 Dec 21 07:16:43 blackbox-02.mysql kernel: [] search_binary_handler+0xf5/0x310 Dec 21 07:16:43 blackbox-02.mysql kernel: [] do_execve_common.isra.22+0x559/0x650 Dec 21 07:16:43 blackbox-02.mysql kernel: [] SyS_execve+0x36/0x50 Dec 21 07:16:43 blackbox-02.mysql kernel: [] stub_execve+0x69/0xa0 Dec 21 07:16:43 blackbox-02.mysql kernel: Mem-Info: Dec 21 07:16:43 blackbox-02.mysql kernel: Node 0 DMA per-cpu: Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 0: hi: 0, btch: 1 usd: 0 Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 1: hi: 0, btch: 1 usd: 0 Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 2: hi: 0, btch: 1 usd: 0 Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 3: hi: 0, btch: 1 usd: 0 Dec 21 07:16:43 blackbox-02.mysql kernel: Node 0 DMA32 per-cpu: Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 0: hi: 186, btch: 31 usd: 0 Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 1: hi: 186, btch: 31 usd: 30 Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 2: hi: 186, btch: 31 usd: 0 Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 3: hi: 186, btch: 31 usd: 58 Dec 21 07:16:43 blackbox-02.mysql kernel: Node 0 Normal per-cpu: Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 0: hi: 186, btch: 31 usd: 30 Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 1: hi: 186, btch: 31 usd: 99 Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 2: hi: 186, btch: 31 usd: 170 Dec 21 07:16:43 blackbox-02.mysql kernel: CPU 3: hi: 186, btch: 31 usd: 171 Dec 21 07:16:43 blackbox-02.mysql kernel: active_anon:11421415 inactive_anon:708913 isolated_anon:0#012 active_file:0 inactive_file:335 isolated_file:0#012 unevictable:0 dirty:41 writeback:0 unstable:0#012 free:66057 slab_reclaimable:7527 slab_unreclaimable:9562#012 mapped:557 shmem:4411 pagetables:27434 bounce:0#012 free_cma:0 Dec 21 07:16:43 blackbox-02.mysql kernel: Node 0 DMA free:15908kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Dec 21 07:16:43 blackbox-02.mysql kernel: lowmem_reserve[]: 0 2813 48124 48124 Dec 21 07:16:43 blackbox-02.mysql kernel: Node 0 DMA32 free:184924kB min:3952kB low:4940kB high:5928kB active_anon:2023112kB inactive_anon:652768kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3129212kB managed:2883084kB mlocked:0kB dirty:0kB writeback:0kB mapped:1844kB shmem:9284kB slab_reclaimable:8156kB slab_unreclaimable:2152kB kernel_stack:160kB pagetables:10192kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Dec 21 07:16:43 blackbox-02.mysql kernel: lowmem_reserve[]: 0 0 45310 45310 Dec 21 07:16:43 blackbox-02.mysql kernel: Node 0 Normal free:63396kB min:63608kB low:79508kB high:95412kB active_anon:43662548kB inactive_anon:2182884kB active_file:8kB inactive_file:1584kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:47185920kB managed:46397800kB mlocked:0kB dirty:164kB writeback:0kB mapped:384kB shmem:8360kB slab_reclaimable:21952kB slab_unreclaimable:36096kB kernel_stack:3920kB pagetables:99544kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1245 all_unreclaimable? yes Dec 21 07:16:43 blackbox-02.mysql kernel: lowmem_reserve[]: 0 0 0 0 Dec 21 07:16:43 blackbox-02.mysql kernel: Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15908kB Dec 21 07:16:43 blackbox-02.mysql kernel: Node 0 DMA32: 351*4kB (UE) 141*8kB (UE) 277*16kB (UEM) 224*32kB (UEM) 193*64kB (UEM) 152*128kB (UE) 124*256kB (UEM) 82*512kB (UE) 62*1024kB (UEM) 1*2048kB (E) 0*4096kB = 185204kB Dec 21 07:16:43 blackbox-02.mysql kernel: Node 0 Normal: 2153*4kB (UE) 431*8kB (UE) 318*16kB (UEM) 217*32kB (UEM) 120*64kB (UE) 68*128kB (EM) 40*256kB (EM) 7*512kB (EM) 5*1024kB (E) 2*2048kB (E) 0*4096kB = 63516kB Dec 21 07:16:43 blackbox-02.mysql kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Dec 21 07:16:43 blackbox-02.mysql kernel: 7207 total pagecache pages Dec 21 07:16:43 blackbox-02.mysql kernel: 2225 pages in swap cache Dec 21 07:16:43 blackbox-02.mysql kernel: Swap cache stats: add 7116573, delete 7114348, find 9601087/9937906 Dec 21 07:16:43 blackbox-02.mysql kernel: Free swap = 0kB Dec 21 07:16:43 blackbox-02.mysql kernel: Total swap = 4194300kB Dec 21 07:16:43 blackbox-02.mysql kernel: 12582781 pages RAM Dec 21 07:16:43 blackbox-02.mysql kernel: 258583 pages reserved Dec 21 07:16:43 blackbox-02.mysql kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Dec 21 07:16:43 blackbox-02.mysql kernel: [ 469] 0 469 19443 679 41 39 0 systemd-journal Dec 21 07:16:43 blackbox-02.mysql kernel: [ 485] 0 485 32253 45 29 331 0 lvmetad Dec 21 07:16:43 blackbox-02.mysql kernel: [ 493] 0 493 11443 2 24 503 -1000 systemd-udevd Dec 21 07:16:43 blackbox-02.mysql kernel: [ 617] 0 617 29171 22 26 93 -1000 auditd Dec 21 07:16:43 blackbox-02.mysql kernel: [ 640] 0 640 154735 780 86 971 0 rsyslogd Dec 21 07:16:43 blackbox-02.mysql kernel: [ 642] 0 642 6622 79 16 33 0 systemd-logind Dec 21 07:16:43 blackbox-02.mysql kernel: [ 645] 0 645 139834 675 89 2467 0 tuned Dec 21 07:16:43 blackbox-02.mysql kernel: [ 646] 0 646 4788 38 14 19 0 irqbalance Dec 21 07:16:43 blackbox-02.mysql kernel: [ 655] 81 655 8689 60 17 47 -900 dbus-daemon Dec 21 07:16:43 blackbox-02.mysql kernel: [ 672] 0 672 1084 3 6 32 0 acpid Dec 21 07:16:43 blackbox-02.mysql kernel: [ 689] 38 689 11265 39 26 139 0 ntpd Dec 21 07:16:43 blackbox-02.mysql kernel: [ 706] 0 706 31574 24 17 133 0 crond Dec 21 07:16:43 blackbox-02.mysql kernel: [ 712] 0 712 27501 2 10 31 0 agetty Dec 21 07:16:43 blackbox-02.mysql kernel: [ 914] 0 914 20744 32 45 184 -1000 sshd Dec 21 07:16:43 blackbox-02.mysql kernel: [ 916] 0 916 443728 1198 129 2688 0 collectd Dec 21 07:16:43 blackbox-02.mysql kernel: [ 2248] 0 2248 23058 19 42 261 0 master Dec 21 07:16:43 blackbox-02.mysql kernel: [ 2308] 89 2308 23128 9 45 274 0 qmgr Dec 21 07:16:43 blackbox-02.mysql kernel: [ 2944] 0 2944 31432 0 17 146 0 screen Dec 21 07:16:43 blackbox-02.mysql kernel: [ 2945] 0 2945 28838 2 14 103 0 bash Dec 21 07:16:43 blackbox-02.mysql kernel: [32367] 997 32367 11742 11 26 165 0 nrpe Dec 21 07:16:43 blackbox-02.mysql kernel: [12276] 0 12276 31533 38 16 221 0 screen Dec 21 07:16:43 blackbox-02.mysql kernel: [12277] 0 12277 28871 14 16 126 0 bash Dec 21 07:16:43 blackbox-02.mysql kernel: [29981] 0 29981 28838 14 13 97 0 bash Dec 21 07:16:43 blackbox-02.mysql kernel: [ 7425] 999 7425 8413054 5093955 12078 993166 0 mysqld Dec 21 07:16:43 blackbox-02.mysql kernel: [25704] 0 25704 33758 305 67 0 0 sshd Dec 21 07:16:43 blackbox-02.mysql kernel: [25707] 1005 25707 33758 310 64 0 0 sshd Dec 21 07:16:43 blackbox-02.mysql kernel: [25708] 1005 25708 28838 99 15 0 0 bash Dec 21 07:16:43 blackbox-02.mysql kernel: [25731] 0 25731 45635 157 46 0 0 sudo Dec 21 07:16:43 blackbox-02.mysql kernel: [25732] 0 25732 42792 462 39 0 0 mc Dec 21 07:16:43 blackbox-02.mysql kernel: [25734] 0 25734 28838 104 14 0 0 bash Dec 21 07:16:43 blackbox-02.mysql kernel: [25756] 0 25756 7559743 7023429 14015 21814 0 innobackupex Dec 21 07:16:43 blackbox-02.mysql kernel: [25757] 0 25757 12618 176 28 0 0 nc Dec 21 07:16:43 blackbox-02.mysql kernel: [28846] 89 28846 23084 272 43 0 0 pickup Dec 21 07:16:43 blackbox-02.mysql kernel: [29298] 0 29298 33758 301 67 2 0 sshd Dec 21 07:16:43 blackbox-02.mysql kernel: [29303] 1005 29303 33758 306 65 2 0 sshd Dec 21 07:16:43 blackbox-02.mysql kernel: [29304] 1005 29304 28838 99 14 0 0 bash Dec 21 07:16:43 blackbox-02.mysql kernel: [29327] 0 29327 45635 156 45 0 0 sudo Dec 21 07:16:43 blackbox-02.mysql kernel: [29328] 0 29328 30805 375 19 0 0 htop Dec 21 07:16:43 blackbox-02.mysql kernel: [29962] 997 29962 11742 72 26 120 0 nrpe Dec 21 07:16:43 blackbox-02.mysql kernel: [29963] 997 29963 11742 79 26 113 0 nrpe Dec 21 07:16:43 blackbox-02.mysql kernel: [29964] 997 29964 2 1 1 0 0 sh Dec 21 07:16:43 blackbox-02.mysql kernel: Out of memory: Kill process 25756 (innobackupex) score 512 or sacrifice child Dec 21 07:16:43 blackbox-02.mysql kernel: Killed process 25756 (innobackupex) total-vm:30238972kB, anon-rss:28093716kB, file-rss:0kB