innobackupex invoke OOMkiller when using lock-ddl-per-table

[Pavel Akhmetshin]

Story.
I have a large MariaDB-10.2.8 InnoDB installation (850Gb), with multimaster replication and also act as master for it own databases.

When i try do backup i get:

# innobackupex --user backup --password PASS --stream=tar ./ | nc blackbox-03.mysql 7000
...
InnoDB: Last flushed lsn: 12412902447068 load_index lsn 12414483598158
InnoDB: An optimized (without redo logging) DDLoperation has been performed. All modified pages may not have been flushed to the disk yet.
PXB will not be able take a consistent backup. Retry the backup operation.

MariaDB does not support LOCK TABLES FOR BACKUP asn i can't use --lock-ddl
I try use:
innobackupex --lock-ddl-per-table --user backup --password PASS --stream=tar ./ | nc blackbox-03.mysql 7000

And some time laster it invoke OOMkiller. My VM have 48Gb memory and 22Gb usually is free.

Affected version:
percona-xtrabackup-24-2.4.8-1.el7.x86_64
percona-xtrabackup-24-2.4.9-1.el7.x86_64

So, i can't do backup any way i choose.

[Shahriyar Rzayev (rzayev-sehriyar)]

Hi,
Could you please paste the output of dmesg, to see the OOM killer message?

[Pavel Akhmetshin (linko22)]

Before it dies:

  1 [|||| 2.7%] Tasks: 43, 109 thr; 1 running
  2 [|||||||| 6.2%] Load average: 1.86 1.98 1.80
  3 [||||||||||||||||||| 16.9%] Uptime: 83 days, 23:12:45
  4 [|||||||||||| 10.1%]
  Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||45.7G/47.0G]
  Swp[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||4.00G/4.00G]

  PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
25704 root 20 0 131M 1236 20 S 0.0 0.0 0:00.18 │ └─ sshd: linko [priv]
25707 linko 20 0 131M 1240 0 S 0.0 0.0 0:00.57 │ └─ sshd: linko@pts/1
25708 linko 20 0 112M 396 0 S 0.0 0.0 0:00.01 │ └─ -bash
25731 root 20 0 178M 640 12 S 0.0 0.0 0:00.00 │ └─ sudo mc
25732 root 20 0 167M 1848 0 S 0.0 0.0 0:00.25 │ └─ mc
25734 root 20 0 112M 416 4 S 0.0 0.0 0:00.00 │ └─ bash -rcfile .bashrc
25757 root 20 0 50472 704 0 S 0.0 0.0 2:41.92 │ ├─ nc blackbox-03.mysql 7000
25756 root 20 0 27.5G 25.5G 256 S 2.0 54.2 25:48.73 │ └─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./
25769 root 20 0 27.5G 25.5G 256 S 0.7 54.2 22:40.48 │ ├─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./
25768 root 20 0 27.5G 25.5G 256 S 0.0 54.2 0:00.00 │ ├─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./
25767 root 20 0 27.5G 25.5G 256 S 0.0 54.2 0:00.00 │ ├─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./
25766 root 20 0 27.5G 25.5G 256 S 0.0 54.2 0:00.00 │ ├─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./
25765 root 20 0 27.5G 25.5G 256 S 0.0 54.2 0:00.00 │ ├─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./
25764 root 20 0 27.5G 25.5G 256 S 1.4 54.2 3:06.15 │ └─ innobackupex --lock-ddl-per-table --user backup --password x xxxxxxxxxxxxx --stream=tar ./

/var/log/messages:
Dec 21 07:16:43 blackbox-02.mysql kernel: sh invoked oom-killer: gfp_mask=0x84d0, order=0, oom_score_adj=0
Dec 21 07:16:43 blackbox-02.mysql kernel: sh cp...

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1491