In some cases xtrabackup generates some huge backups (huge ibdata1)

[Roel Van de Paar]

Original file is 76M, backup is 93G

[roel@qaserver master-data]$ pwd
/ssd/952373/vardir1_6/master-data
[roel@qaserver master-data]$ du -sh
394M .
[roel@qaserver master-data]$ cd ../master-data_xtrabackup_3358/
[roel@qaserver master-data_xtrabackup_3358]$ du -sh
93G .
[roel@qaserver master-data_xtrabackup_3358]$ ls -l
total 96517488
-rw-rw-r-- 1 roel roel 98828369920 Dec 28 15:05 ibdata1
-rw-rw---- 1 roel roel 2729778 Dec 28 14:53 ibdata1.qp
-rw-rw---- 1 roel roel 2768896 Dec 28 14:53 ib_logfile0
-rw-rw-r-- 1 roel roel 0 Dec 28 15:05 ib_logfile101
drwx------ 2 roel roel 4096 Dec 28 14:53 mysql
drwx------ 2 roel roel 4096 Dec 28 14:53 test
-rw-rw---- 1 roel roel 93 Dec 28 14:53 xtrabackup_checkpoints
[roel@qaserver master-data_xtrabackup_3358]$ ls -l ../master-data
total 176144
-rw-rw---- 1 roel roel 56 Dec 28 14:53 auto.cnf
-rw-rw---- 1 roel roel 79691776 Dec 28 14:53 ibdata1
-rw-rw---- 1 roel roel 50331648 Dec 28 14:52 ib_logfile0
-rw-rw---- 1 roel roel 50331648 Dec 28 14:52 ib_logfile1
drwxr-xr-x 2 roel roel 4096 Dec 28 14:52 mysql
drwxrwx--- 2 roel roel 4096 Dec 28 14:53 performance_schema
drwxr-xr-x 2 roel roel 4096 Dec 28 14:53 test
[roel@qaserver master-data_xtrabackup_3358]$ ls -lh ibdata1
-rw-rw-r-- 1 roel roel 93G Dec 28 15:05 ibdata1
[roel@qaserver master-data_xtrabackup_3358]$ ls -lh ../master-data/ibdata1
-rw-rw---- 1 roel roel 76M Dec 28 14:53 ../master-data/ibdata1

[Roel Van de Paar (roel11)]

Options used for xtrabackup:

# 2013-12-28T14:45:35 Executing backup: /ssd/xtrabackup-dbg-281213-1101/bin/xtrabackup --compress --backup --datadir=/ssd//952373/current1_2/master-data --target-dir=/ssd//952373/current1_2/master-data_xtrabackup_3358
# 2013-12-28T14:45:40 Preparing backup: /ssd/xtrabackup-dbg-281213-1101/bin/xtrabackup --compress --prepare --target-dir=/ssd//952373/current1_2/master-data_xtrabackup_3358
# 2013-12-28T14:45:40 Running second prepare: /ssd/xtrabackup-dbg-281213-1101/bin/xtrabackup --compress --prepare --target-dir=/ssd//952373/current1_2/master-data_xtrabackup_3358

[Roel Van de Paar (roel11)]

Another occurrence

[roel@qaserver master-data]$ ls -lh ibdata1
-rw-rw---- 1 roel roel 76M Dec 28 15:12 ibdata1
[roel@qaserver master-data]$ ls -lh ../master-data_xtrabackup_5781/ibdata1
-rw-rw-r-- 1 roel roel 39G Dec 28 15:17 ../master-data_xtrabackup_5781/ibdata1

Options used:

# 2013-12-28T15:08:32 Executing backup: /ssd/xtrabackup-dbg-281213-1101/bin/xtrabackup --compress --parallel=8 --backup --datadir=/ssd//952373/current1_4/master-data --target-dir=/ssd//952373/current1_4/master-data_xtrabackup_5781
# 2013-12-28T15:08:35 Preparing backup: /ssd/xtrabackup-dbg-281213-1101/bin/xtrabackup --compress --parallel=8 --prepare --target-dir=/ssd//952373/current1_4/master-data_xtrabackup_5781
# 2013-12-28T15:08:35 Running second prepare: /ssd/xtrabackup-dbg-281213-1101/bin/xtrabackup --compress --parallel=8 --prepare --target-dir=/ssd//952373/current1_4/master-data_xtrabackup_5781

[Roel Van de Paar (roel11)]

Discussed with Alexey on IRC. May be compiler option related. This build was generated with:

cmake -DBUILD_CONFIG=xtrabackup_release -DWITH_NDBCLUSTER=0 -DWITH_DEBUG=1 && make -j4

Alexey proposed using VERBOSE=1 to see compiler options, so re-ran compile (clean copy of clean/fresh tree) as:

cmake -DBUILD_CONFIG=xtrabackup_release -DWITH_NDBCLUSTER=0 -DWITH_DEBUG=1 && make -j4 VERBOSE=1

And mailed output to Alexey

[Roel Van de Paar (roel11)]

Changing build configuration worked. To build release, use:

opt: cmake -DBUILD_CONFIG=xtrabackup_release && make -j4
dbg: cmake -DWITH_DEBUG=ON && make -j4
val: cmake -DWITH_DEBUG=ON -DWITH_VALGRIND=1 && make -j4

I was caused by using -DBUILD_CONFIG=xtrabackup_release together -DWITH_DEBUG=ON or by using -DWITH_NDBCLUSTER=0. Either may cause the issue, or they may both cause it (to be tested later if needed).

[Alexey Kopytov (akopytov)]

According to the build log cmake -DBUILD_CONFIG=xtrabackup_release -DWITH_DEBUG=ON result in InnoDB being built in release configuration and XtraBackup in debug. Which is known to cause similar issues.

[Roel Van de Paar (roel11)]

The issue is still producible, though seemingly with less frequency then before.

[roel@qaserver 189264.xb]$ ls -lh vardir1_7/master-data/ibdata1
-rw-rw---- 1 roel roel 76M Dec 29 08:54 vardir1_7/master-data/ibdata1
[roel@qaserver 189264.xb]$ ls -lh vardir1_7/master-data_xtrabackup_30174/ibdata1
-rw-rw-r-- 1 roel roel 6.6G Dec 29 09:02 vardir1_7/master-data_xtrabackup_30174/ibdata1

[Roel Van de Paar (roel11)]

This was with a build generated by:
cmake -DWITH_DEBUG=ON && make -j4
Only

[Alexey Kopytov (akopytov)]

Was it a clean build?

[Roel Van de Paar (roel11)]

Yes. Did cp -R 2.2 2.2_dbg before building where 2.2 was a clean bzr pull.

[Alexey Kopytov (akopytov)]

This bug is still missing a reproducible case and/or test artifacts demonstrating the problem.

[Launchpad Janitor (janitor)]

[Expired for Percona XtraBackup 2.3 because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for Percona XtraBackup 2.2 because there has been no activity for 60 days.]

[Nick Le Mouton (7icq)]

I'm seeing this when using Percona xtrabackup 2.2.13 RPM. I have an ibdata1 file that keeps growing when preparing an incremental backup (in this case using Jervin Real's pyxbackup). I stop it before I run out of disk, but it got up to 1.1TB, my original full backup is 47GB, the incremental is 14GB.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1267