parallel option breaks incremental backups

Reported by Charl on 2011-08-15
22
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup
High
Alexey Kopytov
1.6
High
Alexey Kopytov
2.0
High
Alexey Kopytov

Bug Description

The --parallel option causes lots of different 'random' failures.

Overall the parallel option seems very unstable, the only way I can get working backups is to not use --parallel.
Without --parallel my backups and restores works.

I am using xtrabackup 1.6.2 with MySQL 5.5.10 on Ubuntu server 10.04 64 bit.
My server is configured as "innodb_file_per_table" with 4 databases and +- 200 tables and overall 70GB of data.

--------------------------------------

An incremental backup with --parallel=4 succeeds without any errors.

But when later trying to restore such backups there are lots of corrupted .delta files.
Error messages look like this during the restore script fails:

Applying /media/data/backup/active_incremental/bug_genie2/bugs2_components.ibd.delta ...
xtrabackup: error: /media/data/backup/active_incremental/bug_genie2/bugs2_components.ibd.delta seems not .delta file.
xtrabackup: Error: xtrabackup_apply_delta(): failed to apply /media/data/backup/active_incremental/bug_genie2/bugs2_components.ibd.delta to /media/data/backup/active_restore/bug_genie2/bugs2_components.ibd.
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 336.

---------------------------------------

Performing full backups with --parallel=4 also causes the backups to fail at random places.
Error messages then looks like this:

>> log scanned up to (843532850219)
>> log scanned up to (843532850219)
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 336.

Stewart Smith (stewart) on 2011-10-20
Changed in percona-xtrabackup:
importance: Undecided → High
Changed in percona-xtrabackup:
assignee: nobody → Patrick Crews (patrick-crews)
Patrick Crews (patrick-crews) wrote :

Is your server under load when this is happening?
Also, if this is the case, could you describe the load (SELECT / INSERT / UPDATE / DELETE)? This will help us in duplicating / diagnosing the problem greatly.

Changed in percona-xtrabackup:
status: New → In Progress
Patrick Crews (patrick-crews) wrote :
Download full text (5.1 KiB)

NOTE: not 100% certain this is the exact same bug, but this behavior was detected while trying to duplicate it. Appending note to existing bug per discussion with developer

Able to see a similar failure, but *only* under the following conditions:
1) 7 schemas (all identical). Each schema has 100 tables, Tables are 1-100 rows (table1 has 1 row, table 100 has 100 rows)
2) innodb_file_per_table is enabled
3) --parallel=50 passed to xtrabackup.

Reducing either the number of schemas or the --parallel value results in a passing test.

However, with this command line:
./dbqp.py --default-server-type=mysql --basedir=/mysql-5.5 --xtrabackup-path=/percona-xtrabackup/mysql-5.5/storage/innobase/xtrabackup/xtrabackup_innodb55 --innobackupex-path=/percona-xtrabackup/innobackupex --suite=xtrabackup_basic bug826632_test

We see this:
20111101-103351 ===============================================================
20111101-103351 TEST NAME [ RESULT ] TIME (ms)
20111101-103351 ===============================================================
20111101-103351 xtrabackup_basic.bug826632_test [ fail ] 88564
20111101-103351 test_basic1 (bug826632_test.basicTest) ... FAIL
20111101-103351
20111101-103351 ======================================================================
20111101-103351 FAIL: test_basic1 (bug826632_test.basicTest)
20111101-103351 ----------------------------------------------------------------------
20111101-103351 Traceback (most recent call last):
20111101-103351 File "/dbqp/percona_tests/xtrabackup_basic/bug826632_test.py", line 99, in test_basic1
20111101-103351 self.assertTrue(retcode==0,output)
20111101-103351 AssertionError:
20111101-103351 InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
20111101-103351 and Percona Inc 2009-2011. All Rights Reserved.
20111101-103351
20111101-103351 This software is published under
20111101-103351 the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
20111101-103351
20111101-103351 IMPORTANT: Please check that the apply-log run completes successfully.
20111101-103351 At the end of a successful apply-log run innobackupex
20111101-103351 prints "completed OK!".
20111101-103351
20111101-103351
20111101-103351
20111101-103351 111101 10:33:51 innobackupex: Starting ibbackup with command: /percona-xtrabackup/mysql-5.5/storage/innobase/xtrabackup/xtrabackup_innodb55 --prepare --target-dir=/dbqp/workdir/bot0/s0/var/_xtrabackup --use-memory=500M
20111101-103351
20111101-103351 xtrabackup: cd to /dbqp/workdir/bot0/s0/var/_xtrabackup
20111101-103351 xtrabackup: This target seems to be not prepared yet.
20111101-103351 xtrabackup: Temporary instance for recovery is set as followings.
20111101-103351 xtrabackup: innodb_data_home_dir = ./
20111101-103351 xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
20111101-103351 xtrabackup: innodb_log_group_home_dir = ./
20111101-103351 xtrabackup: innodb_log_files_in_group = 1
20111101-103351 xtrabackup: innodb_log_file_size = 2097152
20111101-103351 111101 10:33:51 InnoDB: Using Linux native AIO
20111101-103351 xtrabackup: Starting InnoDB ...

Read more...

Patrick Crews (patrick-crews) wrote :

It should be noted that all tables have the same names across schemas:
<schema>.table1_int_autoinc
<schema>.table2_int_autoinc
...

Patrick Crews (patrick-crews) wrote :

Additionally, could you include the commands / steps you are taking in preparing your incremental backups?
This will also help in duplicating / troubleshooting the issue

Charl (charlretief) wrote :

1) I have 2 severs with the same software setup. A production server and a development server. The only difference between these machines is the hardware. I have the exact same failures on both setups.
1.1) The production server is under load during these failures
1.2) The development sever is not under load, and has no MySQL connections during these failures
My conclusion here is it is not a hardware or server load issue.

2) I get the commands I am using to backup and prepare/restore and post them soon

Charl (charlretief) wrote :
Charl (charlretief) wrote :
Charl (charlretief) wrote :
Charl (charlretief) wrote :

The above files contains shell scripts for backup/restore commands I tried with parrallel=4 where the backups get corrupted.

To use it
1) first adjust the parameters at the top of each file to fit your file locations etc.
2) Then create a full backup
3) Create a few incremental after that (sometimes the first few succeeds).
4) Then try to restore the backups

Example usage for #2-4:

bash mysql_backup_full
bash mysql_backup_incremental
bash mysql_backup_incremental
bash mysql_backup_incremental
bash mysql_backup_incremental
bash mysql_backup_incremental
bash mysql_backup_incremental
bash mysql_backup_restore

Alexey Kopytov (akopytov) wrote :

Changing the summary to reflect this bug is about issues with parallel + incremental. The problem with full parallel backups is likely the same as in bug #900175 and will be tracked there.

summary: - parallel option breaks backups and restores
+ parallel option breaks incremental backups
Changed in percona-xtrabackup:
assignee: Patrick Crews (patrick-crews) → Alexey Kopytov (akopytov)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers