xb_parallel_incremental test sporadically fails

Reported by Stewart Smith on 2012-11-22
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraBackup
High
Alexey Kopytov
2.0
High
Alexey Kopytov
2.1
High
Alexey Kopytov

Bug Description

from http://jenkins.percona.com/job/percona-xtrabackup-2.0/959/BUILD_TYPE=debug,Host=centos6-32,xtrabackuptarget=innodb51/testReport/junit/%28root%29/t_xb_parallel_incremental/sh/

121121 23:15:12 innobackupex: Starting ibbackup with command: /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/src/xtrabackup_plugin --defaults-file="/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/test/var1/my.cnf" --defaults-group="mysqld" --prepare --target-dir=/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/test/var1/full_backup --apply-log-only --incremental-dir=/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/test/var1/inc_backup

/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/src/xtrabackup_plugin version 2.0.3 for MySQL server 5.1.59 pc-linux-gnu (i686) (revision id: undefined)
incremental backup from 6809259 is enabled.
xtrabackup: cd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/test/var1/full_backup
xtrabackup: This target seems to be already prepared.
xtrabackup: xtrabackup_logfile detected: size=7716864, start_lsn=(14790201)
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/test/var1/inc_backup
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 7716864
121121 23:15:12 InnoDB: Warning: allocated tablespace 4, old maximum was 0

<SNIP>

InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 6486322, file name ./mysql-bin.000001
121121 23:15:26 InnoDB: Assertion failure in thread 3078010560 in file buf/buf0lru.c line 908
InnoDB: Failing assertion: buf_LRU_old_ratio >= 51
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
innobackupex: Error:
innobackupex: ibbackup failed at /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos6-32/xtrabackuptarget/innodb51/innobackupex line 378.
2012-11-21 23:15:26: xb_parallel_incremental.sh: ===> innobackupex failed with exit code 255
2012-11-21 23:15:26: run.sh: Found a leftover mysqld processes with PID 29002, stopping it
------------

Also from http://jenkins.percona.com/job/percona-xtrabackup-2.0/954/BUILD_TYPE=release,Host=centos5-64,xtrabackuptarget=innodb51/testReport/junit/%28root%29/t_xb_parallel_incremental/sh/

121119 05:21:12 innobackupex: Starting ibbackup with command: /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/src/xtrabackup_plugin --defaults-file="/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup --incremental-basedir='/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/full_backup' --parallel=8
innobackupex: Waiting for ibbackup (pid=10057) to suspend
innobackupex: Suspend file '/home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/xtrabackup_suspended'

<SNIP>

[05] Copying ./sakila/rental.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/rental.ibd.delta
[04] Copying ./sakila/inventory.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/inventory.ibd.delta
[03] Copying ./sakila/country.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/country.ibd.delta
[07] Copying ./sakila/store.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/store.ibd.delta
[08] Copying ./sakila/category.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/category.ibd.delta
[06] Copying ./sakila/address.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/address.ibd.delta
[07] ...done
[06] ...done
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[05] xtrabackup: Database page corruption detected at page 64. retrying...
[04] ...done
[05] xtrabackup: Error: 10 retries resulted in fail. File ./sakila/rental.ibd seems to be corrupted.
[05] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[05] xtrabackup: Error: failed to copy datafile.
innobackupex: Error: ibbackup child process has died at /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/release/Host/centos5-64/xtrabackuptarget/innodb51/innobackupex line 378.
2012-11-19 05:21:19: xb_parallel_incremental.sh: ===> innobackupex failed with exit code 2
2012-11-19 05:21:19: run.sh: Found a leftover mysqld processes with PID 9048, stopping it
2012-11-19 05:21:19: run.sh: Found a leftover mysqld processes with PID 8844, stopping it
------------

Related branches

Stewart Smith (stewart) wrote :

http://jenkins.percona.com/job/percona-xtrabackup-2.0/949/BUILD_TYPE=debug,Host=centos5-64,xtrabackuptarget=innodb51/testReport/junit/%28root%29/t_xb_parallel_incremental/sh/

also shows a failure similar to the second one:

[07] Copying ./sakila/store.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/store.ibd.delta
[03] Copying ./sakila/film.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/film.ibd.delta
[01] Copying ./ibdata1 to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/ibdata1.delta
[02] Copying ./sakila/inventory.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/inventory.ibd.delta
[05] Copying ./sakila/rental.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/rental.ibd.delta
[06] Copying ./sakila/address.ibd to /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/test/var1/inc_backup/./sakila/address.ibd.delta
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[06] ...done
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Database page corruption detected at page 3. retrying...
[07] xtrabackup: Error: 10 retries resulted in fail. File ./sakila/store.ibd seems to be corrupted.
[07] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[07] xtrabackup: Error: failed to copy datafile.
innobackupex: Error: ibbackup child process has died at /home/jenkins/workspace/percona-xtrabackup-2.0/BUILD_TYPE/debug/Host/centos5-64/xtrabackuptarget/innodb51/innobackupex line 378.
2012-10-23 22:42:43: xb_parallel_incremental.sh: ===> innobackupex failed with exit code 2
2012-10-23 22:42:43: run.sh: Found a leftover mysqld processes with PID 4214, stopping it
2012-10-23 22:42:43: run.sh: Found a leftover mysqld processes with PID 7601, stopping it

Alexey Kopytov (akopytov) wrote :

The "buf_LRU_old_ratio >= 51" is a duplicate of bug #924492. Let's keep this bug as a report about corrupted pages error in xb_parallel_incremental.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers