Concurrent XtraBackup instances fail with same --tmpdir

[viljohakala]

Hey,

I have a script that uses innobackupex to backup MariaDB 5.5.32 database instances (different datadirs and
separate processes). One script per database instance.

2 backups had been scheduled to be run at the same time from 2 different instances at 1:00 AM
I have rescheduled the backups to different times to see if it has any difference.

Since 2.1.4 version, even in 2.1.5, backups sometimes don't get finished like normally
and innobackupex returns an error code.

innobackupex failed:

---------- ERROR OUTPUT from innobackupex ----------

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

131019 01:00:01 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/mariadb1/conf/my.cnf;mysql_read_default_group=xtrabackup' as 'root' (using password: YES).
131019 01:00:01 innobackupex: Connected to MySQL server
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using mysql server version 5.5.32-MariaDB-log

innobackupex: Created backup directory /mariadb1/backup/2013-10-19_01-00-01

131019 01:00:01 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/mariadb1/conf/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/mariadb1/backup/2013-10-19_01-00-01 --tmpdir=/var/tmp
innobackupex: Waiting for ibbackup (pid=27051) to suspend
innobackupex: Suspend file '/mariadb1/backup/2013-10-19_01-00-01/xtrabackup_suspended_2'

xtrabackup_55 version 2.1.5 for Percona Server 5.5.31 Linux (x86_64) (revision id: 680)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /mariadb1/data
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = /mariadb1/data
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = /mariadb1/data
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 134217728
xtrabackup: using O_DIRECT
>> log scanned up to (21810149)
[01] Copying /mariadb1/data/ibdata1 to /mariadb1/backup/2013-10-19_01-00-01/ibdata1
[01] ...done
>> log scanned up to (21810149)
xtrabackup: Creating suspend file '/mariadb1/backup/2013-10-19_01-00-01/xtrabackup_suspended_2' with pid '27051'
print() on closed filehandle XTRABACKUP_PID at /mariabin/percona-xtrabackup/bin/innobackupex line 2602.
131019 01:00:03 innobackupex: Waiting for ibbackup (pid=27051) to finish

So xtrabackup creates an suspend file, and innobackupex starts waiting on ibbackup.
Then there's a print() on closed filehandle.

When I check these processes they're no longer present during the day time.

A bug or two backups cant run at the same time limitation in innobackupex?

[viljohakala (viljo-hakala)]

In innobackupex code:

    open XTRABACKUP_PID, "> $option_tmpdir/$xtrabackup_pid_file";

Could this suggests that two simultaneous runs of innobackupex will write to same tmp/xtrabackup_pid_fie
when running ibbackup ?

Which could be avoided with: --tmpdir option ?

When I tested by running 2 copies of the backup script copies of different instances at the same time I didnt have issues.

Only when the logs are being scanned as suggested by this I have the problem:

xtrabackup: using O_DIRECT
>> log scanned up to (21810149)
[01] Copying /mariadb1/data/ibdata1 to /mariadb1/backup/2013-10-19_01-00-01/ibdata1
[01] ...done
>> log scanned up to (21810149)
xtrabackup: Creating suspend file '/mariadb1/backup/2013-10-19_01-00-01/xtrabackup_suspended_2' with pid '27051'
print() on closed filehandle XTRABACKUP_PID at /mariabin/percona-xtrabackup/bin/innobackupex line 2602.
131019 01:00:03 innobackupex: Waiting for ibbackup (pid=27051) to finish
-----

[viljohakala (viljo-hakala)]

Could this be related:

 innobackupex should remove stale xtrabackup_suspended file on start Edit

    Percona XtraBackup
    Bugs
    Bug #1007446

[Alexey Kopytov (akopytov)]

Multiple concurrent XtraBackup instances do not work correctly with the same --tmpdir, even when running against different server instances. Which is a known issue, but it seems like it has not been reported anywhere. Let's keep this report as a request to fix this.

See also bug #1071938.

[Alexey Kopytov (akopytov)]

This is also not a recent regression -- the same limitation applies to 2.0, but it is unlikely it will be fixed in that branch.

[viljohakala (viljo-hakala)]

Hi,

it seems that i cant pass --tmpdir to innobackupex.

Basically tmpdir is read from my.cnf tmpdir variable, right?

> Alexey Kopytov <email address hidden> kirjoitti 21.10.2013 kello 12.41:
>
> This is also not a recent regression -- the same limitation applies to
> 2.0, but it is unlikely it will be fixed in that branch.
>
> ** Changed in: percona-xtrabackup/2.1
> Status: New => Triaged
>
> ** Changed in: percona-xtrabackup/2.1
> Importance: Undecided => Medium
>
> ** Changed in: percona-xtrabackup/2.2
> Status: New => Triaged
>
> ** Changed in: percona-xtrabackup/2.2
> Importance: Undecided => Medium
>
> ** Also affects: percona-xtrabackup/2.0
> Importance: Undecided
> Status: New
>
> ** Changed in: percona-xtrabackup/2.0
> Status: New => Won't Fix
>
> ** Summary changed:
>
> - Xtrabackup 2.1.5 (since 2.1.4) leaves unfinished backups around
> + Concurrent XtraBackup instances fail with same --tmpdir
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1241987
>
> Title:
> Concurrent XtraBackup instances fail with same --tmpdir
>
> Status in Percona XtraBackup:
> Triaged
> Status in Percona XtraBackup 2.0 series:
> Won't Fix
> Status in Percona XtraBackup 2.1 series:
> Triaged
> Status in Percona XtraBackup 2.2 series:
> Triaged
>
> Bug description:
> Hey,
>
> I have a script that uses innobackupex to backup MariaDB 5.5.32 database instances (different datadirs and
> separate processes). One script per database instance.
>
> 2 backups had been scheduled to be run at the same time from 2 different instances at 1:00 AM
> I have rescheduled the backups to different times to see if it has any difference.
>
> Since 2.1.4 version, even in 2.1.5, backups sometimes don't get finished like normally
> and innobackupex returns an error code.
>
>
> innobackupex failed:
>
> ---------- ERROR OUTPUT from innobackupex ----------
>
> InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
> and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
>
> This software is published under
> the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
>
> 131019 01:00:01 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/mariadb1/conf/my.cnf;mysql_read_default_group=xtrabackup' as 'root' (using password: YES).
> 131019 01:00:01 innobackupex: Connected to MySQL server
> IMPORTANT: Please check that the backup run completes successfully.
> At the end of a successful backup run innobackupex
> prints "completed OK!".
>
> innobackupex: Using mysql server version 5.5.32-MariaDB-log
>
> innobackupex: Created backup directory
> /mariadb1/backup/2013-10-19_01-00-01
>
> 131019 01:00:01 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/mariadb1/conf/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/mariadb1/backup/2013-10-19_01-00-01 --tmpdir=/var/tmp
> innobackupex: Waiting for ibbackup (pid=27051) to suspend
> innobackupex: Suspend file '/mariadb1/backup/2013-10-19_01-00-...

[Alexey Kopytov (akopytov)]

No, the --tmpdir option to innobackupex overrides the one read from my.cnf.

[Alexey Kopytov (akopytov)]

Bug #1408891 is a duplicate of this one.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-669