Concurrent XtraBackup instances fail with same --tmpdir
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Percona XtraBackup moved to https://jira.percona.com/projects/PXB |
Triaged
|
Medium
|
Unassigned | ||
2.0 |
Won't Fix
|
Undecided
|
Unassigned | ||
2.1 |
Won't Fix
|
Undecided
|
Unassigned | ||
2.2 |
Triaged
|
Medium
|
Unassigned | ||
2.3 |
Triaged
|
Medium
|
Unassigned |
Bug Description
Hey,
I have a script that uses innobackupex to backup MariaDB 5.5.32 database instances (different datadirs and
separate processes). One script per database instance.
2 backups had been scheduled to be run at the same time from 2 different instances at 1:00 AM
I have rescheduled the backups to different times to see if it has any difference.
Since 2.1.4 version, even in 2.1.5, backups sometimes don't get finished like normally
and innobackupex returns an error code.
innobackupex failed:
---------- ERROR OUTPUT from innobackupex ----------
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
131019 01:00:01 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:
131019 01:00:01 innobackupex: Connected to MySQL server
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints "completed OK!".
innobackupex: Using mysql server version 5.5.32-MariaDB-log
innobackupex: Created backup directory /mariadb1/
131019 01:00:01 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-
innobackupex: Waiting for ibbackup (pid=27051) to suspend
innobackupex: Suspend file '/mariadb1/
xtrabackup_55 version 2.1.5 for Percona Server 5.5.31 Linux (x86_64) (revision id: 680)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /mariadb1/data
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: using O_DIRECT
>> log scanned up to (21810149)
[01] Copying /mariadb1/
[01] ...done
>> log scanned up to (21810149)
xtrabackup: Creating suspend file '/mariadb1/
print() on closed filehandle XTRABACKUP_PID at /mariabin/
131019 01:00:03 innobackupex: Waiting for ibbackup (pid=27051) to finish
So xtrabackup creates an suspend file, and innobackupex starts waiting on ibbackup.
Then there's a print() on closed filehandle.
When I check these processes they're no longer present during the day time.
A bug or two backups cant run at the same time limitation in innobackupex?
In innobackupex code:
open XTRABACKUP_PID, "> $option_ tmpdir/ $xtrabackup_ pid_file" ;
Could this suggests that two simultaneous runs of innobackupex will write to same tmp/xtrabackup_ pid_fie
when running ibbackup ?
Which could be avoided with: --tmpdir option ?
When I tested by running 2 copies of the backup script copies of different instances at the same time I didnt have issues.
Only when the logs are being scanned as suggested by this I have the problem:
xtrabackup: using O_DIRECT data/ibdata1 to /mariadb1/ backup/ 2013-10- 19_01-00- 01/ibdata1 backup/ 2013-10- 19_01-00- 01/xtrabackup_ suspended_ 2' with pid '27051' percona- xtrabackup/ bin/innobackupe x line 2602.
>> log scanned up to (21810149)
[01] Copying /mariadb1/
[01] ...done
>> log scanned up to (21810149)
xtrabackup: Creating suspend file '/mariadb1/
print() on closed filehandle XTRABACKUP_PID at /mariabin/
131019 01:00:03 innobackupex: Waiting for ibbackup (pid=27051) to finish
-----