Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC

xtrabackup donor runs xtrabackup indefinitely when xtrabackup tmpdir on tmpfs

Bug #1086978 reported by Jay Janssen on 2012-12-05

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	MySQL patches by Codership	Fix Released	Low	Alex Yurchenko	MySQL patches by Codership 5.5.31-23.7.5
	Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC	Fix Released	Undecided	Unassigned	Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC 5.5.30-23.7.4

Bug Description

# whoami
mysql
# /usr//bin/innobackupex --galera-info --tmpdir=/tmp --stream=tar --defaults-file=/etc/my.cnf --socket=/var/lib/mysql/mysql.sock --user=sst --password=xxxxxxxxxx /tmp > backup.tar

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

121205 12:07:10 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:07:10 innobackupex: Connected to database with mysql child process (pid=6933)
121205 12:07:16 innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints "completed OK!".

innobackupex: Using mysql Ver 14.14 Distrib 5.5.28, for Linux (x86_64) using readline 5.1
innobackupex: Using mysql server version Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

innobackupex: Created backup directory /tmp
121205 12:07:16 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:07:16 innobackupex: Connected to database with mysql child process (pid=6964)
121205 12:07:18 innobackupex: Connection to database server closed

121205 12:07:18 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp --stream=tar
innobackupex: Waiting for ibbackup (pid=6972) to suspend
innobackupex: Suspend file '/tmp/xtrabackup_suspended'

xtrabackup_55 version 2.0.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: 470)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /mnt/data
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 5242880
121205 12:07:18 InnoDB: Using Linux native AIO
xtrabackup: using O_DIRECT
>> log scanned up to (41681167645)
[01] Streaming ./ibdata1
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
[01] ...done
121205 12:10:15 InnoDB: Failed to set O_DIRECT on file /tmp/xtrabackup_suspended: OVERWRITE: Invalid argument, continuing anyway
121205 12:10:15 InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662

121205 12:10:16 innobackupex: Continuing after ibbackup has suspended
121205 12:10:16 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:10:16 innobackupex: Connected to database with mysql child process (pid=7172)
121205 12:10:18 innobackupex: Starting to lock all tables...
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
121205 12:10:32 innobackupex: All tables locked and flushed to disk

121205 12:10:32 innobackupex: Starting to backup .frm, .MRG, .MYD, .MYI,
innobackupex: .TRG, .TRN, .ARM, .ARZ, .CSM, .CSV and .opt files in
innobackupex: subdirectories of '/mnt/data'
innobackupex: Backing up files '/mnt/data/wiredrive_ci/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (103 files)
innobackupex: Backing up files '/mnt/data/performance_schema/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (18 files)
innobackupex: Backing up file '/mnt/data/percona@002dtoolkit/dsns.frm'
innobackupex: Backing up file '/mnt/data/percona@002dtoolkit/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (85 files)
innobackupex: Backing up file '/mnt/data/blah/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive_qed/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (83 files)
>> log scanned up to (41681167645)
innobackupex: Backing up files '/mnt/data/mysql/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (72 files)
innobackupex: Backing up file '/mnt/data/testpls/example.frm'
innobackupex: Backing up file '/mnt/data/testpls/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive_qa/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (103 files)
121205 12:10:34 innobackupex: Finished backing up .frm, .MRG, .MYD, .MYI, .TRG, .TRN, .ARM, .ARZ, .CSV, .CSM and .opt files

innobackupex: Resuming ibbackup

xtrabackup: The latest check point (for incremental): '41681167491'
xtrabackup: Stopping log copying thread.
.>> log scanned up to (41681167645)

xtrabackup: Streaming transaction log from a temporary file...
xtrabackup: Done.
xtrabackup: Transaction log of lsn (41681167491) to (41681167645) was copied.
121205 12:10:37 innobackupex: All tables unlocked
121205 12:10:37 innobackupex: Connection to database server closed

innobackupex: Backup created in directory '/tmp'
innobackupex: MySQL binlog position: filename '', position
innobackupex: You must use -i (--ignore-zeros) option for extraction of the tar stream.
121205 12:10:37 innobackupex: completed OK!

Note the errors related to /tmp.

# mount
tmpfs on /tmp type tmpfs (rw,noatime,mode=1777)

I do have an appropriately sized backup.tar at the end of this. This same error happens in the innobackup.backup.log on a donor, or if I invoke xtrabackup manually on the donor.

If I use a different tmpdir on ext4, then I do not get these errors:

-bash-4.1$ /usr//bin/innobackupex --galera-info --tmpdir=/tmpdir --stream=tar --defaults-file=/etc/my.cnf --socket=/var/lib/mysql/mysql.sock --user=sst --password=9OOgzSVOI95ko /tmpdir > backup.tar

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

121205 12:16:55 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:16:55 innobackupex: Connected to database with mysql child process (pid=8444)
121205 12:17:01 innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints "completed OK!".

innobackupex: Created backup directory /tmpdir
121205 12:17:01 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:17:01 innobackupex: Connected to database with mysql child process (pid=8475)
121205 12:17:03 innobackupex: Connection to database server closed

121205 12:17:03 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmpdir --stream=tar
innobackupex: Waiting for ibbackup (pid=8483) to suspend
innobackupex: Suspend file '/tmpdir/xtrabackup_suspended'

>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
[01] ...done
>> log scanned up to (41681167645)
121205 12:20:05 innobackupex: Continuing after ibbackup has suspended
121205 12:20:05 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:20:05 innobackupex: Connected to database with mysql child process (pid=8676)
121205 12:20:07 innobackupex: Starting to lock all tables...
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
121205 12:20:21 innobackupex: All tables locked and flushed to disk

121205 12:20:21 innobackupex: Starting to backup .frm, .MRG, .MYD, .MYI,
innobackupex: .TRG, .TRN, .ARM, .ARZ, .CSM, .CSV and .opt files in
innobackupex: subdirectories of '/mnt/data'
innobackupex: Backing up files '/mnt/data/wiredrive_ci/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (103 files)
innobackupex: Backing up files '/mnt/data/performance_schema/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (18 files)
innobackupex: Backing up file '/mnt/data/percona@002dtoolkit/dsns.frm'
innobackupex: Backing up file '/mnt/data/percona@002dtoolkit/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (85 files)
innobackupex: Backing up file '/mnt/data/blah/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive_qed/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (83 files)
innobackupex: Backing up files '/mnt/data/mysql/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (72 files)
innobackupex: Backing up file '/mnt/data/testpls/example.frm'
innobackupex: Backing up file '/mnt/data/testpls/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive_qa/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (103 files)
>> log scanned up to (41681167645)
121205 12:20:24 innobackupex: Finished backing up .frm, .MRG, .MYD, .MYI, .TRG, .TRN, .ARM, .ARZ, .CSV, .CSM and .opt files

innobackupex: Resuming ibbackup

xtrabackup: The latest check point (for incremental): '41681167491'
xtrabackup: Stopping log copying thread.
.>> log scanned up to (41681167645)

xtrabackup: Streaming transaction log from a temporary file...
xtrabackup: Done.
xtrabackup: Transaction log of lsn (41681167491) to (41681167645) was copied.
121205 12:20:26 innobackupex: All tables unlocked
121205 12:20:26 innobackupex: Connection to database server closed

innobackupex: Backup created in directory '/tmpdir'
innobackupex: MySQL binlog position: filename '', position
innobackupex: You must use -i (--ignore-zeros) option for extraction of the tar stream.
121205 12:20:26 innobackupex: completed OK!

Related branches

lp:codership-mysql/5.5

Revision history for this message

Alex Yurchenko (ayurchen) wrote on 2012-12-06:

Committed a workaround in lp:codership-mysql r3831: made wsrep_sst_xtrabackup script to try to use tmpdir setting from my.cnf file for its temporary directory. So it is possible to specify tmpdir different from /tmp.

But frankly, I'm quite surprised this is necessary.

Changed in codership-mysql:
assignee:	nobody → Alex Yurchenko (ayurchen)
importance:	Undecided → Low
milestone:	none → 5.5.28-23.8
status:	New → Fix Committed

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-03-28:

Xtrabackup can use tmpdir set in my.cnf under [xtrabackup], which
is why there is no need for this. Also, see the comments in
lp:1160047

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-04-08:

Updated lp:1160047 with further details.

Raghavendra D Prabhu (raghavendra-prabhu) on 2013-04-08

Changed in percona-xtradb-cluster:
status:	New → Fix Released
milestone:	none → 5.5.30-23.7.4

Alex Yurchenko (ayurchen) on 2013-05-31

Changed in codership-mysql:
milestone:	5.5.30-24.8 → 5.5.31-23.7.4

Alex Yurchenko (ayurchen) on 2013-06-29

Changed in codership-mysql:
status:	Fix Committed → Fix Released

Revision history for this message

Shahriyar Rzayev (rzayev-sehriyar) wrote on 2018-01-18:

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1268

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.