Percona XtraDB Cluster - HA scalable solution for MySQL

xtrabackup donor runs xtrabackup indefinitely when xtrabackup tmpdir on tmpfs

Reported by Jay Janssen on 2012-12-05
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MySQL patches by Codership
Low
Alex Yurchenko
Percona XtraDB Cluster
Undecided
Unassigned

Bug Description

# whoami
mysql
# /usr//bin/innobackupex --galera-info --tmpdir=/tmp --stream=tar --defaults-file=/etc/my.cnf --socket=/var/lib/mysql/mysql.sock --user=sst --password=xxxxxxxxxx /tmp > backup.tar

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Inc 2009-2012. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

121205 12:07:10 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:07:10 innobackupex: Connected to database with mysql child process (pid=6933)
121205 12:07:16 innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using mysql Ver 14.14 Distrib 5.5.28, for Linux (x86_64) using readline 5.1
innobackupex: Using mysql server version Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

innobackupex: Created backup directory /tmp
121205 12:07:16 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:07:16 innobackupex: Connected to database with mysql child process (pid=6964)
121205 12:07:18 innobackupex: Connection to database server closed

121205 12:07:18 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp --stream=tar
innobackupex: Waiting for ibbackup (pid=6972) to suspend
innobackupex: Suspend file '/tmp/xtrabackup_suspended'

xtrabackup_55 version 2.0.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: 470)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /mnt/data
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 5242880
121205 12:07:18 InnoDB: Using Linux native AIO
xtrabackup: using O_DIRECT
>> log scanned up to (41681167645)
[01] Streaming ./ibdata1
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
[01] ...done
121205 12:10:15 InnoDB: Failed to set O_DIRECT on file /tmp/xtrabackup_suspended: OVERWRITE: Invalid argument, continuing anyway
121205 12:10:15 InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662

121205 12:10:16 innobackupex: Continuing after ibbackup has suspended
121205 12:10:16 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:10:16 innobackupex: Connected to database with mysql child process (pid=7172)
121205 12:10:18 innobackupex: Starting to lock all tables...
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
121205 12:10:32 innobackupex: All tables locked and flushed to disk

121205 12:10:32 innobackupex: Starting to backup .frm, .MRG, .MYD, .MYI,
innobackupex: .TRG, .TRN, .ARM, .ARZ, .CSM, .CSV and .opt files in
innobackupex: subdirectories of '/mnt/data'
innobackupex: Backing up files '/mnt/data/wiredrive_ci/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (103 files)
innobackupex: Backing up files '/mnt/data/performance_schema/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (18 files)
innobackupex: Backing up file '/mnt/data/percona@002dtoolkit/dsns.frm'
innobackupex: Backing up file '/mnt/data/percona@002dtoolkit/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (85 files)
innobackupex: Backing up file '/mnt/data/blah/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive_qed/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (83 files)
>> log scanned up to (41681167645)
innobackupex: Backing up files '/mnt/data/mysql/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (72 files)
innobackupex: Backing up file '/mnt/data/testpls/example.frm'
innobackupex: Backing up file '/mnt/data/testpls/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive_qa/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (103 files)
121205 12:10:34 innobackupex: Finished backing up .frm, .MRG, .MYD, .MYI, .TRG, .TRN, .ARM, .ARZ, .CSV, .CSM and .opt files

innobackupex: Resuming ibbackup

xtrabackup: The latest check point (for incremental): '41681167491'
xtrabackup: Stopping log copying thread.
.>> log scanned up to (41681167645)

xtrabackup: Streaming transaction log from a temporary file...
xtrabackup: Done.
xtrabackup: Transaction log of lsn (41681167491) to (41681167645) was copied.
121205 12:10:37 innobackupex: All tables unlocked
121205 12:10:37 innobackupex: Connection to database server closed

innobackupex: Backup created in directory '/tmp'
innobackupex: MySQL binlog position: filename '', position
innobackupex: You must use -i (--ignore-zeros) option for extraction of the tar stream.
121205 12:10:37 innobackupex: completed OK!

Note the errors related to /tmp.

# mount
tmpfs on /tmp type tmpfs (rw,noatime,mode=1777)

I do have an appropriately sized backup.tar at the end of this. This same error happens in the innobackup.backup.log on a donor, or if I invoke xtrabackup manually on the donor.

If I use a different tmpdir on ext4, then I do not get these errors:

-bash-4.1$ /usr//bin/innobackupex --galera-info --tmpdir=/tmpdir --stream=tar --defaults-file=/etc/my.cnf --socket=/var/lib/mysql/mysql.sock --user=sst --password=9OOgzSVOI95ko /tmpdir > backup.tar

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Inc 2009-2012. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

121205 12:16:55 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:16:55 innobackupex: Connected to database with mysql child process (pid=8444)
121205 12:17:01 innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using mysql Ver 14.14 Distrib 5.5.28, for Linux (x86_64) using readline 5.1
innobackupex: Using mysql server version Copyright (c) 2000, 2012, Oracle and/or its affiliates. All rights reserved.

innobackupex: Created backup directory /tmpdir
121205 12:17:01 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:17:01 innobackupex: Connected to database with mysql child process (pid=8475)
121205 12:17:03 innobackupex: Connection to database server closed

121205 12:17:03 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmpdir --stream=tar
innobackupex: Waiting for ibbackup (pid=8483) to suspend
innobackupex: Suspend file '/tmpdir/xtrabackup_suspended'

xtrabackup_55 version 2.0.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: 470)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /mnt/data
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 5242880
121205 12:17:03 InnoDB: Using Linux native AIO
xtrabackup: using O_DIRECT
>> log scanned up to (41681167645)
[01] Streaming ./ibdata1
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)

>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
[01] ...done
>> log scanned up to (41681167645)
121205 12:20:05 innobackupex: Continuing after ibbackup has suspended
121205 12:20:05 innobackupex: Starting mysql with options: --defaults-file='/etc/my.cnf' --password=xxxxxxxx --user='sst' --socket='/var/lib/mysql/mysql.sock' --unbuffered --
121205 12:20:05 innobackupex: Connected to database with mysql child process (pid=8676)
121205 12:20:07 innobackupex: Starting to lock all tables...
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
>> log scanned up to (41681167645)
121205 12:20:21 innobackupex: All tables locked and flushed to disk

121205 12:20:21 innobackupex: Starting to backup .frm, .MRG, .MYD, .MYI,
innobackupex: .TRG, .TRN, .ARM, .ARZ, .CSM, .CSV and .opt files in
innobackupex: subdirectories of '/mnt/data'
innobackupex: Backing up files '/mnt/data/wiredrive_ci/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (103 files)
innobackupex: Backing up files '/mnt/data/performance_schema/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (18 files)
innobackupex: Backing up file '/mnt/data/percona@002dtoolkit/dsns.frm'
innobackupex: Backing up file '/mnt/data/percona@002dtoolkit/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (85 files)
innobackupex: Backing up file '/mnt/data/blah/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive_qed/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (83 files)
innobackupex: Backing up files '/mnt/data/mysql/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (72 files)
innobackupex: Backing up file '/mnt/data/testpls/example.frm'
innobackupex: Backing up file '/mnt/data/testpls/db.opt'
innobackupex: Backing up files '/mnt/data/wiredrive_qa/*.{frm,MYD,MYI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (103 files)
>> log scanned up to (41681167645)
121205 12:20:24 innobackupex: Finished backing up .frm, .MRG, .MYD, .MYI, .TRG, .TRN, .ARM, .ARZ, .CSV, .CSM and .opt files

innobackupex: Resuming ibbackup

xtrabackup: The latest check point (for incremental): '41681167491'
xtrabackup: Stopping log copying thread.
.>> log scanned up to (41681167645)

xtrabackup: Streaming transaction log from a temporary file...
xtrabackup: Done.
xtrabackup: Transaction log of lsn (41681167491) to (41681167645) was copied.
121205 12:20:26 innobackupex: All tables unlocked
121205 12:20:26 innobackupex: Connection to database server closed

innobackupex: Backup created in directory '/tmpdir'
innobackupex: MySQL binlog position: filename '', position
innobackupex: You must use -i (--ignore-zeros) option for extraction of the tar stream.
121205 12:20:26 innobackupex: completed OK!

Related branches

Alex Yurchenko (ayurchen) wrote :

Committed a workaround in lp:codership-mysql r3831: made wsrep_sst_xtrabackup script to try to use tmpdir setting from my.cnf file for its temporary directory. So it is possible to specify tmpdir different from /tmp.

But frankly, I'm quite surprised this is necessary.

Changed in codership-mysql:
assignee: nobody → Alex Yurchenko (ayurchen)
importance: Undecided → Low
milestone: none → 5.5.28-23.8
status: New → Fix Committed

Xtrabackup can use tmpdir set in my.cnf under [xtrabackup], which
is why there is no need for this. Also, see the comments in
lp:1160047

Updated lp:1160047 with further details.

Changed in percona-xtradb-cluster:
status: New → Fix Released
milestone: none → 5.5.30-23.7.4
Changed in codership-mysql:
milestone: 5.5.30-24.8 → 5.5.31-23.7.4
Changed in codership-mysql:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers