Too easy to backup wrong datadir with multiple instances

Bug #1343722 reported by MikeG on 2014-07-18
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Fix Released
Wishlist
Alexey Kopytov
2.1
Won't Fix
Undecided
Unassigned
2.2
Fix Released
Wishlist
Alexey Kopytov

Bug Description

If you have a valid datadir that is referenced in /etc/my.cnf and another instance you specify by socket, you can silently back up the wrong directory when you forget to pass --defaults-file:

# innobackupex --socket=/s/56master/data/mysql.sock --stream=tar . > /dev/null

<SNIP>

140716 19:05:51 innobackupex: Starting to backup non-InnoDB tables and files
innobackupex: in subdirectories of '/var/lib/mysql'
innobackupex: Backing up files '/var/lib/mysql/test/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (28 files)

# mysql -S/s/56master/data/mysql.sock -e'select @@datadir'
/s/56master/data/

A straight forward fix would be to compare $orig_datadir/ to @@datadir that you SELECT over the socket:

For percona-xtrabackup-2.2.3-4982.el6.x86_64:

@@ -1899,6 +1899,13 @@
     my $buffer_pool_filename = get_option(\%config, $option_defaults_group,
                                           'innodb_buffer_pool_filename');

+ # Validate datadir for current socket before doing any heavy lifting
+ my ($datadir_var) = eval { $mysql{dbh}->selectrow_array("SELECT \@\@datadir") };
+ if ( "$orig_datadir/" ne "$datadir_var" ) {
+ print STDERR "$prefix Error: $orig_datadir incorrect for this unix socket\n";
+ exit 1;
+ }
+
     detect_mysql_capabilities_for_backup(\%mysql);

     #

For percona-xtrabackup-2.1.9-744.rhel6.x86_64 it is quite similar @@ -1916,6 +1916,22 @@

If this is not acceptable or conflicts with goals in https://blueprints.launchpad.net/percona-xtrabackup/+spec/read-server-options-with-show-variables it might be ok to create a view and then search for it:

     my $buffer_pool_filename = get_option(\%config, $option_defaults_group,
                                           'innodb_buffer_pool_filename');

+ # Validate datadir for current socket before doing any heavy lifting
+ use Digest::MD5 qw(md5_hex);
+ my $test_view = md5_hex("$now $prefix $orig_datadir"); # Avoid replication race
+ print STDERR "\n";
+ print STDERR "$now $prefix Creating a test view to validate that datadir is $orig_datadir\n";
+ mysql_query(\%mysql, "CREATE VIEW mysql.xtrabackup_$test_view AS SELECT 'xtrabackup_test_view'");
+ if (!-f "$orig_datadir/mysql/xtrabackup_$test_view.frm" ) {
+ print STDERR "$now $prefix Dropping test view\n";
+ mysql_query(\%mysql, "DROP VIEW mysql.xtrabackup_$test_view");
+ print STDERR "Detected $orig_datadir to be incorrect for this unix socket\n";
+ exit 1;
+ } else {
+ print STDERR "$now $prefix Datadir validated, dropping test view\n";
+ mysql_query(\%mysql, "DROP VIEW mysql.xtrabackup_$test_view");
+ }
+
     detect_mysql_capabilities_for_backup(\%mysql);

     #

At least one problem with the heavy handed approach is that there is a race condition if the slave runs a backup at the same second but server_id does not seem to be available via xtrabackup --print-param. Perhaps it could be extended to toggle the session variable for sql_log_bin/wsrep_on to avoid this. Another potential problem is that a "read only" tool uses DDL.

Related branches

MikeG (mikegriffin) wrote :

This might also occur without multiple instances, if you have changed datadir in /etc/my.cnf but have not yet restarted the server.

Easy to reproducible.

nilnandan@nilnandan-Dell-XPS:~$ mysql --socket=/tmp/mysql_sandbox5615.sock -e "select @@datadir"
+--------------------------------------------+
| @@datadir |
+--------------------------------------------+
| /home/nilnandan/sandboxes/msb_5_6_15/data/ |
+--------------------------------------------+

nilnandan@nilnandan-Dell-XPS:~$ sudo innobackupex --socket=/tmp/mysql_sandbox5615.sock /home/nilnandan/backup

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

...
xtrabackup version 2.2.3 based on MySQL server 5.6.17 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
...
innobackupex: in subdirectories of '/var/lib/mysql'
innobackupex: Backing up files '/var/lib/mysql/mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (74 files)
>> log scanned up to (3171230103)
innobackupex: Backing up files '/var/lib/mysql/performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (53 files)
innobackupex: Backing up file '/var/lib/mysql/dbtest/db.opt'
innobackupex: Backing up file '/var/lib/mysql/dbtest/sbtest.frm'
innobackupex: Backing up files '/var/lib/mysql/test/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (10 files)
140723 08:58:33 innobackupex: Finished backing up non-InnoDB tables and files

140723 08:58:33 innobackupex: Executing FLUSH ENGINE LOGS...
140723 08:58:33 innobackupex: Waiting for log copying to finish

>> log scanned up to (3171230103)
xtrabackup: The latest check point (for incremental): '3171230103'
xtrabackup: Stopping log copying thread.
.>> log scanned up to (3171230103)

xtrabackup: Creating suspend file '/home/nilnandan/backup/2014-07-23_08-57-58/xtrabackup_log_copied' with pid '3297'
xtrabackup: Transaction log of lsn (3171230103) to (3171230103) was copied.
140723 08:58:34 innobackupex: All tables unlocked

innobackupex: Backup created in directory '/home/nilnandan/backup/2014-07-23_08-57-58'
140723 08:58:34 innobackupex: Connection to database server closed
140723 08:58:34 innobackupex: completed OK!
nilnandan@nilnandan-Dell-XPS:~$

Changed in percona-xtrabackup:
status: New → Confirmed
Alexey Kopytov (akopytov) wrote :

This looks like another issue that would be automatically fixed by https://blueprints.launchpad.net/percona-xtrabackup/+spec/read-server-options-with-show-variables. I don't see any conflicts here.

MikeG (mikegriffin) wrote :

This seems to be fixed as of 2.2.6:

"""
Percona XtraBackup now reads server options from SHOW VARIABLES rather than my.cnf configuration file
"""

In testing, the backup rightfully fails with the following error:

[ERROR] ! innobackupex: Error: option 'datadir' has different values:
[ERROR] ! '/stage/mysqlstage/data' in defaults file
[ERROR] ! '/live/mysqllive/data/' in SHOW VARIABLES

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1009

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers