Server options innodb_log_group_home_dir and innodb_data_home_dir not getting read from SHOW VARIABLES

[Zsolt Nagy]

Server options innodb_log_group_home_dir and innodb_data_home_dir not geeting read from SHOW VARIABLES
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_log_group_home_dir = ./

/usr/bin/innobackupex --defaults-file=/some_path/db.my.cnf --slave-info --rsync --compress --parallel=8 --compress-threads=8 /backup_path/

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

150414 13:17:09 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/some_path/db.my.cnf;mysql_read_default_group=xtrabackup' (using password: NO).
150414 13:17:09 innobackupex: Connected to MySQL server
150414 13:17:09 innobackupex: Executing a version check against the server...
150414 13:17:09 innobackupex: Done.
150414 13:17:09 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using server version 5.6.23-log

innobackupex: Created backup directory /backup_path/2015-04-14_13-17-09

150414 13:17:09 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/some_path/db.my.cnf" --backup --suspend-at-end --target-dir=/backup_path/2015-04-14_13-17-09 --datadir="/home/db/mysql/db/data/" --innodb_log_file_size="50331648" --innodb_data_file_path="ibdata1:1024M:autoextend" --tmpdir=/home/db/mysql/db/tmp --compress --compress-threads=8 --extra-lsndir='/home/db/mysql/db/tmp' --parallel=8
innobackupex: Waiting for ibbackup (pid=22076) to suspend
innobackupex: Suspend file '/backup_path/2015-04-14_13-17-09/xtrabackup_suspended_2'

xtrabackup version 2.2.10 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /home/db/mysql/db/data/
xtrabackup: open files limit requested 0, set to 100000000
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:1024M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
2015-04-14 13:17:09 2ad69b759540 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: File ./ib_logfile0: 'open' returned OS error 71. Cannot continue operation
2015-04-14 13:17:09 2ad69b759540 InnoDB: Assertion failure in thread 47101219542336 in file os0file.cc line 658
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
13:17:09 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x1d4deb0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 101010101010101 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x24) [0x8eb3f4]
xtrabackup(handle_fatal_signal+0x254) [0x72e304]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x2ad699a99cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x2ad69b3d1445]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x2ad69b3d4bab]
xtrabackup() [0x657525]
xtrabackup(os_file_create_func(char const*, unsigned long, unsigned long, unsigned long, unsigned long*)+0x1f8) [0x658c98]
xtrabackup() [0x5a2aae]
xtrabackup(main+0xdf6) [0x58afe6]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x2ad69b3bc76d]
xtrabackup() [0x59bd65]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2703
        main::wait_for_ibbackup_file_create('/backup_path/2015-04-14_13-17-09/xtra...') called at /usr/bin/innobackupex line 2723
        main::wait_for_ibbackup_suspend('/backup_path/2015-04-14_13-17-09/xtra...') called at /usr/bin/innobackupex line 1976
        main::backup() called at /usr/bin/innobackupex line 1601
innobackupex: Error: The xtrabackup child process has died at /usr/bin/innobackupex line 2703.

Sample db.my.cnf values:
[client]
user = backupuser
password = "password"
socket = /socket_path/socket

Workaround:
db.my.cnf values:
[client]
user = backupuser
password = "password"
socket = /socket_path/socket
[xtrabackup]
innodb_data_home_dir = /home/basekit/mysql/db/innodbdata
innodb_log_group_home_dir = /home/basekit/mysql/db/innodbdata

[Nikhil (nikhil-lingala)]

I get the same error while using this backup command syntax, though the very same command worked very well before:

# innobackupex --defaults-file=/path/to/my.cnf --user=<username> --password=<password> --socket=/path/to/mysql.sock /path/to/base-dir/

[Zsolt Nagy (zs-linux)]

Sure thing, but since 2.2.6 this should be a feature and works fine for the rest of the server options.
"Percona XtraBackup now reads server options from SHOW VARIABLES rather than my.cnf configuration file"
http://www.percona.com/doc/percona-xtrabackup/2.2/release-notes/2.2/2.2.6.html

[Nilnandan Joshi (nilnandan-joshi)]

Verified with XB 2.2.10. It will give error when innodb data will be stored other than default datadir.

root@desktop:~# cat /etc/mysql/my.cnf | grep "innodb"
innodb_data_file_path=ibdata1:50M;ibdata2:253755392;ibdata3:50M:autoextend:max:4G
innodb_data_home_dir = /var/lib/mysql/innodb
innodb_log_group_home_dir = /var/lib/mysql/innodb
root@desktop:~#

root@desktop:~# cat db.my.cnf
[client]
user=root
password=root
socket=/var/run/mysqld/mysqld.sock
root@desktop:~#

root@desktop:~# innobackupex --defaults-file=/root/db.my.cnf --slave-info --rsync --compress --parallel=8 --compress-threads=8 /home/nilnandan/backup/

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

150427 13:09:31 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/root/db.my.cnf;mysql_read_default_group=xtrabackup' (using password: NO).
150427 13:09:31 innobackupex: Connected to MySQL server
150427 13:09:31 innobackupex: Executing a version check against the server...
150427 13:09:31 innobackupex: Done.
150427 13:09:31 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using server version 5.6.23-72.1-log

innobackupex: Created backup directory /home/nilnandan/backup/2015-04-27_13-09-31

150427 13:09:31 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/root/db.my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/home/nilnandan/backup/2015-04-27_13-09-31 --datadir="/var/lib/mysql/" --innodb_log_file_size="50331648" --innodb_data_file_path="ibdata1:50M;ibdata2:253755392;ibdata3:50M:autoextend:max:4G" --tmpdir=/tmp --compress --compress-threads=8 --extra-lsndir='/tmp' --parallel=8
innobackupex: Waiting for ibbackup (pid=13895) to suspend
innobackupex: Suspend file '/home/nilnandan/backup/2015-04-27_13-09-31/xtrabackup_suspended_2'

xtrabackup version 2.2.10 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql/
xtrabackup: open files limit requested 0, set to 10000
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:50M;ibdata2:253755392;ibdata3:50M:autoextend:max:4G
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
2015-04-27 13:09:31 7f91a8056780 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: File ./ib_logfile0: 'open' returned OS error 71. Cannot continue operation
2015-04-27 13:09:31 7f91a8056780 InnoDB: Assertion failure in thread 140263565911936 in file os0file.cc line 658
InnoDB: We intentiona...

[Agustín (agustin-gallego)]

I could reproduce this with 2.3, too, and strace shows that a `cd /` is issued. Then while looking for ./ib_logfile0 we get that error.

31368 getcwd("/data/mysql1", 4096) = 13
31368 lstat("/data", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
31368 lstat("/data/mysql1", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
31368 write(2, "innobackupex version 2.3.2 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 306a2e0)\n", 94) = 94
31368 write(2, "xtrabackup: uses posix_fadvise().\n", 34) = 34
31368 chdir("/") = 0
31368 write(2, "xtrabackup: cd to \n", 19) = 19
31368 getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0
31368 write(2, "xtrabackup: open files limit requested 0, set to 1024\n", 54) = 54
31368 write(2, "xtrabackup: using the following InnoDB configuration:\n", 54) = 54
31368 write(2, "xtrabackup: innodb_data_home_dir = ./\n", 40) = 40
31368 write(2, "xtrabackup: innodb_data_file_path = ibdata1:100M:autoextend\n", 62) = 62
31368 write(2, "xtrabackup: innodb_log_group_home_dir = ./\n", 45) = 45
31368 write(2, "xtrabackup: innodb_log_files_in_group = 2\n", 44) = 44
31368 write(2, "xtrabackup: innodb_log_file_size = 5242880\n", 45) = 45
31368 brk(0x2baa000) = 0x2baa000
31368 brk(0x2bcf000) = 0x2bcf000
31368 brk(0x2bf4000) = 0x2bf4000
31368 brk(0x2c1a000) = 0x2c1a000
31368 brk(0x2c3f000) = 0x2c3f000
31368 mmap(NULL, 184320, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feeaba04000
31368 brk(0x2c66000) = 0x2c66000
31368 mmap(NULL, 1052672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feeab903000
31368 mmap(NULL, 561152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feeab87a000
31368 mmap(NULL, 241664, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feea973d000
31368 mmap(NULL, 290816, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feea96f6000
31368 open("./ib_logfile0", O_RDONLY) = -1 ENOENT (No such file or directory)

Note two things:

1- The tool was in the correct directory, and decided to change to '/'.
2- The tool prints out an incomplete message: "xtrabackup: cd to \n" --not showing where xtrabackup is changing dir to--.

I haven't had time to check the code, to see if the `cd /` is really necessary for anything else, or if it can perhaps be avoided as a simple workaround (using SHOW VARIABLES and calling innobackupex from the datadir).

[Shahriyar Rzayev (rzayev-sehriyar)]

Another way to reproduce same assertion:

sh@percona-XPS-15:~$ sudo docker run --name mariadb-docker -v /home/dokcerdatadir:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=12345 -d mariadb:10.0.23
1dd163611045fe04033940377ca26268dc4d209e3b807e069e563d5e160bf53a

Mounted /var/lib/mysql of MariaDB in /home/dokcerdatadir of host.

If you have /var/lib/mysql on HOST system just rename it to /var/lib/mysql2

And try to backup using connection to docker's host:

sh@percona-XPS-15:~$ sudo xtrabackup --backup --target-dir=/home/xtrabackupdir --host=172.17.0.2 --user=root --password=12345

151224 14:50:02 Connecting to MySQL server host: 172.17.0.2, user: root, password: set, port: 3306, socket: /var/run/mysqld/mysqld.sock
Using server version 10.0.23-MariaDB-1~jessie
xtrabackup version 2.3.3 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 525ca7d)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 50331648
2015-12-24 14:50:02 7f893dfb5780 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: File ./ib_logfile0: 'open' returned OS error 71. Cannot continue operation
2015-12-24 14:50:02 7f893dfb5780 InnoDB: Assertion failure in thread 140227427129216 in file os0file.cc line 658

xtrabackup(my_print_stacktrace+0x2e) [0x975d2e]
xtrabackup(handle_fatal_signal+0x262) [0x73f102]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f893dbac340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f893c310cc9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f893c3140d8]
xtrabackup(os_file_create_func(char const*, unsigned long, unsigned long, unsigned long, unsigned long*)+0x48c) [0x5d3ebc]
xtrabackup() [0x594387]
xtrabackup(xtrabackup_backup_func()+0x30f) [0x59a13f]
xtrabackup(main+0x9c8) [0x57e498]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5) [0x7f893c2fbec5]
xtrabackup() [0x592308]

As i understand xtrabackup finds '/var/lib/mysql' path from "show variables" and tries to backup it on host system, but there is no '/var/lib/mysql' on host.

sh@percona-XPS-15:~$ xtrabackup --version
xtrabackup version 2.3.3 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 525ca7d)

[Sergei Glushchenko (sergei.glushchenko)]

This one has been fixed with patch for bug 1526467