Incorrect error about ib_logfile0 when datadir isn't defined

[Geoff Montee]

When using the xtrabackup-v2 SST method, "datadir" apparently needs to be defined in the [mysqld] group of the server's configuration file. If it is not defined, the SST will fail.

XtraBackup 2.2 makes this somewhat obvious because it writes the following error message to innobackup.move.log:

innobackupex: got a fatal error with the following stacktrace: at /usr//bin/innobackupex line 4545
 main::get_option('datadir') called at /usr//bin/innobackupex line 2450
 main::copy_back(1) called at /usr//bin/innobackupex line 1578
innobackupex: Error: no 'datadir' option in group 'mysqld' in server configuration file '/etc/my.cnf' at /usr//bin/innobackupex line 4545.

However, when this happens with XtraBackup 2.3, the errors in innobackup.move.log don't make any sense. Instead of pointing out that datadir isn't defined, it writes something about ib_logfile0, and then it crashes.

innobackup.move.log contains the following:

160302 15:57:55 innobackupex: Starting the move-back operation

IMPORTANT: Please check that the move-back run completes successfully.
           At the end of a successful move-back run innobackupex
           prints "completed OK!".

innobackupex version 2.3.3 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 525ca7d)
Error: Move file ib_logfile0 to ./ib_logfile0 failed: Destination file exists
20:57:55 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x2e) [0x93827e]
innobackupex(handle_fatal_signal+0x273) [0x81b483]
/lib64/libpthread.so.0() [0x3dd720f790]
/lib64/libc.so.6(cfree+0x1c) [0x3dd6e7b96c]
innobackupex(copy_back()+0x377) [0x5a1767]
innobackupex(main+0xacb) [0x591f9b]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x3dd6e1ed5d]
innobackupex() [0x585ea9]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup

I reproduced this in the following way with MariaDB Galera Cluster 10.0.23:

1.) Make sure "datadir" is NOT defined in the configuration file.

2.) Make sure XtraBackup 2.3 is installed.

3.) Make sure xtrabackup-v2 is the SST method.

4.) Stop the server on the joiner.

service mysql stop

5.) Clear the datadir and SST directory:

rm -fr /var/lib/mysql/*
rm -fr /var/lib/mysql/.sst/*

6.) Start the server on the joiner.

service mysql start

7.) When the SST fails, check innobackup.move.log.

[trent-- (sylvainfaivre)]

Debian/Ubuntu style config has the datadir setting in /etc/mysql/my.cnf, while xtrabackup looks for it in /etc/my.sql, which is a symlink to /etc/mysql/debian.cnf

A workaround is to copy from /etc/mysql/my.cnf to /etc/mysql/debian.cnf the lines :
[mysqld]
datadir = /var/lib/mysql

[Geoff Montee (geoff-montee)]

Hi trent--,

I appreciate the feedback, but I don't need a workaround for this, and I'm not using Debian or Ubuntu anyway.

I submitted this bug to let Percona know that XtraBackup 2.3 crashes and writes a seemingly inaccurate error message about ib_logfile0 when datadir is not defined during an xtrabackup-v2 SST.

[trent-- (sylvainfaivre)]

Sure thing, your bug report was very clear. I assumed you were using Debian or Ubuntu like me, because I had the same problem, but I can see now that there was no info about a specific distro.

Anyway, this can be a head's up to people who might find this bug by searching the web for this error message, just like I did :
Error: Move file ib_logfile0 to ./ib_logfile0 failed: Destination file exists

[Sergei Glushchenko (sergei.glushchenko)]

Fixed with the bug 1566228