xtrabackup fails with binlogs in non-standard directory

[Brad Jorgensen]

On my servers that have binlogs enabled but in a non-standard directory via "log-bin=/seq/mysql" (the main datadir is /var/lib/mysql), xtrabackup fails at the end when looking for the latest binlog file. It looks like the problem is that since the change to reading the server config from the running server instead of getting it from the config file, xtrabackup assumes that the binlogs are in the datadir. In the first section of the attached log, the binlogs are in /seq/mysql/ but xtrabackup looks in /var/lib/mysql/. The second section is after I moved the binlogs to /var/lib/mysql. Based on the log, it is probably a different issue since it finished copying the binlog. To get xtrabackup to complete successfully, I had to disable binlogs. I am running xtrabackup 2.3.2 against MariaDB 10.0.21 on CentOS 6. Let me know if you need any more information.

[Muhammad Irfan (muhammad-irfan)]

Can you please show us backup command you used to took backup along with my.cnf

[Sergei Glushchenko (sergei.glushchenko)]

Looks like duplicate of bug 1512281 to me

[Sergei Glushchenko (sergei.glushchenko)]

SHOW MASTER STATUS, SHOW VARIABLES and my.cnf would help it identify.
Is it Galera node?

[Brad Jorgensen (5-brad)]

Yes it is a galera node. The command I'm running is "innobackupex --galera-info --parallel=4 --no-version-check --no-timestamp --user=backup --password=xxx --stream=xbstream ./ | nice pigz -Rc > backup_path".

Looking at bug 1512281, I see that percona server has the "log_bin_basename" server variable which mariadb does not. That appears to be the source of my original problem. Would it be possible to read the binlog directory from "log-bin" in my.cnf if "log_bin_basename" does not exist instead of just assuming datadir? I will be submitting an issue for mariadb to add "log_bin_basename".

I can post my my.cnf if you still want to see it, but I think we should have this figured out now.

I have three servers in a galera cluster:

db3 has binlogs disabled so I can get nightly backups.

db2 has binlogs in the other directory via "log-bin=/seq/mysql", but I think the problem is as stated above.

db1 has binlogs enabled in in the data dir by "log-bin=mysql-bin". I changed the binlog path to see if it would backup successfully with binlogs in the datadir. It also fails, but it looks like the same problem as in bug 1512281. Here is an excerpt from the log when it fails:

151203 01:32:47 Finished backing up non-InnoDB tables and files
151203 01:32:47 [00] Streaming xtrabackup_galera_info
151203 01:32:47 [00] ...done
151203 01:32:47 [00] Streaming /var/lib/mysql//mysql-bin.000090 to <STDOUT>
151203 01:32:47 [00] ...done
*** glibc detected *** innobackupex: double free or corruption (fasttop): 0x000000000145a1c0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x75e66)[0x7f2de154be66]
innobackupex(_Z25write_current_binlog_fileP8st_mysql+0x258)[0x59d678]
innobackupex(_Z12backup_startv+0x177)[0x5a3fb7]
innobackupex(_Z22xtrabackup_backup_funcv+0xc00)[0x58def0]
innobackupex(main+0xaea)[0x591efa]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f2de14f4d5d]
innobackupex[0x585e79]
...
07:32:47 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x2e) [0x8c502e]
innobackupex(handle_fatal_signal+0x273) [0x736dc3]
/lib64/libpthread.so.0(+0xf710) [0x7f2de2eec710]
/lib64/libc.so.6(gsignal+0x35) [0x7f2de1508625]
/lib64/libc.so.6(abort+0x175) [0x7f2de1509e05]
/lib64/libc.so.6(+0x70537) [0x7f2de1546537]
/lib64/libc.so.6(+0x75e66) [0x7f2de154be66]
innobackupex(write_current_binlog_file(st_mysql*)+0x258) [0x59d678]
innobackupex(backup_start()+0x177) [0x5a3fb7]
innobackupex(xtrabackup_backup_func()+0xc00) [0x58def0]
innobackupex(main+0xaea) [0x591efa]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x7f2de14f4d5d]
innobackupex() [0x585e79]

I'm assuming that since this is a double free issue, it is the same as bug bug 1512281, not related ...

[Sergei Glushchenko (sergei.glushchenko)]

You are right, MariaDB not having log_bin_basename is a real issue actually. Yes, reading --log-bin from my.cnf is an option, but I personally don't like it, since we want to move from reading server configuration from my.cnf.

I see that log_dir_basename has been implemented in MariaDB 10.1.6 https://mariadb.atlassian.net/browse/MDEV-7110.

[Philippe Jean (philippejean)]

Do you have any idea how to work around the problem with MariaDB Galera 10.0?

[Sergei Glushchenko (sergei.glushchenko)]

https://github.com/percona/percona-xtrabackup/pull/224
https://github.com/percona/percona-xtrabackup/pull/225

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-936