xtrabackup 2.3/2.4 fails after backup/prepare is completed when --defaults-file points to non-existing file

[Shahriyar Rzayev]

Using binary from -> https://github.com/gl-sergei/percona-xtrabackup/tree/xb-5.7.9 tree (compiled/installed)

As Documentation states:

https://www.percona.com/doc/percona-xtrabackup/2.3/xtrabackup_bin/preparing_the_backup.html

"--defaults-file=backup-my.cnf option should be passed to the xtrabackup binary if it is used for preparing the backup"

It is turns out that --defaults-file must be path to backup-my.cnf otherwise it will append to current working directory:

[root@mysql-57 bin]# ./xtrabackup --defaults-file=backup-my.cnf --prepare --target-dir=/home/backup_dir/full/
xtrabackup: [ERROR] Could not open required defaults file: /usr/local/xtrabackup/bin/backup-my.cnf
xtrabackup: [ERROR] Fatal error in defaults handling. Program aborted!

Interesting note from stderr:

InnoDB: Loading buffer pool(s) from /home/backup_dir/full/(null)
InnoDB: not started
InnoDB: Cannot open '/home/backup_dir/full/(null)' for reading: No such file or directory

(null) is created in backup directory:

[root@mysql-57 full]# ls
backup-my.cnf ib_buffer_pool ib_logfile0 ibtmp1 (null) sys xtrabackup_info
bck ibdata1 ib_logfile1 mysql performance_schema xtrabackup_checkpoints xtrabackup_logfile

And the result is:

InnoDB: Shutdown completed; log sequence number 2491944
completed OK!
11:08:21 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
./xtrabackup(my_print_stacktrace+0x2c)[0xf8955c]
./xtrabackup(handle_fatal_signal+0x262)[0x9fabe2]
/lib64/libpthread.so.0(+0xf100)[0x7fb0fee4a100]
./xtrabackup(free_root+0x36)[0xf85856]
./xtrabackup(free_defaults+0x6b)[0x73b20b]
./xtrabackup(main+0x954)[0x6f0374]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fb0fd39ab15]
./xtrabackup[0x709ae5]

[Shahriyar Rzayev (rzayev-sehriyar)]

Full log:

[root@mysql-57 bin]# ./xtrabackup --prepare --target-dir=/home/backup_dir/full/ --defaults-file=backup-my.cnf
xtrabackup: Error: --defaults-file must be specified first on the command line
[root@mysql-57 bin]# ./xtrabackup --defaults-file=backup-my.cnf --prepare --target-dir=/home/backup_dir/full/
xtrabackup: [ERROR] Could not open required defaults file: /usr/local/xtrabackup/bin/backup-my.cnf
xtrabackup: [ERROR] Fatal error in defaults handling. Program aborted!
./xtrabackup version 2.3.1beta1 based on MySQL server 5.7.10 Linux (x86_64) (revision id: )
xtrabackup: cd to /home/backup_dir/full
xtrabackup: This target seems to be not prepared yet.
InnoDB: Number of pools: 1
xtrabackup: xtrabackup_logfile detected: size=2097152, start_lsn=(2491701)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = .
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 2097152
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = .
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = .
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 2097152
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: PUNCH HOLE support available
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Uses event mutexes
InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
InnoDB: Compressed tables use zlib 1.2.7
InnoDB: Number of pools: 1
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, total size = 100M, instances = 1, chunk size = 100M
InnoDB: Completed initialization of buffer pool
InnoDB: page_cleaner coordinator priority: -20
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 2491701
InnoDB: Doing recovery: scanned up to log sequence number 2491710 (0%)
InnoDB: Doing recovery: scanned up to log sequence number 2491710 (0%)
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Creating shared tablespace for temporary tables
InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
InnoDB: File './ibtmp1' size is now 12 MB.
InnoDB: 96 redo rollback segment(s) found. 1 redo rollback segment(s) are active.
InnoDB: 32 non-redo rollback segment(s) are active.
InnoDB: 5.7.10 started; log sequence number 2491710
InnoDB: Loading buffer pool(s) from /home/backup_dir/full/(null)
InnoDB: not started
InnoDB: Cannot open '/home/backup_dir/full/(null)' for reading: No such file or directory

[notice (again)]
  If you use binary log and don't use any hack of group commit,
  the binary log position seems to be:

xtrabackup: starting shutdown with innodb_fast_shutdown = 1
InnoDB: FTS optimize thread exiting.
InnoDB: Starting shutdown...
InnoDB: Dumping buffer pool(s) to /home/backup_dir/full/(null)
InnoDB: Buffer pool(s) dum...

[Sergei Glushchenko (sergei.glushchenko)]

Crahses if --defaults-file is not found

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-744