xtrabackup encounters fatal error trying to create temp table

[Lachlan Mulcahy]

xtrabackup 1.6-amd64 on Solaris encounters an error during --apply-log-only when attempting to create a temporary table file, causing apply-log to fail.

I think this is a dupe of bug 402884 which is supposed to be fixed long ago, but it doesn't seem to look that way in the build I am using??

Bug Link:

https://bugs.launchpad.net/percona-xtrabackup/+bug/402884

Command Run:

xtrabackup --defaults-file=/backup/xbm/dbp36-int/46/backup-my.cnf --use-memory=1073741824 --prepare --apply-log-only --target-dir=/backup/xbm/dbp36-int/46

Output:

xtrabackup: cd to /backup/xbm/dbp36-int/46
xtrabackup: This target seems to be not prepared yet.
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:2000M;ibdata2:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 75174641664
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 1073741824 bytes for buffer pool (set by --use-memory parameter)
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Warning: innodb_file_io_threads is deprecated. Please use innodb_read_io_threads and innodb_write_io_threads instead
110826 12:59:37 InnoDB: Initializing buffer pool, size = 1.0G
110826 12:59:38 InnoDB: Completed initialization of buffer pool
110826 12:59:38 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 5157631495386
110826 12:59:38 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Doing recovery: scanned up to log sequence number 5157636738048 (0 )

..<snip>..

InnoDB: Doing recovery: scanned up to log sequence number 5161464040448 (5 )
InnoDB: Doing recovery: scanned up to log sequence number 5161469283328 (5 )
InnoDB: Doing recovery: scanned up to log sequence number 5161474526208 (5 )
InnoDB: Doing recovery: scanned up to log sequence number 5161479769088 (5 )
110826 14:05:59 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name .//mysqldb/tmp/#sql2413_4fa7a_1.ibd
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.
xtrabackup Ver undefined Rev undefined for 5.1.55 pc-solaris2.11 (i386)
xtrabackup: xtrabackup_logfile detected: size=75174641664, start_lsn=(5157631495386)

[Andrew Hutchings (linuxjedi)]

To me it looks like a duplicate/regression of bug #402884 since the os0file.c fix to change os_file_handle_error to os_file_handle_error_no_exit appears to be missing from current xtrabackup.

[Lachlan Mulcahy (lachlan-mulcahy)]

Given this code in os0file.c:

if (srv_file_per_table) {
 retry = os_file_handle_error_no_exit(name,
   create_mode == OS_FILE_CREATE ?
   "create" : "open");
} else {
 retry = os_file_handle_error(name,
   create_mode == OS_FILE_CREATE ?
   "create" : "open");
}

I can confirm that you cannot work around by simply giving xtrabackup binary --innodb_file_per_table=1 on the command line. The only work around is actually creating the tmpdir you need.

[Jani Tolonen (jani-askmonty)]

According to the code you should not get an error if innodb_file_per_table=1, since then it should use os_file_handle_error_no_exit() -function. Could you please verify?

[Lachlan Mulcahy (lachlan-mulcahy)]

Hi Jani - Sorry it took so long to respond.

As I mentioned:

"I can confirm that you cannot work around by simply giving xtrabackup binary --innodb_file_per_table=1 on the command line. The only work around is actually creating the tmpdir you need."

It is possible that the option is not correctly being passed from the xtrabackup command line through the the MySQL engine internally. In any case - it does not work as a viable work around.

This is still a bug that is affecting us.

[Chris Calender (chriscalender)]

This bug (#837143) is said to be a re-incarnation of this:

https://bugs.launchpad.net/percona-xtrabackup/+bug/402884

Bug #402884 says it is due to the following MySQL/InnoDB bugs:

http://bugs.mysql.com/bug.php?id=41609
http://bugs.mysql.com/bug.php?id=45638
http://bugs.mysql.com/bug.php?id=45976

The first 2 of those had been fixed, but not the last, 45976.

However, bug 45976 was recently fixed in MySQL 5.1.62, 5.5.22, and 5.6.5.

Is it reasonable to expect that this bug (#837143) can now be fixed?

Or, is the latest 2.0 Xtrabackup built off a newer version where this would not occur?

Or, would the an even older Xtrabackup work fine as long as the user is using a MySQL that contains this fix?

[Laurynas Biveinis (laurynas-biveinis)]

Chris -

XtraBackup here depends not on the MySQL version being backuped, but rather the version that XtraBackup is compiled against. Currently these are 5.1.59 and 5.5.16/17. We rebase XtraBackup on newer versions periodically, and when we do this the next time, this bug will have to verified again.

[Alexey Kopytov (akopytov)]

This should be fixed when https://blueprints.launchpad.net/percona-xtrabackup/+spec/rebase-2.0-on-newer-server-versions is implemented.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-580