--stream=tar fails (archive_write_header failed)

[Steven Ayre]

--stream=tar appears to fail with xtrabackup 2.0.0 (--stream=xbstream works fine).

Platform is Debian Squeeze x64, using the dotdeb percona binaries.

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Inc 2009-2012. All Rights Reserved.
[2012-04-10 12:34:11 GMT]
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
[2012-04-10 12:34:11 GMT]
120410 13:33:46 innobackupex: Starting mysql with options: --password=xxxxxxxx --user='root' --unbuffered --
120410 13:33:46 innobackupex: Connected to database with mysql child process (pid=5367)
120410 13:33:52 innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".
[2012-04-10 12:34:11 GMT]
innobackupex: Using mysql Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (x86_64) using readline 5.2
innobackupex: Using mysql server version Copyright (C) 2002 MySQL AB
[2012-04-10 12:34:11 GMT]
innobackupex: Created backup directory /tmp/tmp.hKYMnWlyCa
120410 13:33:53 innobackupex: Starting mysql with options: --password=xxxxxxxx --user='root' --unbuffered --
120410 13:33:53 innobackupex: Connected to database with mysql child process (pid=5434)
120410 13:33:55 innobackupex: Connection to database server closed
[2012-04-10 12:34:11 GMT]
120410 13:33:55 innobackupex: Starting ibbackup with command: xtrabackup_51 --backup --suspend-at-end --target-dir=/tmp --compress --compress-threads=2 --stream=tar
innobackupex: Waiting for ibbackup (pid=5464) to suspend
innobackupex: Suspend file '/tmp/xtrabackup_suspended'
[2012-04-10 12:34:11 GMT]
xtrabackup_51 version 2.0.0 for MySQL server 5.1.59 unknown-linux-gnu (x86_64) (revision id: undefined)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = /var/lib/mysql
xtrabackup: innodb_data_file_path = ibdata1:100M:autoextend
xtrabackup: innodb_log_group_home_dir = /var/lib/mysql
xtrabackup: innodb_log_files_in_group = 4
xtrabackup: innodb_log_file_size = 268435456
>> log scanned up to (375 4004314859)
archive_write_header() failed.
[01] xtrabackup: error: cannot open the destination stream for ibdata1
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 371.

[Steven Ayre (steveayre)]

Problem occurs both with and without --compress options.

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Inc 2009-2012. All Rights Reserved.
[2012-04-10 12:37:33 GMT]
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
[2012-04-10 12:37:33 GMT]
120410 13:37:14 innobackupex: Starting mysql with options: --password=xxxxxxxx --user='root' --unbuffered --
120410 13:37:14 innobackupex: Connected to database with mysql child process (pid=7038)
120410 13:37:20 innobackupex: Connection to database server closed
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".
[2012-04-10 12:37:33 GMT]
innobackupex: Using mysql Ver 14.12 Distrib 5.0.51a, for debian-linux-gnu (x86_64) using readline 5.2
innobackupex: Using mysql server version Copyright (C) 2002 MySQL AB
[2012-04-10 12:37:33 GMT]
innobackupex: Created backup directory /tmp/tmp.QkgecwCPry
120410 13:37:21 innobackupex: Starting mysql with options: --password=xxxxxxxx --user='root' --unbuffered --
120410 13:37:21 innobackupex: Connected to database with mysql child process (pid=7111)
120410 13:37:23 innobackupex: Connection to database server closed
[2012-04-10 12:37:33 GMT]
120410 13:37:23 innobackupex: Starting ibbackup with command: xtrabackup_51 --backup --suspend-at-end --target-dir=/tmp --stream=tar
innobackupex: Waiting for ibbackup (pid=7128) to suspend
innobackupex: Suspend file '/tmp/xtrabackup_suspended'
[2012-04-10 12:37:33 GMT]
xtrabackup_51 version 2.0.0 for MySQL server 5.1.59 unknown-linux-gnu (x86_64) (revision id: undefined)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = /var/lib/mysql
xtrabackup: innodb_data_file_path = ibdata1:100M:autoextend
xtrabackup: innodb_log_group_home_dir = /var/lib/mysql
xtrabackup: innodb_log_files_in_group = 4
xtrabackup: innodb_log_file_size = 268435456
>> log scanned up to (375 4007342403)
archive_write_header() failed.
[01] xtrabackup: error: cannot open the destination stream for ibdata1
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
innobackupex: Error: ibbackup child process has died at /usr/bin/innobackupex line 371.

[Alexey Kopytov (akopytov)]

Could you provide some additional info? I cannot reproduce it, so I'd like to see the full command line used to start the backup (incl. the commands executed on the remote host, if the tar stream is being piped to a remote host).

Is this failure persistent, i.e. does it always occur on ibdata1 on each run?

Thanks.

[Arnaud Le Blanc (arnaud-lb)]

I'm seeing this error too, with the following command:

xtrabackup_55 --backup --suspend-at-end --target-dir=/home/xbm/tmp/xbm-1160806 --stream=tar

It happens when xtrabackups attempts to add ibdata1 to the archive. ibdata1 has a size of 31065112576 bytes, which is apparently more than what is allowed.

The error comes from

if (format_number(archive_entry_size(entry), h + USTAR_size_offset, USTAR_size_size, USTAR_size_max_size, strict)) {

in src/libarchive/libarchive/archive_write_set_format_ustar.c:266 (__archive_write_format_header_ustar)

Is there any workaround for this ?

[Andrew Garner (muzazzi)]

I ran into this as well with --stream=tar. I see xtrabackup 2.0 uses libarchive:archive_write_set_format_ustar - a format which is limited to 8GB files. If the archive error were reported by xtrabackup it would state something along the lines of:

archive_write_header: File size out of range

The pax (or restricted pax) format is likely to be more useful here as the pax format supports much larger files, but it will generate warnings with GNU tar:

# tar ztivf foo.tar.gz
-rw-r--r-- root/root 264 2012-04-19 17:59:47 backup-my.cnf
tar: Ignoring unknown extended header keyword `SCHILY.dev'
tar: Ignoring unknown extended header keyword `SCHILY.ino'
tar: Ignoring unknown extended header keyword `SCHILY.nlink'
-rw-rw---- 0/0 2594177024 2012-04-19 17:18:19 ibdata1
...

[Steven Ayre (steveayre)]

Well done for tracking that size limit down. That certainly sounds like what I'm hitting - my ibdata1 is close to 100GiB.

[Alexey Kopytov (akopytov)]

Andrew,

Thank you for tracking down the problem.

From the libarchive man page, it looks like the 'restricted' PAX format would remove the 8 GB limit and avoid the extended keyword at the same time (if possible). Though looking at the source, files larger than 8 GB set the 'need_extension' flag and thus trigger the extended attributes even with the restricted PAX format. Those warnings are benign, but I guess they would cause a lot of confusion for users, and a lot of work on answering the related questions for us.

However, since XtraBackup uses bundled libarchive, I'm going to fix this by modifying libarchive source to suppress those SCHILY.* attributes that GNU tar is unhappy with. GNU tar is a requirement for tar streams generated by XtraBackup anyway, so we can safely assume those attributes are never used.

Thanks again for your help in verifying this bug.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-232