Innobackupex hangs on fatal error in some cases

[Raghavendra D Prabhu]

https://bugs.launchpad.net/percona-xtrabackup/+bug/1217426 has introduced a serious regression reported earlier here: https://bugs.launchpad.net/codership-mysql/+bug/1294760

This happens when backing up an empty test database and a directory called /tmp/test already exists. Innobackupex fails here

This also hangs innobackupex and xtrabackup indefinitely.

Happens here:

 opendir(DBDIR, "$source_dir/$database");
 @list = grep(/\.(frm|isl|MYD|MYI|MAD|MAI|MRG|TRG|TRN|ARM|ARZ|CSM|CSV|opt|par)$/, readdir(DBDIR));
 closedir DBDIR;
        $file_c = @list;
        if ($file_c <= $backup_file_print_limit) {
            $print_each_file = 1;
        } else {
            print STDERR "$prefix Backing up files " .
                "'$source_dir/$database/$wildcard' ($file_c files)\n";
        }

        if ($file_c == 0 && $option_stream) {
            # Stream/encrypt empty directories by backing up a fake empty
            # db.opt file, so that empty databases are created in the backup
            mkdir("$option_tmpdir/$database") ||
                die "Failed to create directory $option_tmpdir/$database: $!";

            open XTRABACKUP_FH, "> $option_tmpdir/$database/db.opt"
                || die "Cannot create file $option_tmpdir/db.opt: $!";
            close XTRABACKUP_FH;

            backup_file_via_stream("$option_tmpdir", "$database/db.opt");

            unlink("$option_tmpdir/$database/db.opt") ||
                die "Failed to remove file $database/db.opt: $!";
            rmdir("$option_tmpdir/$database") ||
                die "Failed to remove directory $database: $!";
        }

 mkdir("$option_tmpdir/$database") is triggering it, instead a mkdtemp-like function can be used to create unique directory to avoid collisions (which is pretty common for a name like 'test').

[Raghavendra D Prabhu (raghavendra-prabhu)]

Even providing a temporary directory on command line doesn't work (ie. the last option), only tmpdir argument works here, even though this is an innobackupex exception.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Currently, working around this, with tmpdir for both xtrabackup and innobackupex:

/usr/sbin/innobackupex --defaults-file=/pxc56/etc/my.cnf.local --no-version-check --tmpdir=/tmp/tmp.IhuFVZftFn --socket=/pxc56/datadir/pxc.sock --galera-info --stream=xbstream /tmp/tmp.mUBsJCxrld

However, two bugs remain in this:

a) option_tmpdir being used by innobackupex than option backup_dir (/tmp/tmp.mUBsJCxrld)

b) That innobackupex/xtrabackup hang when it fails.

[Alexey Kopytov (akopytov)]

The only wrong behavior that I see here is b) from comment #2, i.e. that innobackupex hangs on error.

Everything else is expected:

Conflicts with existing files in the temporary directory are always possible. XtraBackup uses --tmpdir to create a PID file, and (for streaming backups) temporary copies of xtrabackup_logfile, xtrabackup_checkpoints, backup-my.cnf, xtrabackup_binlog_info, xtrabackup_slave_info, etc, or (for --rsync) xtrabackup_rsync_pass1, xtrabackup_rsync_pass2. If naming conflicts are possible with any of those files, one has to ensure that --tmpdir points to a unique temporary directory.

The fact that --tmpdir specifies the location for temporary files is also documented.

I'm changing the title to reflect the actual problem.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-403